Predicting word boundaries for on-device batch processing of end-to-end speech recognition models

Abstract

The method (500) includes receiving a sequence of input audio frames (110) corresponding to an utterance (106) captured by a user device (10). The utterance includes a plurality of words. For each input audio frame, the method includes predicting whether the input audio frame is a word boundary using a word boundary detection model (210) configured to receive as input the sequence of input audio frames. The method includes batching the input audio frames into a plurality of batches (222) based on the input audio frames predicted as word boundaries. Each batch includes a corresponding plurality of batched input audio frames. For each of the plurality of batches, the method includes processing the corresponding plurality of batched input audio frames in parallel using a speech recognition model (230) to generate a speech recognition result (120).

Description

[Technical Field]

[0001] The present disclosure relates to predicting word boundaries for on-device batch processing of end-to-end speech recognition models. [Background technology]

[0002] Modern automatic speech recognition (ASR) systems focus on providing speech recognition of speech that is not only high quality (e.g., low word error rate) but also low latency (e.g., short delay between when a user speaks and when it is transcribed). For example, when using a device that implements an ASR system, it is often expected that the ASR system will decode the speech in real time, or even faster than real time, in a streaming manner. As an example, when an on-device ASR system is deployed on a mobile phone where direct user interaction occurs, it may be necessary to stream speech recognition to an application on the mobile phone that uses the ASR system, so that words appear on the screen immediately after they are spoken. Here, mobile phone users are also likely to have a low tolerance for latency. Due to this low tolerance, efforts are made to operate speech recognition as it runs on a mobile device in a way that minimizes the impact of latency and inaccuracies that can adversely affect the user's experience. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present disclosure provides a method including receiving a sequence of input audio frames corresponding to speech captured by a user device, the utterance including a plurality of words. For each input audio frame, the method includes predicting whether the input audio frame is a word boundary using a word boundary detection model configured to receive as input the sequence of input audio frames. The method batches the input audio frames into a plurality of batches based on the input audio frames predicted as word boundaries, each batch including a corresponding plurality of batched input audio frames. For each of the plurality of batches, the method includes processing the corresponding plurality of batched input audio frames in parallel using a speech recognition model to generate a speech recognition result.

[0004] Implementations of this disclosure may include one or more of the following optional features: In some implementations, the word boundary detection model includes a voice activity detection (VAD) model.

[0005] In some examples, the speech recognition model includes a first speech recognition model, and the word boundary detection model includes a second speech recognition model trained using transcript labels augmented with special boundary tokens between each pair of adjacent words. In some implementations, the transcript labels are tokenized into word pieces, and the first speech recognition model predicts the word pieces to generate speech recognition results. In some examples, the first speech recognition model is trained as a teacher model using transcript labels augmented with special boundary tokens between each pair of words, and the second speech recognition model is trained as a student model by performing gradient grafting, which determines a logit loss of the teacher model and applies a gradient of the logit loss of the teacher model to the weights of the student model. In some implementations, the second speech recognition model includes a miniaturized speech recognition model.

[0006] In some implementations, a first batch of the plurality of batches includes a first number of input audio frames of the sequence of input audio frames, and a second batch of the plurality of batches includes a different second number of input audio frames of the sequence of input audio frames. In some examples, the input audio frames of the first batch represent a first utterance, and the input audio frames of the second batch represent a different second utterance. In some implementations, each batch of the plurality of batches includes all of the input audio frames of the sequence of input audio frames that represent a particular utterance in the entire input audio frames.

[0007] In some implementations, the speech recognition model includes a transducer-based speech recognition model having an encoder network, a prediction network, and a joint network. In some examples, the encoder network includes multiple multi-head attention layers.

[0008] In some examples, the prediction network is configured to receive as input, at each of a plurality of time steps following the initial time step, the sequence of non-blank symbols output by the final softmax layer. At each of the plurality of time steps, the prediction network is configured to, for each non-blank symbol in the sequence of non-blank symbols received as input at the corresponding time step, generate an embedding representation of the corresponding non-blank symbol using a shared embedding matrix, assign a respective position vector to the corresponding non-blank symbol, and weight the embedding representation in proportion to the similarity between the embedding representation and the respective position vector. The prediction network is further configured to generate as output, a single embedding vector at the corresponding time step, the single embedding vector being based on a weighted average of the weighted embedding representations. In some implementations, the joint network is configured to receive as input, at each of a plurality of time steps following the initial time step, the single embedding vector generated as output from the prediction network at the corresponding time step, and generate a probability distribution over possible speech recognition hypotheses at the corresponding time step. In some examples, the prediction network ties the dimension of the shared embedding matrix to the dimension of the output layer of the joint network.

[0009] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving a sequence of input audio frames corresponding to speech captured by a user device. The speech includes a plurality of words. For each input audio frame, the operations include predicting whether the input audio frame is a word boundary using a word boundary detection model configured to receive as input the sequence of input audio frames. Based on the input audio frames predicted as word boundaries, the operations include batching the input audio frames into a plurality of batches, each batch including a corresponding plurality of batched input audio frames. For each of the plurality of batches, the operations include processing the corresponding plurality of batched input audio frames in parallel using a speech recognition model to generate a speech recognition result.

[0010] Implementations of this disclosure include one or more of the following optional features: In some implementations, the word boundary detection model includes a voice activity detection (VAD) model.

[0011] In some examples, the speech recognition model includes a first speech recognition model, and the word boundary detection model includes a second speech recognition model trained using transcript labels extended with special boundary tokens between each pair of adjacent words. In some implementations, the transcript labels are tokenized into word pieces, and the first speech recognition model predicts the word pieces to generate speech recognition results. In some examples, the first speech recognition model is trained as a teacher model using transcript labels extended with special boundary tokens between each pair of words, and the second speech recognition model is trained as a student model by performing gradient grafting, which determines the logit loss of the teacher model and applies the gradient of the logit loss of the teacher model to the weights of the student model. In some implementations, the second speech recognition model includes a compact speech recognition model.

[0012] In some implementations, a first batch of the plurality of batches includes a first number of input audio frames of the sequence of input audio frames, and a second batch of the plurality of batches includes a different second number of input audio frames of the sequence of input audio frames. In some examples, the input audio frames of the first batch represent a first utterance, and the input audio frames of the second batch represent a different second utterance. In some implementations, each batch of the plurality of batches includes all of the input audio frames of the sequence of input audio frames that represent a particular utterance in the entire input audio frames.

[0013] In some implementations, the speech recognition model includes a transducer-based speech recognition model having an encoder network, a prediction network, and a joint network. In some examples, the encoder network includes multiple multi-head attention layers.

[0014] In some examples, the prediction network is configured to receive as input, at each of a plurality of time steps following the initial time step, the sequence of non-blank symbols output by the final softmax layer. At each of the plurality of time steps, the prediction network is configured to, for each non-blank symbol in the sequence of non-blank symbols received as input at the corresponding time step, generate an embedding representation of the corresponding non-blank symbol using a shared embedding matrix, assign a respective position vector to the corresponding non-blank symbol, and weight the embedding representation in proportion to the similarity between the embedding representation and the respective position vector. The prediction network is further configured to generate as output, a single embedding vector at the corresponding time step, the single embedding vector being based on a weighted average of the weighted embedding representations. In some implementations, the joint network is configured to receive as input, at each of a plurality of time steps following the initial time step, the single embedding vector generated as output from the prediction network at the corresponding time step, and generate a probability distribution over possible speech recognition hypotheses at the corresponding time step. In some examples, the prediction network ties the dimension of the shared embedding matrix to the dimension of the output layer of the joint network.

[0015] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a schematic diagram of an exemplary speaking environment using an automatic speech recognition (ASR) system for transcription. [Figure 2] FIG. 2 is a schematic diagram of the example ASR system of FIG. 1. [Figure 3]FIG. 3 is a schematic diagram of an exemplary tied and reduced predictive network of the recurrent neural network transducer (RNN-T) model of FIG. [Figure 4A] FIG. 3 is a schematic diagram of an exemplary training module for training the ASR system of FIG. 2. [Figure 4B] FIG. 3 is a schematic diagram of another exemplary training module for training the ASR system of FIG. 2. [Figure 5] 1 is a flowchart of an exemplary arrangement of operations for a computer-implemented method of predicting word boundaries for on-device batch processing of an end-to-end speech recognition model. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0017] Like reference symbols in the various drawings indicate like elements.

[0018] While transcription quality (e.g., as measured by word error rate (WER)) is important for any ASR system, latency and power consumption are often particularly important for on-device ASR systems (e.g., ASR systems implemented on or by a user device). That is, on-device ASR systems often face a trade-off between battery usage and the amount of computation performed. For example, larger speech recognition models are generally more accurate, but often have higher computational costs and consume more power.

[0019] Batching input audio frames can improve battery usage for on-device ASR systems. That is, the ASR system advantageously batches a large number of discrete input audio frames and computes speech recognition predictions for all batched frame times in parallel. However, batching input audio frames can increase latency and, in some instances, decrease speech recognition accuracy. That is, it can take longer for the ASR system to recognize speech and output its transcription.

[0020] One way to better balance computation and latency while maintaining speech recognition quality is to batch together sets or batches of input audio frames representing full words. In such a scheme, each entire utterance is represented by a single batch of consecutive input audio frames, such that each batch of input audio frames is completed substantially simultaneously with the ASR system being able to first emit a transcription of the utterance. In an ideal case, this reduces or minimizes latency, as word transcriptions can be emitted immediately after a batch of input audio frames is processed. That is, in some cases, including those preceded by silence, there may be no additional latency even while the batch size is as large as an individual word. To implement such a scheme, it is important to quickly, accurately, and reliably identify word boundaries in the stream of input audio frames (i.e., specific input audio frames corresponding to the end of an utterance) so that processing of the corresponding batch of input audio frames by the speech recognition model can begin immediately after the end boundary of the word is detected.

[0021] In the examples disclosed herein, word boundary detection is used to predict word boundaries in the speech recognition output, and the predicted word boundaries are used to batch-process input audio frames into batches of input audio frames, each representing a corresponding entire utterance. That is, the predicted word boundaries can be used to correlate word end timing for the input audio frames. The disclosed examples do not affect speech recognition quality or final WER because the examples disclosed herein do not require modification of the underlying speech recognition model. Furthermore, battery usage is reduced or minimized because the size of each batch of input audio frames is dynamically and individually maximized to correspond to an entire utterance. Furthermore, latency is not increased because each batch of input audio frames ends at a respective word boundary. In some embodiments, a method includes receiving a sequence of input audio frames corresponding to speech captured by a user device, where the speech includes multiple words. For each input audio frame, the method may include predicting whether the input audio frame is at a word boundary using a word boundary detection model configured to receive as input the sequence of input audio frames. In these implementations, the method also includes batch-processing the input audio frames into a plurality of batches based on the input audio frames predicted as word boundaries, where each batch includes a corresponding plurality of batched input audio frames. The method may then include, for each of the plurality of batches, processing the corresponding plurality of batched input audio frames in parallel using a speech recognition model to generate a speech recognition result.

[0022] 1 is a schematic diagram of an exemplary system 100 operating in a speech environment 101. In the speech environment 101, a user 104 may interact with a computing device, such as a user device 10, via voice input. The user device 10 (also commonly referred to as a device 10) is configured to capture speech (e.g., streaming audio data) from one or more users 104 in the speech environment 101. Here, streaming audio data may refer to utterances 106 by the users 104, which serve as audible queries, commands to the device 10, audible communications captured by the device 10, etc. The voice-enabled system of the device 10 may appropriately process the queries or commands by answering the queries and / or having the commands executed / completed by one or more downstream applications.

[0023] The user device 10 may correspond to any computing device associated with a user and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, a mobile device (e.g., a mobile phone, a tablet, a laptop, etc.), a computer, a wearable device (e.g., a smart watch), a smart appliance, an Internet of Things (IoT) device, an in-vehicle infotainment system, a smart display, a smart speaker, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 in communication with the data processing hardware 12, storing instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 having audio capture devices (e.g., microphones) 16, 16a for capturing and converting speech 106 in a speech environment 101 into electrical signals, and speech output devices (e.g., speakers) 16, 16b for communicating audible audio signals (e.g., as output audio data from the device 10). Although the user device 10 implements a single audio capture device 16a in the illustrated example, it may implement an array of audio capture devices 16a without departing from the scope of this disclosure, and one or more capture devices 16a in the array may not be physically present on the user device 10 but may communicate with the audio system 16.

[0024] In particular, the system 100, including an automatic speech recognition (ASR) system 200 implementing a recurrent neural network transducer (RNN-T)-based speech recognition model 230 and an optional rescorer 180, resides on a user device 10 of a user 104 and / or on a remote computing device 60 (e.g., one or more remote or central servers of a distributed system implemented in a cloud computing environment) that communicates with the user device 10 via a network 40. In some examples, the ASR system 200 is an end-to-end ASR system. The user device 10 and / or the remote computing device 60 also includes an audio subsystem 108 configured to receive an utterance 106 spoken by the user 104 and captured by the audio capture device 16a, and to convert the utterance 106 into a corresponding digital format associated with input audio frames 110 that can be processed by the ASR system 200. In the illustrated example, a user utters a respective utterance 106, and the audio subsystem 108 converts the utterance 106 into corresponding input audio data (e.g., input acoustic or audio frames) 110 for input to the ASR system 200. The RNN-T model 230 then receives or batch-processes the audio data 110 corresponding to the utterance 106 as input and generates / predicts a corresponding transcription 120 (e.g., a recognition result / hypothesis) of the utterance 106 as output. In the illustrated example, the RNN-T model 230 may perform streaming speech recognition to generate initial speech recognition results 120, 120a, and the rescorer 180 may update (rescore) the initial speech recognition result 120a to generate final speech recognition results 120, 120b. The server 60 includes data processing hardware 62 and memory hardware 64 in communication with the data processing hardware 62. Memory hardware 64 stores instructions that, when executed by data processing hardware 62, cause data processing hardware 62 to perform one or more operations, such as those disclosed herein.

[0025] The user device 10 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of a transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 may stream initial speech recognition results 120a for time 1 and then display final speech recognition results 120b for time 2. In some configurations, the transcription 120 output from the ASR system 200 is processed by a natural language processing / understanding (NLP / NLU) module executing, for example, on the user device 10 or the remote computing device 60 to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech (TSS) system (not shown) (e.g., executing on any combination of the user device 10 or the remote computing device 60) may convert the transcription into synthesized speech for audible output by the user device 10 and / or another device.

[0026] In the illustrated example, a user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) running on a user device 10 that uses an ASR system 200. For example, FIG. 1 shows a user 104 communicating with the digital assistant application 50, with the user device 10 running a graphical user interface 18 displayed on the screen of the user device 10, illustrating a conversation between the user 104 and the digital assistant application 50. For example, FIG. 1 shows a user 104 communicating with the digital assistant application 50 and the digital assistant application 50 displaying a digital assistant interface 18 on the screen of the user device 10, illustrating a conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, "What time does the concert start tonight?" This question from the user 104 is utterance 106 that is captured by an audio capture device 16a and processed by the audio system 16 of the user device 10. In this example, audio system 16 receives and converts speech 106 into input audio frames 110 for input to ASR system 200 .

[0027] Continuing with this example, as the RNN-T model 230 receives audio frames 110 or a batch thereof corresponding to an utterance 106 uttered by the user 104, it encodes the audio frames 110 and then decodes the encoded audio frames 110 into initial speech recognition results 120a. During time 1, the user interface generator 107 presents a representation of the initial speech recognition results 120a of the utterance 106 to the user 104 of the user device 10 in a streaming manner via the digital assistant interface 18, such that words, word pieces, and / or individual characters appear on the screen immediately after they are spoken. In some examples, the first look-ahead audio context is equal to zero.

[0028] During time 2, user interface generator 107 presents a representation of final speech recognition result 120b of utterance 106 to user 104 of user device 10 via digital assistant interface 18, to be rescored by rescorer 180. In some implementations, user interface generator 107 replaces the representation of initial speech recognition result 120a presented at time 1 with the representation of final speech recognition result 120b presented at time 2. Here, time 1 and time 2 may include timestamps corresponding to when user interface generator 107 presents the respective speech recognition results. In this example, the timestamp for time 1 indicates that user interface generator 107 presents initial speech recognition result 120a at an earlier time than final speech recognition result 120b. For example, because final speech recognition result 120b is estimated to be more accurate than initial speech recognition result 120a, final speech recognition result 120b, which is ultimately displayed as transcription 120, can correct any words that may have been misrecognized in initial speech recognition result 120a. In this example, the streaming initial speech recognition results 120a output by the RNN-T model 230 and displayed on the screen of the user device 10 at time 1 involve low latency and provide the user 104 with a response that their query is being processed, while the final speech recognition results 120b output by the rescorer 180 and displayed on the screen at time 2 utilize additional speech recognition and / or language models to improve speech recognition quality in terms of accuracy but increase latency. However, because the initial speech recognition results 120a are displayed as the user utters the utterance 106, the higher latency involved in generating and ultimately displaying the final recognition results is not noticeable to the user 104.

[0029] In the example shown in FIG. 1 , digital assistant application 50 may respond to a question posted by user 104 by using NLP / NLU processing. NLP / NLU generally refers to the process of interpreting written language (e.g., initial speech recognition result 120a and / or final speech recognition result 120b) and determining whether the written language prompts an action. In this example, digital assistant application 50 uses NLP / NLU to recognize that the question from user 104 is about the user's schedule, more specifically, a concert on the user's schedule. By recognizing these details through NLP / NLU, the automated assistant returns a response 19 to the user's query, which in this case states, "Doors open at 6:30 PM, and the concert starts at 8 PM." In some configurations, NLP / NLU occurs on a remote server 60 in communication with data processing hardware 12 of user device 10.

[0030] 2 is a schematic diagram of an example of the ASR system 200 of FIG. 1 , including a word boundary detection model 210, a batch module 220, and an RNN-T model 230. The word boundary detection model 210 processes input audio frames 110 to determine word boundaries. Specifically, the word boundary detection model 210 is configured and trained to determine, for each particular input audio frame 110, an output 212 that indicates whether the particular input audio frame 110 corresponds to a word boundary. That is, the word boundary detection model 210 identifies an input audio frame 110 as a word boundary that represents a boundary between adjacent pairs of utterances in the input audio frame 110. In some examples, if the word boundary detection model 210 identifies a particular input audio frame 110 as a word boundary, it outputs a corresponding output 212 having a first value (e.g., representing a logical "true"). If the word boundary detection model 210 does not identify a particular input audio frame 110 as a word boundary, it outputs a corresponding output 212 having a different second value (e.g., representing a logical "false").

[0031] In some examples, the word boundary detection model 210 includes a voice activity detection (VAD) model configured and trained to determine an output 212 for each input audio frame 110 that indicates whether the corresponding input audio frame 110 includes data representing speech.

[0032] In additional or alternative implementations, the word boundary detection model 210 includes a speech recognition model configured and trained to detect word boundaries, for example, using acoustic and textual information. In some examples, the speech recognition model includes an RNN-T model, such as the RNN-T model 230. Here, the speech recognition model may be a smaller, simpler, or more compact version of the RNN-T model 230. The speech recognition model includes an encoder that may be configured as a conformer with a cell width of 512 (e.g., including 32 million parameters) or a cell width of 144 (e.g., including 4 million parameters), and a decoder configured as an embedded decoder. In some examples, the word boundary detection model 210 is trained using sequence-level emission regularization to encourage the speech recognition model to identify word boundaries more quickly. In some additional examples, the word boundary detection model 210 is trained using supervised learning based on paired training data. Here, the paired training data consists of audio-transcript pairs, i.e., input audio frames 110 and special boundary tokens (e.g., " ) inserted between pairs of adjacent words. <wb>"), where the transcript labels may be manually annotated. However, the transcript labels may be generated or determined in any other manner. In some examples, the RNN-T model 230 predicts word-piece units while performing speech recognition, and the transcript labels are tokenized into word-piece units.

[0033] In additional or alternative implementations, word boundary detection model 210 includes a speech recognition model configured and trained to predict word boundaries predicted by RNN-T model 230. In some examples, the speech recognition model includes an RNN-T model, such as RNN-T model 230. Here, the speech recognition model may be a smaller, simpler, or more compact version of RNN-T model 230. The speech recognition model includes an encoder that may be configured as a conformer with a cell width of 512 (e.g., including 32 million parameters) or a cell width of 144 (e.g., including 4 million parameters), and a decoder configured as an embedded decoder. In some examples, word boundary detection model 210 is trained using knowledge distillation and gradient grafting to learn to predict word boundaries expected by RNN-T model 230. Here, training of ASR model 200 uses RNN-T model 230 as a teacher model used in training word boundary detection model 210 as a student model. During training, an additional loss is applied to the word boundary detection model 210 by calculating the full or approximate KL divergence between the teacher and student predictions, i.e., a lattice distribution of conditional probabilities over word boundary detection. Here, the RNN-T model 230 is trained using the same type of paired training data in audio-transcript pairs described above. In one example, the RNN-T loss is determined based on the logits of the teacher model during training of the teacher model (i.e., the RNN-T model 230) using the paired training data, and a gradient is determined based on the logits of the teacher model, and the gradient is applied to the student model to train the student model (i.e., the word boundary detection model 210). For example, the logits of the teacher model (i.e., the RNN-T model 230) are teacher and the logits of the student model (i.e., the word boundary detection model 210) student Given this, the logits for training the student model are calculated as follows: logits = stop_gradient(logits teacher -logits student ) + logits student ... (1) The logits in equation (1) may be used to determine a weighted sum that is calculated for and applied to the standard loss used to update the student model (i.e., word boundary detection model 210). Alternatively, the logits in equation (1) may be applied as the sole loss function used to update the student model (i.e., word boundary detection model 210).

[0034] The batch module 220 batches the input audio frames 110 into one or more batches 222, each representing a respective utterance in the entire input audio frames 110. Each batch 222 includes a plurality of consecutive input audio frames 110 that together represent a corresponding utterance. The batch module 220 divides the input audio frames 110 into batches 222 based on the output 212 of the word boundary detection model 210, which identifies which audio input frames 110 correspond to word boundaries. The batch module 220 uses these identified word boundaries to designate a plurality of consecutive input audio frames 110 as batches 222. The batch module 220 batches the input audio frames 110 based on the word boundaries identified by the word boundary detection model 210, and because utterances have different lengths, the batches 222 may include different numbers of input audio frames 110.

[0035] The RNN-T model 230 offers a small computational footprint and utilizes lower memory requirements than traditional ASR architectures, making the RNN-T model 230 suitable for performing speech recognition entirely on the user device 102 (e.g., no communication with a remote server is required).

[0036] As shown, the RNN-T model 230 includes an encoder network 240, a prediction network 300, a joint network 250, and a final softmax layer 260. The prediction network 300 and the joint network 250 collectively constitute an RNN-T decoder. The encoder network 240 is similar to an acoustic model (AM) in a conventional ASR system and may include a network of multi-head attention layers, such as, but not limited to, a conformer layer or a transformer layer. For example, the encoder may generate a batch 222 of d-dimensional feature vectors (e.g., a batch 222 of input audio frames 110 (FIG. 1)) x = (x1, x2, ..., x T ) where

number

number

[0037] Similarly, the prediction network 300 is also an LSTM network, and like the language model (LM), it computes the sequence of non-blank symbols 232 output by the final softmax layer 260 so far, y 0 , ..., y ui-1 tightly coupled or hidden representation

number

number

[0038] FIG. 3 shows the non-blank symbols 232 y , limited to the N previous non-blank symbols 232 a to 232 n output by the final softmax layer 260. ui-n , ..., y ui-1 1 illustrates a prediction network 300 of an RNN-T model 230 receiving as input a sequence of non-blank symbols 232a-232n. In some examples, N is equal to 2. In other examples, N is equal to 5, although this disclosure is non-limiting and N may be equal to any integer. The sequence of non-blank symbols 232a-232n represents the initial speech recognition result 120a (FIG. 1). In some implementations, the prediction network 300 includes a multi-head attention mechanism 302, which shares a shared embedding matrix 304 across each of its heads 302A-302H. In one example, the multi-head attention mechanism 302 includes four heads. However, any number of heads may be used by the multi-head attention mechanism 302. Notably, the multi-head attention mechanism significantly improves performance with minimal increase in model size. As explained in more detail below, each head 302A-302H contains its own row of position vectors, and instead of increasing the model size by concatenating the outputs 318A-318H from all heads, the outputs 318A-318H are averaged by a head averaging module 322.

[0039] Referring to the first head 302A of the multi-head attention mechanism 302, the head 302A uses a shared embedding matrix 304 to generate a set of non-blank symbols 232a-232n y received as input at corresponding time steps from the plurality of time steps. ui-n , ..., y ui-1 For each non-blank symbol in the sequence, a corresponding embedded representation 306, 306a-306n (e.g.,

number

number

[0040] Non-blank symbols 232a-232n, y ui-n , ..., y ui-1 The corresponding embedding representation generated by the shared embedding matrix 304 for each non-blank symbol in the sequence is the same for all of the heads 302A-302H of the multi-head attention mechanism 302, but each head 302A-302H defines a different set / row of position vectors 308. For example, the first head 302A defines a position vector PV Aa~An The second head 302B defines a row of 308Aa to 308An, and the second head 302B defines a position vector PV Ba~Bn 308 Ba~Bn The Hth head 302H defines a position vector PV Ha~Hn 308 Ha~Hn define another different row of

[0041] For each non-blank symbol in the received sequence of non-blank symbols 232a-232n, the first head 302A also weights the corresponding embedded representation 306 via the weighting layer 310 in proportion to the similarity between the corresponding embedded representation and its assigned respective position vector PV 308. In some examples, the similarity may include cosine similarity (e.g., cosine distance). In the illustrated example, the weighting layer 310 outputs a sequence of weighted embedded representations 312, 312Aa-312An, each associated with a corresponding embedded representation 306 weighted in proportion to its assigned respective position vector PV 308. In other words, the weighted embedded representation 312 output by the weighting layer 310 for each embedded representation 306 may correspond to a dot product between the embedded representation 306 and its assigned respective position vector PV 308. The weighted embedded representation 312 may be interpreted as an attenuation for the embedded representation proportional to how similar it is to the position associated with its assigned respective position vector PV 308. To increase computational speed, the prediction network 300 includes a non-recurrent layer, so that the sequence of weighted embeddings 312Aa-312An are not concatenated but instead averaged by a weighted average module 316 to produce a weighted average 318A of the weighted embeddings 312Aa-312An as output from the first head 302A, expressed as follows:

number

[0042] The operations described above with respect to the first head 302A are similarly performed by each of the other heads 302B-302H of the multi-head attention mechanism 302. Due to the different sets of position vectors PV 308 defined by each head 302, the weighting layer 310 outputs a sequence of weighted embedding representations 312Ba-312Bn, 312Ha-312Hn for each of the other heads 302B-302H that differs from the sequence of weighted embedding representations 312Aa-312An for the first head 302A. The weighted average module 316 then generates a weighted average 318A-318H of the corresponding weighted embedding representations 312 of the sequences of non-blank symbols as output from each of the other corresponding heads 302B-302H.

[0043] In the illustrated example, the prediction network 300 includes a head averaging module 322 that averages weighted averages 318A-318H output from corresponding heads 302A-302H. A projection layer 326 with SWISH may receive as input an output 324 from the head averaging module 322 corresponding to the average of the weighted averages 318A-318H and generate as output a projected output 328. A final layer normalization 330 normalizes the projected output 328 to produce a single embedding vector at corresponding time steps from multiple time steps.

number

number

[0044] In some configurations, the prediction network 300 does not implement a multi-head attention mechanism 302, and performs only the operations described above for the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-312An is simply passed through the projection layer 326 and layer normalization 330 to produce a single embedding vector

number

[0045] Referring again to FIG. 2, the high-dimensional feature representation generated by the encoder network 240

number

number

number

[0046] The final softmax layer 260 receives the probability distribution Zi 252 for the final speech recognition result 120b and selects the output label / symbol with the highest probability to generate the transcription. i Any technique may be used to select the output label / symbol with the highest probability in 232. In this way, the RNN-T model 230 does not make a conditional independence assumption, but rather, each symbol y u The prediction of 232 is based not only on the acoustics but also on the labels output up to that point. ui-n , ..., y ui-1 The RNN-T model 230 assumes that the output symbols 232 are independent of future acoustic frames 110, which allows the RNN-T model to be used in a streaming manner.

[0047] Because the RNN-T model 230 determines the initial speech recognition result 120a in a streaming manner and determines the final speech recognition result 120b using previous non-blank symbols from the initial speech recognition result 120a, the final speech recognition result 120b is presumed to be more accurate than the initial speech recognition result 120a. That is, the final speech recognition result 120b takes into account previous non-blank symbols, and therefore, the final speech recognition result 120b is presumed to be more accurate because the initial speech recognition result 120a does not take into account previous non-blank symbols. Furthermore, the rescorer 180 (FIG. 1) may update the initial speech recognition result 120a with the final speech recognition result 120b and provide the transcription to the user 104 via the user interface generator 170.

[0048] In some implementations, to further reduce the size of the RNN-T decoder (i.e., the prediction network 300 and the joint network 250), parameter tying between the prediction network 300 and the joint network 250 is applied. Specifically, the vocabulary size |V| and the embedding dimension d e , the shared embedding matrix 304 in the prediction network is

number

number

[0049] In some examples, audio data captured by the audio subsystem 108 is processed as 128-dimensional log-mel features calculated using a 32 millisecond (ms) window and a 10 ms shift, then stacked three frames left and downsampled by a factor of 3 to obtain a frame rate of 30 ms.

[0050] The softmax layer 260 may consist of a unified set of word pieces or graphemes generated using all unique word pieces or graphemes in the training data. When the output symbols / labels include word pieces, the set of output symbols / labels may include 4096 different word pieces. When the output symbols / labels include graphemes, the set of output symbols / labels may include fewer than 100 different graphemes.

[0051] 4A is a schematic diagram of an exemplary training process 400a for training the word boundary detection model 210 (FIG. 2) of the ASR system 200. Here, the training process 400a performs supervised training using paired training samples 405 containing audio-transcript pairs. That is, the training samples 405 are generated from input audio frames 410 and special boundary tokens (e.g., " ) inserted between pairs of adjacent words. <wb>") and corresponding ground truth transcript labels 415, where the transcript labels 415 may be manually annotated. However, the transcript labels may be obtained or generated in any other manner. In some examples, the transcript labels 415 are tokenized into word pieces.

[0052] For each particular training sample 405, the word boundary detection model 210 receives as input the input training audio frames 410 of the particular training sample 405. As described in more detail above, the word boundary detection model 210 processes each input audio frame 410 to determine whether the audio frame 410 corresponds to a word boundary. In particular, the training process 400a trains the word boundary detection model 210 to determine a corresponding output 422 that indicates whether the corresponding training input audio frame 410 corresponds to a word boundary. Thereafter, the loss term module 430 determines, for each particular training sample 405, a loss term 432 associated with the output 422 of the word boundary detection model 210 for the particular training sample 405 based on the corresponding ground truth transcription label 415 for the particular training sample 405.

[0053] The training process 400a trains the word boundary detection model 210 by updating or refining the parameters of the word boundary detection model 210 based on the loss term 432. For example, the training process 400a may update the parameters of the word boundary detection model 210 based on the loss term 432 using stochastic gradient descent.

[0054] 4B is a schematic diagram of another exemplary training process 400b for training word boundary detection model 210 of ASR system 200. Here, training process 400b performs supervised training using paired training samples 405, such as the paired training samples 405 described above in connection with training process 400a of FIG. 4A. Word boundary detection model 210, output 422, loss term module 430, and loss term 432 are described above in connection with FIG. 4a.

[0055] For each particular training sample 405, the ASR model 230 receives as input the input training audio frames 410 for the particular training sample 405 and processes the input training audio frames 410 to predict a transcription 442 for the particular training sample 405. A loss term module 450 then determines, for each particular training sample 405, a loss term 452 associated with the predicted transcription 442 for the particular training sample 405 based on the corresponding ground truth transcription label 415 for the particular training sample 405. In some examples, the training process 400b also trains the ASR model 230 by updating or refining parameters of the ASR model 230 based on the loss terms 452. For example, the training process 400b may update the parameters of the ASR (e.g., RNN-T) model 230 based on the loss terms 452 using stochastic gradient descent.

[0056] The gradient grafting module 460 determines a loss 462 that can be used to update the word boundary detection model 210 based on the loss 432 of the word boundary detection model 210 and the loss 452 of the RNN-T model 440. In one example, the gradient grafting module 460 processes the loss 452 to obtain the logits of the ASR model 230 that are related to predicting word boundaries. teacher and process the loss 432 to obtain the logits of the word boundary detection model 210. student and determines the logits for training the word boundary detection model 210, for example, using equation (1). The gradient grafting module 460 determines a weighted sum based on the logits and applies the weighted sum to the loss 432 to determine the loss 462 for updating the word boundary detection model 210. Alternatively, the gradient grafting module 460 can apply the logits as the sole loss function used to determine the loss 462 for updating the word boundary detection model 210.

[0057] In some examples, the training process 400b also trains the word boundary detection model 210 by updating or refining the parameters of the word boundary detection model 210 based on the loss term 462. For example, the training process 400b may update the parameters of the word boundary detection model 210 based on the loss term 462 using stochastic gradient descent.

[0058] 5 is a flowchart illustrating an exemplary sequence of operations for a computer-implemented method 500 of predicting word boundaries for on-device batch processing of an end-to-end speech recognition model. At operation 502, the method 500 includes receiving a sequence of input audio frames 110 corresponding to an utterance 106 captured by the user device 10. The utterance 106 includes a plurality of words. At operation 504, the method 500 includes, for each input audio frame 110, predicting whether the input audio frame 110 is a word boundary using a word boundary detection model (e.g., word boundary detection model 210) configured to receive as input the sequence of input audio frames 110.

[0059] At operation 506, the method 500 includes batching the input audio frames 110 into a plurality of batches 222 based on the input audio frames 110 predicted as word boundaries, with each batch 222 including a corresponding plurality of batched input audio frames 110. At operation 508, the method 500 includes, for each of the plurality of batches 222, processing the corresponding plurality of batched input audio frames 110 in parallel using a speech recognition model (e.g., RNN-T 230) to generate speech recognition results 232.

[0060] 6 is a schematic diagram of an exemplary computing device 600 that may be used to implement the systems and methods described herein. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality are merely exemplary and do not limit the implementation of the inventions described and / or claimed herein.

[0061] Computing device 600 includes a processor 610 (i.e., data processing hardware) that may be used to implement data processing hardware 12 and / or 62, a memory 620 (i.e., memory hardware) that may be used to implement memory hardware 14 and / or 64, a storage device 630 (i.e., memory hardware) that may be used to implement memory hardware 14 and / or 64, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be mounted on a common motherboard or otherwise mounted as appropriate. The processor 610 processes instructions executed within the computing device 600, including instructions stored in memory 620 or on storage device 630, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 680 coupled to a high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memory, as appropriate. Multiple computing devices 600 may also be connected (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system), with each device providing a portion of the required operations.

[0062] The memory 620 stores information non-transiently within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disk or tape.

[0063] The storage device 630 can provide mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory, or other similar solid-state memory device, or a device in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 620, the storage device 630, or memory on the processor 610.

[0064] The high-speed controller 640 manages bandwidth-intensive operations for the computing device 600, and the low-speed controller 660 manages lower-bandwidth intensive operations. Such duty allocation is merely exemplary. In some implementations, the high-speed controller 640 is coupled to the memory 620, coupled to the display 680 (e.g., via a graphics processor or accelerator), and coupled to a high-speed expansion port 650 that may accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and to a low-speed expansion port 690. The low-speed expansion port 690 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., via a network adapter.

[0065] Computing device 600 may be implemented in several different forms, as shown, for example, as a standard server 600a, or multiple such servers 600a, or as a laptop computer 600b, or as part of a rack server system 600c.

[0066] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, and which may be coupled to receive and transmit data and instructions from a storage system, at least one input device, and at least one output device.

[0067] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0068] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0069] The processes and logic flows described herein can be performed by one or more programmable processors, also known as data processing hardware, which execute one or more computer programs to perform functions by processing input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuitry, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random-access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more storage devices, such as magnetic, magneto-optical, or optical disks, for storing data and is operably coupled to receive data from, transfer data to, or both of these devices. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0070] To enable user interaction, one or more aspects of the present disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display), monitor, or touch screen, for displaying information to the user, and possibly a keyboard and pointing device, e.g., a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to enable user interaction; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can send and receive documents to and from devices used by the user, and interact with the user by, for example, sending web pages to a web browser on the user's client device in response to a request received from the web browser.

[0071] Unless expressly stated to the contrary, "or" refers to an inclusive or, not an exclusive or. For example, "A, B, or C" refers to any combination or subset of A, B, or C, such as (1) A only, (2) B only, (3) C only, (4) A and B, (5) A and C, (6) B and C, or (7) A, B, and C. Similarly, the phrase "at least one of A or B" is intended to refer to any combination or subset of A and B, such as (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Furthermore, the phrase "at least one of A and B" is intended to refer to any combination or subset of A and B, such as (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0072] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0073] 10 User Devices 12 Data Processing Hardware 14 Memory Hardware 16, 16a Audio Capture Device 18 Interface 19 responses 40 Network 50 Digital Assistant Applications 60 Remote Computing Devices 62 Data Processing Hardware 64 Memory Hardware 100 systems 101 Speech Environment 102 User Devices 104 users 106 utterances 107 User Interface Generator 108 Audio Subsystem 110 input audio frames 120 Transcription 120a Initial speech recognition results 120b Final speech recognition result 170 User Interface Generator 180 Lisco Ara 200 ASR System 210 Word Boundary Detection Model 212 Output 220 Batch Module 222 batch 230 RNN-T model, speech recognition model 232, 232a-232n non-blank symbols 240 Encoder Network 242 High-dimensional feature representation 250 Joint Network 252 Probability Distribution 260 Final softmax layer 300 Prediction Network 302 Multi-head Attention Mechanism 302A~302H Head 302B~302H Other heads 304 Shared Embedding Matrix 306, 306a-306n Embedded Representation 308, 308Aa~308An position vectors 310 Weight Layer 312, 312Aa~312An Weighted embedding representation 316 Weighted Average Module 318A~318H Output 322 head average module 324 output 326 Projection layer 328 projected output 350 embedding vectors 400a, 400b Training Process 405 training samples 410 Input Training Audio Frames 415 Transcript Label 422 output 430 Loss Term Module 432 Loss term 440 RNN-T model 442 Transcription 450 Loss Term Module 452 Loss term 460 Gradient Grafting Module 462 loss 500 Computer Implementation Methods 600 computing devices 600a Server 600b laptop computer 600c Rack Server System 610 processor 620 memory 630 Storage Devices 640 High-Speed ​​Interface / Controller 650 High-Speed ​​Expansion Port 660 Low-Speed ​​Interface / Controller 670 Slow Bus 680 display 690 Low-Speed ​​Expansion Port< / wb> < / wb>

Claims

1. A method (500) implemented in data processing hardware (610), comprising: receiving a sequence of input audio frames (110) corresponding to an utterance (106) captured by a user device (10), the utterance (106) including a plurality of words; predicting, for each input audio frame (110), whether the input audio frame (110) is a word boundary between a pair of adjacent words of the plurality of words using a word boundary detection model (210) configured to receive as input the sequence of input audio frames (110), wherein the word boundary detection model has been trained using transcript labels augmented with special boundary tokens between each pair of adjacent words; batching the sequence of input audio frames (110) into a plurality of batches (222) based on the input audio frames (110) predicted as word boundaries, each batch (222) comprising a corresponding plurality of batched input audio frames (110) representing a respective one of the plurality of words in the utterance; for each of the plurality of batches (222), processing the corresponding plurality of batched input audio frames (110) in parallel using a neural network-based speech recognition model (230) to generate a speech recognition result (120); A method (500) comprising:

2. 10. The method of claim 1, wherein the word boundary detection model comprises a voice activity detection (VAD) model.

3. 10. The method of claim 1, wherein the speech recognition model comprises a first speech recognition model and the word boundary detection model comprises a second speech recognition model trained using transcript labels augmented with special boundary tokens between each pair of adjacent words.

4. 4. The method of claim 3, wherein the transcript labels are tokenized into word pieces, and the first speech recognition model predicts the word pieces to generate the speech recognition results.

5. the first speech recognition model is trained as a teacher model using the transcript labels (415) augmented with the special boundary tokens between each pair of adjacent words; 4. The method (500) of claim 3, wherein the second speech recognition model is trained as the student model by determining a logit loss of the teacher model and performing gradient grafting, applying a gradient of the logit loss of the teacher model to weights of a student model.

6. 4. The method (500) of claim 3, wherein the second speech recognition model has a smaller number of parameters than the first speech recognition model.

7. a first batch (222) of the plurality of batches (222) including a first number of input audio frames (110) of the sequence of input audio frames (110); 2. The method of claim 1, wherein a second batch of the plurality of batches includes a second, different number of input audio frames from the sequence of input audio frames.

8. the input audio frames (110) of the first batch (222) represent a first utterance; 8. The method of claim 7, wherein the input audio frames of the second batch represent a different second utterance.

9. 2. The method of claim 1, wherein each batch of the plurality of batches includes all of the input audio frames of a sequence of the input audio frames representing a particular utterance in the entire input audio frames.

10. 10. The method of claim 1, wherein the speech recognition model comprises a transducer-based speech recognition model comprising an encoder network, a prediction network, and a joint network.

11. 11. The method of claim 10, wherein the encoder network comprises multiple multi-head attention layers.

12. The prediction network (300) performs, at each of a plurality of time steps following an initial time step: receiving as input the sequence of non-blank symbols (232) output by the final softmax layer (260); For each non-blank symbol (232) in the sequence of non-blank symbols (232) received as input at the corresponding time step: generating embedded representations (306) of the corresponding non-blank symbols (232) using the shared embedding matrix (304); assigning respective position vectors (308) to the corresponding non-blank symbols (232); weighting the embedded representations (306) in proportion to the similarity between the embedded representations (306) and the respective position vectors (308); generating as output a single embedding vector (350) for the corresponding time step, the single embedding vector (350) being based on a weighted average (316) of the weighted embedding representations (312); configured to: The joint network (250) performs, at each of the plurality of time steps following the initial time step: receiving as input the single embedding vector (350) generated as output from the prediction network (300) at the corresponding time step; generating a probability distribution (252) over the possible speech recognition hypotheses at the corresponding time step; 12. The method (500) according to claim 10 or 11, configured to:

13. 13. The method of claim 12, wherein the prediction network ties the dimensions of the shared embedding matrix to the dimensions of an output layer of the joint network.

14. data processing hardware (610); and memory hardware (620) in communication with the data processing hardware (610), the memory hardware (620) storing instructions that, when executed on the data processing hardware (610), cause the data processing hardware (610) to perform operations, the operations including: receiving a sequence of input audio frames (110) corresponding to an utterance (106) captured by a user device (10), the utterance (106) including a plurality of words; predicting, for each input audio frame (110), whether the input audio frame (110) is a word boundary between a pair of adjacent words of the plurality of words using a word boundary detection model (210) configured to receive as input the sequence of input audio frames (110), wherein the word boundary detection model has been trained using transcript labels augmented with special boundary tokens between each pair of adjacent words; batching the sequence of input audio frames (110) into a plurality of batches (222) based on the input audio frames (110) predicted as word boundaries, each batch (222) comprising a corresponding plurality of batched input audio frames (110) representing a respective one of the plurality of words in the utterance; for each of the plurality of batches (222), processing the corresponding plurality of batched input audio frames (110) in parallel using a neural network-based speech recognition model (230) to generate a speech recognition result; A system (100) including:

15. 15. The system of claim 14, wherein the word boundary detection model comprises a voice activity detection (VAD) model.

16. 15. The system of claim 14, wherein the speech recognition model comprises a first speech recognition model and the word boundary detection model comprises a second speech recognition model trained using transcript labels augmented with special boundary tokens between each pair of adjacent words.

17. 17. The system of claim 16, wherein the transcript labels are tokenized into word pieces, and the first speech recognition model predicts word pieces to generate the speech recognition result.

18. the first speech recognition model is trained as a teacher model using the transcript labels (415) augmented with the special boundary tokens between each pair of adjacent words; 17. The system (100) of claim 16, wherein the second speech recognition model is trained as the student model by determining a logit loss of the teacher model and performing gradient grafting, applying a gradient of the logit loss of the teacher model to weights of a student model.

19. 19. The system (100) of any one of claims 16 to 18, wherein the number of parameters of the second speech recognition model is less than the number of parameters of the first speech recognition model.

20. a first batch (222) of the plurality of batches (222) including a first number of input audio frames (110) of the sequence of input audio frames (110); 15. The system (100) of claim 14, wherein a second batch (222) of the plurality of batches (222) includes a different second number of input audio frames (110) from the sequence of input audio frames (110).

21. the input audio frames (110) of the first batch (222) represent a first utterance; 21. The system of claim 20, wherein the input audio frames of the second batch represent a different second utterance.

22. 15. The system of claim 14, wherein each batch of the plurality of batches includes all of the input audio frames of a sequence of the input audio frames representing a particular utterance in the entire input audio frames.

23. 15. The system of claim 14, wherein the speech recognition model comprises a transducer-based speech recognition model comprising an encoder network, a prediction network, and a joint network.

24. 24. The system of claim 23, wherein the encoder network comprises multiple multi-head attention layers.

25. The prediction network (300) performs, at each of a plurality of time steps following an initial time step: receiving as input the sequence of non-blank symbols (232) output by the final softmax layer (260); For each non-blank symbol (232) in the sequence of non-blank symbols (232) received as input at the corresponding time step: generating embedded representations (306) of the corresponding non-blank symbols (232) using the shared embedding matrix (304); assigning respective position vectors (308) to the corresponding non-blank symbols (232); weighting the embedded representations (306) in proportion to the similarity between the embedded representations (306) and the respective position vectors (308); generating as output a single embedding vector (350) for the corresponding time step, the single embedding vector (350) being based on a weighted average (316) of the weighted embedding representations (312); configured to: The joint network (250) performs, at each of the plurality of time steps following the initial time step: receiving as input the single embedding vector (350) generated as output from the prediction network (300) at the corresponding time step; generating a probability distribution (252) over the possible speech recognition hypotheses at the corresponding time step; 25. The system (100) of claim 23 or 24, configured to:

26. 26. The system of claim 25, wherein the prediction network ties the dimensions of the shared embedding matrix to the dimensions of an output layer of the joint network.

Images