Training for long-form speech recognition

Abstract

The method includes obtaining a set of training samples, each training sample including a corresponding sequence of speech segments corresponding to a training utterance and a corresponding sequence of ground truth transcriptions for the sequence of speech segments, each ground truth transcription including a start time and an end time of the corresponding speech segment. For each training sample, the method includes processing the corresponding sequence of speech segments using a speech recognition model to obtain speech recognition hypotheses for the training utterance, and identifying, for each speech recognition hypothesis obtained for the training utterance, a number of word errors relative to the corresponding sequence of ground truth transcriptions. The method trains the speech recognition model to minimize a word error rate based on the number of word errors identified for each obtained speech recognition hypothesis.

Description

[Technical Field]

[0001] The present disclosure relates to training speech recognition models for long-form speech recognition. [Background technology]

[0002] Modern automatic speech recognition (ASR) systems focus on providing speech recognition of speech that is not only high quality (e.g., low word error rate) but also low latency (e.g., short delay between when a user speaks and when the speech is transcribed). For example, when using a device that implements an ASR system, it is often expected that the ASR system will decode the speech in a streaming manner that is responsive to real time or, in some cases, faster than real time. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present disclosure provides a computer-implemented method for training a speech recognition model to recognize long-form speech. When executed on data processing hardware, the computer-implemented method causes the data processing hardware to perform operations including obtaining a set of training samples. Each training sample in the set of training samples includes a corresponding sequence of speech segments corresponding to a training utterance and a corresponding sequence of ground truth transcriptions for the sequence of speech segments. Each ground truth transcription in the corresponding sequence of ground truth transcriptions includes a start time and an end time of the corresponding speech segment. For each training sample in the set of training samples, the method includes processing the corresponding sequence of speech segments using a speech recognition model to obtain one or more speech recognition hypotheses for the training utterance; identifying, for each speech recognition hypothesis obtained for the training utterance, a respective number of word errors relative to the corresponding sequence of ground truth transcriptions; and training the speech recognition model to minimize a word error rate based on the respective number of word errors identified for each speech recognition hypothesis obtained for the training utterance.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, processing the sequence of speech segments to obtain one or more speech recognition hypotheses includes processing the sequence of speech segments to obtain an N-best list of speech recognition hypotheses, and training the speech recognition model to minimize a word error rate includes training the speech recognition model to minimize a word error rate based on an average of a respective number of word errors identified for each speech recognition hypothesis in the N-best list of speech recognition hypotheses.

[0005] In some examples, each training sample in the set of training samples further includes a respective non-speech segment between each pair of adjacent speech segments in the sequence of speech segments, hi some embodiments, each speech segment in the sequence of speech segments has an average duration of at least 6 seconds.

[0006] In some implementations, the speech recognition model includes a transducer-based speech recognition model having an encoder network, a prediction network, and a joint network. In some examples, the encoder network includes multiple multi-head attention layers.

[0007] In some examples, the prediction network is configured to receive as input, at each of a plurality of time steps following the initial time step, the sequence of non-blank symbols output by the final softmax layer. At each of the plurality of time steps, the prediction network is configured to, for each non-blank symbol in the sequence of non-blank symbols received as input at the corresponding time step, generate an embedding of the corresponding non-blank symbol using a shared embedding matrix, assign a respective position vector to the corresponding non-blank symbol, and weight the embedding in proportion to the similarity between the embedding and the respective position vector. The prediction network is further configured to generate as output, a single embedding vector at the corresponding time step, the single embedding vector being based on a weighted average of the weighted embeddings. In some implementations, the joint network is configured to receive as input, at each of a plurality of time steps following the initial time step, the single embedding vector generated as output from the prediction network at the corresponding time step, and generate a probability distribution over possible speech recognition hypotheses at the corresponding time step. In some examples, the prediction network ties the dimensions of the shared embedding matrix to the dimensions of the output layer of the joint network.

[0008] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining a set of training samples. Each training sample in the set of training samples includes a corresponding sequence of speech segments corresponding to a training utterance and a corresponding sequence of ground truth transcriptions for the sequence of speech segments. Each ground truth transcription in the corresponding sequence of ground truth transcriptions includes a start time and an end time of the corresponding speech segment. For each training sample in the set of training samples, the operations include processing the corresponding sequence of speech segments using a speech recognition model to obtain one or more speech recognition hypotheses for the training utterance; identifying, for each speech recognition hypothesis obtained for the training utterance, a respective number of word errors relative to the corresponding sequence of ground truth transcriptions; and training the speech recognition model to minimize a word error rate based on the respective number of word errors identified for each speech recognition hypothesis obtained for the training utterance.

[0009] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, processing the sequence of speech segments to obtain one or more speech recognition hypotheses includes processing the sequence of speech segments to obtain an N-best list of speech recognition hypotheses, and training the speech recognition model to minimize a word error rate includes training the speech recognition model to minimize a word error rate based on an average of a respective number of word errors identified for each speech recognition hypothesis in the N-best list of speech recognition hypotheses.

[0010] In some examples, each training sample in the set of training samples further includes a respective non-speech segment between each pair of adjacent speech segments in the sequence of speech segments, hi some embodiments, each speech segment in the sequence of speech segments has an average duration of at least 6 seconds.

[0011] In some implementations, the speech recognition model includes a transducer-based speech recognition model having an encoder network, a prediction network, and a joint network. In some examples, the encoder network includes multiple multi-head attention layers.

[0012] In some examples, the prediction network is configured to receive as input, at each of a plurality of time steps following the initial time step, the sequence of non-blank symbols output by the final softmax layer. At each of the plurality of time steps, the prediction network is configured to, for each non-blank symbol in the sequence of non-blank symbols received as input at the corresponding time step, generate an embedding representation of the corresponding non-blank symbol using a shared embedding matrix, assign a respective position vector to the corresponding non-blank symbol, and weight the embedding representation in proportion to the similarity between the embedding representation and the respective position vector. The prediction network is further configured to generate as output, a single embedding vector at the corresponding time step, the single embedding vector being based on a weighted average of the weighted embedding representations. In some implementations, the joint network is configured to receive as input, at each of a plurality of time steps following the initial time step, the single embedding vector generated as output from the prediction network at the corresponding time step, and generate a probability distribution over possible speech recognition hypotheses at the corresponding time step. In some examples, the prediction network ties the dimension of the shared embedding matrix to the dimension of the output layer of the joint network.

[0013] Yet another aspect of the present disclosure provides a computer-implemented method for training a speech recognition model to recognize long-form speech. The method includes obtaining a set of training samples. Each training sample in the set of training samples includes a corresponding sequence of raw speech segments and a corresponding sequence of ground truth transcriptions for the sequence of raw speech segments. Each ground truth transcription in the corresponding sequence of ground truth transcriptions includes a start time and an end time of the corresponding raw speech segment. The method includes concatenating two or more groups of consecutive raw speech segments from the sequence of raw speech segments to form a respective long training utterance for each concatenated group of consecutive raw speech segments. During a first training stage, for each long training utterance, the method includes processing the concatenated group of consecutive raw speech segments using a speech recognition model to predict speech recognition hypotheses for each long training utterance, and training the speech recognition model using a logarithmic loss based on the predicted speech recognition hypotheses and the corresponding group of ground truth transcriptions for each long training utterance.

[0014] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the method further includes, during the second training stage, for each training sample in the set of training samples, processing a corresponding sequence of raw speech segments using the speech recognition model initialized by the first training stage to obtain one or more speech recognition hypotheses, for each speech recognition hypothesis, identifying a respective number of word errors relative to a corresponding sequence of the ground truth transcription, and training the speech recognition model to minimize a word error rate based on the respective number of word errors identified for each speech recognition hypothesis.

[0015] In some examples, processing the sequence of raw speech segments to obtain one or more speech recognition hypotheses includes processing the sequence of raw speech segments to obtain an N-best list of speech recognition hypotheses, and training the speech recognition model to minimize a word error rate includes training the speech recognition model to minimize a word error rate based on an average of the respective numbers of word errors identified for each speech recognition hypothesis in the N-best list of speech recognition hypotheses.

[0016] In some implementations, the speech recognition model includes a transducer-based speech recognition model including an encoder network and a decoder, where the decoder includes a prediction network and a joint network. In some examples, training the speech recognition model to minimize a word error rate during the second training stage includes fine-tuning parameters of the decoder initialized by the first training stage without fine-tuning parameters of the encoder network initialized by the first training stage.

[0017] Yet another aspect of the present disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware that stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining a set of training samples. Each training sample in the set of training samples includes a corresponding sequence of raw speech segments and a corresponding sequence of ground truth transcriptions for the sequence of raw speech segments. Each ground truth transcription in the corresponding sequence of ground truth transcriptions includes a start time and an end time of the corresponding raw speech segment. The operations include concatenating two or more groups of consecutive raw speech segments from the sequence of raw speech segments to form a respective longer training utterance for each concatenated group of consecutive raw speech segments. During the first training stage, for each long training utterance, the operations include processing concatenated groups of consecutive raw speech segments using a speech recognition model to predict speech recognition hypotheses for each long training utterance, and training the speech recognition model using logarithmic loss based on the predicted speech recognition hypotheses and corresponding groups of ground truth transcriptions for each long training utterance.

[0018] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the operations further include, during the second training stage, for each training sample in the set of training samples, processing a corresponding sequence of raw speech segments using the speech recognition model initialized by the first training stage to obtain one or more speech recognition hypotheses, for each speech recognition hypothesis, identifying a respective number of word errors relative to a corresponding sequence of the ground truth transcription, and training the speech recognition model to minimize a word error rate based on the respective number of word errors identified for each speech recognition hypothesis.

[0019] In some examples, processing the sequence of raw speech segments to obtain one or more speech recognition hypotheses includes processing the sequence of raw speech segments to obtain an N-best list of speech recognition hypotheses, and training the speech recognition model to minimize the word error rate includes training the speech recognition model to minimize the word error rate based on an average of the respective numbers of word errors identified for each speech recognition hypothesis in the N-best list of speech recognition hypotheses.

[0020] In some implementations, the speech recognition model includes a transducer-based speech recognition model including an encoder network and a decoder, where the decoder includes a prediction network and a joint network. In some examples, training the speech recognition model to minimize a word error rate during the second training stage includes fine-tuning parameters of the decoder initialized by the first training stage without fine-tuning parameters of the encoder network initialized by the first training stage.

[0021] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a schematic diagram of an exemplary speech environment in which speech is transcribed using a recurrent neural network transducer (RNN-T) model. [Figure 2] FIG. 1 is a schematic diagram of an exemplary RNN-T model architecture. [Figure 3] FIG. 3 is a schematic diagram of an exemplary tied and reduced prediction network of the RNN-T model architecture of FIG. [Figure 4] FIG. 1 illustrates exemplary long-form training utterances. [Figure 5]FIG. 1 is a schematic diagram of an exemplary training process for training a speech recognition model using long-form training utterances. [Figure 6] FIG. 1 is a schematic diagram of an exemplary two-stage process module for training a speech recognition model using long-form training utterances. [Figure 7] 10 is a flowchart of another exemplary arrangement of operations for a computer-implemented method of training a speech recognition model using long-form training utterances. [Figure 8] 10 is a flowchart of another exemplary arrangement of operations for a computer-implemented method of training a speech recognition model for long-form speech recognition. [Figure 9] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0023] Like reference symbols in the various drawings indicate like elements.

[0024] Many voice interaction applications, such as telephony applications, voice-activated digital assistants, and dialogue systems, use automatic speech recognition (ASR) systems. ASR on telephone speech is a key issue for many real-world applications, such as medical conversations and call centers. However, in such applications and uses, ASR systems must accurately recognize long-form speech. Long-form speech includes, for example, utterances having a length of tens of seconds or utterances containing multiple speech segments. Long-form speech recognition remains a challenging ASR task due to the long and conversational characteristics of speech, the presence of non-speech signals in audio data, and noisy channels. Conventional systems are trained using training data representing short utterances, and therefore perform poorly on long-form tasks such as telephone audio. While conventional ASR systems have achieved state-of-the-art performance on numerous large-scale benchmarks, the datasets associated with these benchmarks often represent short utterances with good acoustic conditions and high sound quality. Therefore, when conventional ASR systems perform ASR on long-form tasks such as speech, long and noisy acoustic conditions can pose significant challenges to the generalization ability and robustness of the ASR model.

[0025] Implementations disclosed herein are directed to training an ASR model with training data including long-form training utterances. An ASR model trained as disclosed herein can transcribe long-form speech during inference without first having to segment the incoming long-form speech into shorter segments. Segmenting long-form speech can result in the loss of useful contextual information and the introduction of transcription errors due to incomplete segmentation. While special treatments can mitigate the problems associated with segmentation, such treatments undesirably increase the complexity and latency of the ASR model. During inference, an ASR model trained as disclosed herein can exploit the contextual information present in long-form utterances and eliminate the need for segmentation without increasing the complexity of the ASR system or introducing additional latency. Exemplary training samples include long-form utterances obtained from long audio recordings of phone calls paired with corresponding ground truth transcriptions. In some examples, the long-form training utterances include various combinations of transcribed consecutive live speech segments of audio recordings, each representing a different segment of the audio recording. In some instances, non-voiced segments between voiced segments are retained in the training samples.

[0026] Implementations disclosed herein are directed to a method for training a speech recognition model with a set of training samples to teach the speech recognition model to learn to recognize long-form speech. Each training sample in the set of training samples includes a corresponding sequence of speech segments corresponding to a training utterance and a corresponding sequence of ground truth transcriptions for the sequence of speech segments, where each ground truth transcription in the corresponding sequence of ground truth transcriptions includes a start time and an end time of the corresponding speech segment. For each training sample in the set of training samples, the method includes processing the corresponding sequence of speech segments using the speech recognition model to obtain one or more speech recognition hypotheses for the training utterance, and identifying, for each speech recognition hypothesis obtained for the training utterance, a respective number of word errors relative to the corresponding sequence of ground truth transcriptions. The method further includes training the speech recognition model to minimize a word error rate based on the respective number of word errors identified for each speech recognition hypothesis obtained for the training utterance.

[0027] 1 is a schematic diagram of an example speech environment 100. In speech environment 100, a user 104 may interact with a computing device, such as user device 10, via voice input. User device 10 (also commonly referred to as device 10) is configured to capture speech (e.g., streaming audio data) from one or more users 104 in speech environment 100. Here, streaming audio data may refer to utterances 106 by users 104 that serve as audible queries, commands to device 10, or audible communications captured by device 10. A voice-enabled system of device 10 may appropriately process the queries or commands by answering the queries and / or having the commands executed / completed by one or more downstream applications.

[0028] The user device 10 may correspond to any computing device associated with a user and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, a mobile device (e.g., a mobile phone, a tablet, a laptop, etc.), a computer, a wearable device (e.g., a smart watch), a smart appliance, an Internet of Things (IoT) device, an in-vehicle infotainment system, a smart display, a smart speaker, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 in communication with the data processing hardware 12, storing instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 having audio capture devices (e.g., microphones) 16, 16a for capturing and converting speech 106 in the speech environment 100 into electrical signals, and speech output devices (e.g., speakers) 16, 16b for communicating audible audio signals (e.g., as output audio data from the device 10). Although the user device 10 implements a single audio capture device 16a in the illustrated example, it may implement an array of audio capture devices 16a without departing from the scope of this disclosure, and one or more capture devices 16a in the array may not be physically present on the user device 10 but may communicate with the audio system 16.

[0029] In the speech environment 100, an automatic speech recognition (ASR) system 118 implementing a recurrent neural network transducer (RNN-T) model 200 and an optional rescorer 180 resides on the user device 10 of the user 104 and / or on a remote computing device 70 (e.g., one or more servers of a distributed system running in a cloud computing environment) that communicates with the user device 10 via the network 40. The user device 10 and / or the remote computing device 70 also include an audio subsystem 108 configured to receive utterances 106 spoken by the user 104 and captured by the audio capture device 16a and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the ASR system 118. In the illustrated example, the user utters each utterance 106, and the audio subsystem 108 converts the utterances 106 into corresponding audio data (e.g., acoustic frames) 110 for input to the ASR system 118. The RNN-T model 200 then receives as input audio data 110 corresponding to the utterance 106 and generates / predicts as output a corresponding transcription 120 (e.g., a recognition result / hypothesis) of the utterance 106. In the illustrated example, the RNN-T model 200 may perform streaming speech recognition to generate initial speech recognition results 120, 120a, and the rescorer 180 may update (rescore) the initial speech recognition result 120a to generate final speech recognition results 120, 120b. The server 70 includes data processing hardware 72 and memory hardware 74 in communication with the data processing hardware 72. The memory hardware 74 stores instructions that, when executed by the data processing hardware 72, cause the data processing hardware 72 to perform one or more operations, such as those disclosed herein.

[0030] The user device 10 and / or the remote computing device 70 also execute a user interface generator 107 configured to present a representation of a transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 may stream initial speech recognition results 120a for time 1 and then display final speech recognition results 120b for time 2. In some configurations, the transcription 120 output from the ASR system 118 is processed by a natural language processing / understanding (NLP / NLU) module executing, for example, on the user device 10 or the remote computing device 70 to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device 10 or the remote computing device 70) may convert the transcription into synthesized speech for audible output by the user device 10 and / or another device.

[0031] In the illustrated example, a user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) on a user device 10 that uses an ASR system 118. For example, FIG. 1 shows a user 104 communicating with the digital assistant application 50 and the digital assistant application 50 displaying a digital assistant interface 18 on the screen of the user device 10, illustrating a conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, "What time does the concert start tonight?" This question from the user 104 is utterance 106 that is captured by an audio capture device 16a and processed by an audio system 16 of the user device 10. In this example, the audio system 16 receives the utterance 106 and converts it into acoustic frames 110 for input to the ASR system 118.

[0032] Continuing with this example, as the RNN-T model 200 receives acoustic frames 110 corresponding to an utterance 106 uttered by the user 104, it encodes the acoustic frames 110 and then decodes the encoded acoustic frames 110 into initial speech recognition results 120a. During time 1, the user interface generator 107 presents a representation of the initial speech recognition results 120a of the utterance 106 to the user 104 of the user device 10 in a streaming manner via the digital assistant interface 18, such that words, word pieces, and / or individual characters appear on the screen immediately after they are spoken. In some examples, the first look-ahead audio context is equal to zero.

[0033] During time 2, user interface generator 107 presents a representation of final speech recognition result 120b of utterance 106 to user 104 of user device 10 via digital assistant interface 18, to be rescored by rescorer 180. In some implementations, user interface generator 107 replaces the representation of initial speech recognition result 120a presented at time 1 with the representation of final speech recognition result 120b presented at time 2, where time 1 and time 2 may include timestamps corresponding to when user interface generator 107 presents the respective speech recognition results. In this example, the timestamp for time 1 indicates that user interface generator 107 presents initial speech recognition result 120a at an earlier time than final speech recognition result 120b. For example, because the final speech recognition result 120b is estimated to be more accurate than the initial speech recognition result 120a, the user interface generator 107 ultimately displays the final speech recognition result 120b as the transcription 120 and corrects any words that may have been misrecognized in the initial speech recognition result 120a. In this example, the streaming initial speech recognition result 120a output by the RNN-T model 200 and displayed on the screen of the user device 10 at Time 1 involves low latency and provides the user 104 with a response that their query is being processed, while the final speech recognition result 120b output by the rescorer 180 and displayed on the screen at Time 2 utilizes additional speech recognition and / or language models to improve speech recognition quality in terms of accuracy but increase latency. However, because the initial speech recognition result 120a is displayed as the user utters the utterance 106, the higher latency associated with generating and ultimately displaying the final recognition result is not noticeable to the user 104.

[0034] In the example shown in FIG. 1 , digital assistant application 50 may respond to a question posted by user 104 by using natural language processing. Natural language processing generally refers to the process of interpreting written language (e.g., initial speech recognition result 120a and / or final speech recognition result 120b) and determining whether the written language prompts an action. In this example, digital assistant application 50 uses NLP / NLU to recognize that the question from user 104 is about the user's schedule, more specifically, a concert on the user's schedule. By recognizing these details through NLP / NLU, the automated assistant returns a response 19 to the user's query, which in this case states, "Doors open at 7:30 PM, and the concert starts at 9 PM." In some configurations, NLP / NLU occurs on a remote server 70 in communication with data processing hardware 12 of user device 10.

[0035] 2 is a schematic diagram of an exemplary recurrent neural network transducer model 200 (i.e., RNN-T model 200) that is trained using long-form training utterances to improve speech recognition of long-form utterances during inference. The RNN-T model 200 offers a small computational footprint and utilizes lower memory requirements than traditional ASR architectures, making the RNN-T model 200 suitable for performing speech recognition entirely on the user device 10 (e.g., no communication with a remote server is required).

[0036] As shown, the RNN-T model 200 includes an encoder network 210, a prediction network 300, a joint network 220, and a final softmax layer 230. The prediction network 300 and the joint network 220 collectively constitute an RNN-T decoder. The encoder network 210 is generally similar to an acoustic model (AM) in a conventional ASR system and includes a recurrent network of stacked long short-term memory (LSTM) layers. For example, the encoder receives a sequence of d-dimensional feature vectors (e.g., acoustic frames 110 (FIG. 1)) x = (x1, x2, ..., x T ), where

number

number

[0037] Similarly, the prediction network 300 is also an LSTM network, and like the language model (LM), it computes the sequence of non-blank symbols 232 output by the final softmax layer 230 so far, y 0 , ..., y ui-1 tightly coupled or hidden representation

number

number

[0038] The joint network 300 uses the high-level feature representation generated by the encoder network 210.

number

number

number

[0039] The final softmax layer 230 calculates the probability distribution Z i 232 and selects the output label / symbol with the highest probability of generating the transcription. The final softmax layer 230 uses any technique to generate the distribution Z i232. In this way, the RNN-T model 200 does not make a conditional independence assumption, but rather selects the output label / symbol with the highest probability for each symbol y u The prediction of 232 is based not only on the acoustics but also on the labels output up to that point. ui-n , …, y ui-1 The RNN-T model 200 does not assume that the output symbols 232 are independent of future acoustic frames 110, which allows the RNN-T model 200 to be used in a streaming manner.

[0040] Because the RNN-T model 200 determines the initial speech recognition result 120a in a streaming manner and determines the final speech recognition result 120b using previous non-blank symbols from the initial speech recognition result 120a, the final speech recognition result 120b is presumed to be more accurate than the initial speech recognition result 120a. That is, the final speech recognition result 120b takes into account previous non-blank symbols, and therefore, the final speech recognition result 120b is presumed to be more accurate because the initial speech recognition result 120a does not take into account any previous non-blank symbols. Furthermore, the rescorer 180 (FIG. 1) may update the initial speech recognition result 120a with the final speech recognition result 120b and provide the transcription to the user 104 via the user interface generator 170.

[0041] In some examples, the encoder network 210 of the RNN-T model 200 includes eight 2048-dimensional LSTM layers, each followed by a 740-dimensional projection layer. In other embodiments, the encoder network 210 includes multiple multi-head attention layers. For example, the multiple multi-head attention layers may include a conformer layer or a transformer layer network. The prediction network 220 may have two 2048-dimensional LSTM layers, each followed by a 740-dimensional projection layer and a 128-unit embedding layer. Finally, the joint network 220 may also have 740 hidden units. The softmax layer 230 may be composed of a unified word piece or grapheme set generated using all unique word pieces or graphemes in the training data. When the output symbols / labels include word pieces, the set of output symbols / labels may include 4096 distinct word pieces. When the output symbols / labels contain graphemes, the set of output symbols / labels may contain fewer than 100 different graphemes.

[0042] 3 is a schematic diagram of an exemplary prediction network 300 for the RNN-T model 200. The prediction network 300 is based on the non-blank symbols 232a-232n y , which are limited to the N previous non-blank symbols 232a-232n output by the final softmax layer 230. ui-n , ..., y ui-1As input, the prediction network 300 receives a sequence of N symbols 232a-232n. In some examples, N is equal to 2. In other examples, N is equal to 5, although this disclosure is non-limiting and N may be equal to any integer. The sequence of non-blank symbols 232a-232n represents the initial speech recognition result 120a (FIG. 1). In some implementations, the prediction network 300 includes a multi-head attention mechanism 302, which shares a shared embedding matrix 304 across each of its heads 302A-302H. In one example, the multi-head attention mechanism 302 includes four heads. However, any number of heads may be used by the multi-head attention mechanism 302. Notably, the multi-head attention mechanism significantly improves performance with minimal increase in model size. As explained in more detail below, each head 302A-302H contains its own row of position vectors, and instead of increasing the model size by concatenating the outputs 318A-318H from all heads, the outputs 318A-318H are averaged by a head averaging module 322.

[0043] Referring to the first head 302A of the multi-head attention mechanism 302, the head 302A uses a shared embedding matrix 304 to generate a set of non-blank symbols 232a-232n y received as input at corresponding time steps from the plurality of time steps. ui-n , ..., y ui-1 For each non-blank symbol in the sequence, a corresponding embedded representation 306, 306a-306n (e.g.,

number

number

[0044] Non-blank symbols 232a-232n, y ui-n , ..., y ui-1 The corresponding embedding representation generated by the shared embedding matrix 304 for each non-blank symbol in the sequence is the same for all of the heads 302A-302H of the multi-head attention mechanism 302, but each head 302A-302H defines a different set / row of position vectors 308. For example, the first head 302A defines a position vector PV Aa~An The second head 302B defines a row of 308Aa to 308An, and the second head 302B defines a position vector PV Ba~Bn 308 Ba~Bn The Hth head 302H defines a position vector PV Ha~Hn 308 Ha~Hn define another different row of

[0045] For each non-blank symbol in the received sequence of non-blank symbols 232a-232n, the first head 302A also weights the corresponding embedded representation 306 via the weighting layer 310 in proportion to the similarity between the corresponding embedded representation and its assigned respective position vector PV 308. In some examples, the similarity may include cosine similarity (e.g., cosine distance). In the illustrated example, the weighting layer 310 outputs a sequence of weighted embedded representations 312, 312Aa-312An, each associated with a corresponding embedded representation 306 weighted in proportion to its assigned respective position vector PV 308. In other words, the weighted embedded representation 312 output by the weighting layer 310 for each embedded representation 306 may correspond to a dot product between the embedded representation 306 and its assigned respective position vector PV 308. The weighted embedded representation 312 may be interpreted as an attenuation for the embedded representation proportional to how similar it is to the position associated with its assigned respective position vector PV 308. To increase computational speed, the prediction network 300 includes a non-recurrent layer, so that the sequence of weighted embeddings 312Aa-312An are not concatenated but instead averaged by a weighted average module 316 to produce a weighted average 318A of the weighted embeddings 312Aa-312An as output from the first head 302A, expressed as follows:

number

[0046] The operations described above with respect to the first head 302A are similarly performed by each of the other heads 302B-302H of the multi-head attention mechanism 302. Due to the different sets of position vectors PV 308 defined by each head 302, the weighting layer 310 outputs a sequence of weighted embedding representations 312Ba-312Bn, 312Ha-312Hn for each of the other heads 302B-302H that differs from the sequence of weighted embedding representations 312Aa-312An for the first head 302A. The weighted average module 316 then generates a weighted average 318A-318H of the corresponding weighted embedding representations 312 of the sequences of non-blank symbols as output from each of the other corresponding heads 302B-302H.

[0047] In the illustrated example, the prediction network 300 includes a head averaging module 322 that averages weighted averages 318A-318H output from corresponding heads 302A-302H. A projection layer 326 with SWISH may receive as input an output 324 from the head averaging module 322 corresponding to the average of the weighted averages 318A-318H and generate as output a projected output 328. A final layer normalization 330 normalizes the projected output 328 to produce a single embedding vector at corresponding time steps from multiple time steps.

number

number

[0048] In some configurations, the prediction network 300 does not implement a multi-head attention mechanism 302, and performs only the operations described above for the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-312An is simply passed through the projection layer 326 and layer normalization 330 to produce a single embedding vector

number

[0049] In some implementations, parameter tying between the prediction network 300 and the joint network 220 is applied to further reduce the size of the RNN-T decoder, i.e., the prediction network 300 and the joint network 220. Specifically, the vocabulary size |V| and the embedding dimension d e , the shared embedding matrix 304 in the prediction network is

number

number

[0050] Referring again to FIG. 1 , the training process 500 trains the RNN-T model 200 with a set of training samples 400, 400a-400n. In some examples, the training sample generator 150 generates the training samples 400 by concatenating a group of continuous speech segments 405, each paired with a corresponding ground truth transcription 415. Each training sample 400 corresponds to a training utterance that includes a corresponding sequence of speech segments x 405 and a corresponding sequence of ground truth transcriptions 415. Each speech segment 405 may correspond to speech captured during streaming audio by an audio capture device and stored along with the speech segment's corresponding transcription. All speech segments in the concatenated sequence that form the corresponding training sample 400 may be captured by the same audio capture device and / or spoken by the same speaker, although the disclosure is not so limited. For example, at least one speech segment in the concatenated sequence forming the corresponding training sample 400 may have been captured by a different audio capture device than the other speech segments in the training sample 400 and / or may have been spoken by a different speaker than the speakers who spoke the other speech segments in the training sample 400.

[0051] In some examples, the training sample generator 150 is further configured to vary acoustic diversity between corresponding sequences of speech segments 405 in the training samples 400 used to train the RNN-T model 200. For example, the training sample generator 150 may apply a data augmentation technique including at least one of adding / injecting noise, adding reverberation, and / or manipulating the timing (e.g., stretching) of the speech segment x 405. In some cases, the data augmentation technique may include time warping. Another data augmentation technique includes injecting various environmental noises into the speech segment x 405 using multi-style training (MTR).

[0052] 4 illustrates exemplary training samples 400, 400a-400b, each including a corresponding sequence of raw speech segments x 405a-405c, 405d-405e. As shown, adjacent raw speech segments 405 may be separated by a non-speech segment 425. While the illustrated example shows a first training sample 400a including a sequence of three speech segments 405a-405c and a second training sample 400b including a sequence of two speech segments 405d-405e, a training sample may include dozens of speech segments 405. Each raw speech segment 405 is trained based on a respective ground truth transcription y *415. Thus, each ground truth transcription y* 415, 415a-415e includes a start time 414 and an end time 416 of the corresponding speech segment 405. In some examples, the average duration of the speech segments 420 is at least 6 seconds. The training sample generator 150 generates each training sample 410 associated with a training utterance by merging, concatenating, or otherwise combining various combinations of consecutive raw speech segments 405 representing different divisions of speech to form each training sample 400. In particular, the training sample generator 150 can generate training samples 400 associated with training utterances of different lengths (e.g., raw, short, medium, and long) by selecting how many raw speech segments 405 are spliced ​​(e.g., one, two, ..., N) to form the corresponding training utterance. A single speech segment 405 may be included in more than one of the training samples 400. In some examples, the training sample generator 150 uses the start and end times of the audio segments 405 to determine which and how many audio segments 405 to combine / concatenate to form a corresponding training utterance of a particular length.

[0053] Referring again to FIG. 1 , a training process 500 trains the RNN-T model 200 on a speech recognition performance task. In some examples, the training process 500 includes a logarithmic loss metric L u For a particular training sample 400, an exemplary log-loss metric / term is the logarithmic loss metric / term obtained by multiplying the ground truth transcription y conditioned on the corresponding sequence of audio segments 405: * The probability Z for the corresponding sequence of 415 i This logarithmic loss metric L u By reducing *415, i.e., the probability Z for the sequence of speech segments 405. i The RNN-T model 200 is trained to increase the negative logarithm of the probability L u may be expressed as follows: L u = - log Pr (y * |x) (2)

[0054] Additionally or alternatively, the training process 500 may include a minimum word error rate (MWER) loss L mwer The RNN-T model 200 may be trained to reduce MWER. When minimizing MWER, the training process 500 trains the RNN-T model 200 to increase the respective probability / confidence for each of one or more speech recognition hypotheses 522 (FIGS. 5 and 6) (e.g., an N-best list of hypotheses) for training utterances with fewer word errors and decrease the respective probability / confidence for each of one or more speech recognition hypotheses that are, on average, worse. For example, the training process 500 may train the RNN-T model 200 to increase the respective probability / confidence for each of the ground truth transcriptions y for the training samples 400. * The word error l(y,y) for each speech recognition hypothesis 522 output by the RNN-T model for the corresponding sequence of speech segments 405 of the training samples 400 relative to the corresponding sequence of 415. * The training process 500 then calculates the number of word errors l(y,y * ) average number of

number

number

number

number

number

number

number

number

[0055] In some implementations, the training process 500a may include a logarithmic loss L u and MWER loss L mwerFor example, the training process 500a trains the RNN-T model 200 to reduce both the logarithmic loss L u 537 and MWER loss L mwer Loss L, which is a combination of 535 combo The RNN-T model 200 may be trained to reduce the logarithmic loss L. u and MWER loss L mwer and may be expressed as: L combo = L mwer + λL ll ... (6) In the above equation, λ is the logarithmic loss L for training the RNN-T model 200. u and MWER loss L mwer is a parameter selected by the training process 500 to control the relative contribution of the logarithmic loss L u and MWER loss L mwer Other combinations of may be used.

[0056] MWER loss L mwer 535-based training yields good speech recognition performance for long-form utterances, but the MWER loss L mwer Training based on can be computationally expensive. Furthermore, during training, the logarithmic loss L u MWER loss L compared to using mwer The benefit of using logarithmic loss L decreases as the length of the training utterances 410 increases. Therefore, alternatively, the training process 500 trains the RNN-T model 200 using a two-stage training process, as described in more detail below with reference to FIG. 6. In the first training stage, the training process 500 applies a logarithmic loss L to the longer long-form training utterances 410 to reduce training complexity. u In a second training stage, starting with the best-performing RNN-T model 200 from the first training stage, the training process 500 trains the RNN-T model 200 using the MWER loss L for shorter long-form training utterances 410 (e.g., raw utterances containing only a single speech segment 420).u We fine-tune the RNN-T model 200 based on the MWER loss L in the single-stage training process. mwer In some examples, the training process 500 updates the parameters of the encoder network 210 and the decoder network 300 during the first training stage, and does not update the parameters of the encoder network 210 during the second training stage.

[0057] 5 is a schematic diagram of an example training process 500, 500a for training a speech recognition model 200 to minimize a word error rate (WER) on a set of training samples 400 associated with long-form training utterances. The speech recognition model 200 may include the RNN-T model architecture of FIG. 2, including an encoder 210 and a decoder 520, which collectively include the prediction network 300 and the joint network 220. The training process 500a may be performed on a remote system 70 (i.e., on data processing hardware 72) or on the user device 10 (i.e., on data processing hardware 12). For each training sample 400 in the set of training samples 400 generated by the training sample generator 150 (FIG. 1), the training process 500a processes a corresponding sequence of speech segments 405 using the RNN-T model 200 to obtain one or more speech recognition hypotheses 522, 522a-522n for the training utterance.

[0058] Thereafter, for each training sample 400, a loss term module 530 receives one or more speech recognition hypotheses 522 output by the RNN-T model 200 for the corresponding training utterance and identifies a respective number of word errors for each speech recognition hypothesis 522 relative to a corresponding sequence of the ground truth transcription 415. Here, the loss term module 530 may determine (i.e., using equation (5)) an MWER loss term 535 that represents the average number of word errors in the one or more speech recognition hypotheses 522. In some examples, the loss term module 300 receives an N-best list of speech recognition hypotheses 522 that correspond to the top N speech recognition hypotheses 522 of the beam search for the corresponding training utterance. Thus, the loss term module 530 may apply equation (5) to determine / calculate an MWER loss term 535 that indicates the average number of word errors in the top N hypotheses 522 of the beam search. Based on the MWER loss term 535 output by the loss term module 530 for each training sample 400, the training process 500a trains the RNN-T model 200 to minimize the MWER based on the respective number of word errors identified for each speech recognition hypothesis obtained for the corresponding training utterance.

[0059] In some implementations, in addition to or instead of calculating an MWER loss term 535 indicating the average number of word errors in the top-N hypotheses 522 output by the RNN-T model 200 for each training sample 400, the training process 500a also calculates a ground truth transcription y for each training sample 400. * Calculate the logarithmic loss term 537 (i.e., using equation (2)) which represents the negative of the logarithm of the probability 222 for the corresponding sequence of 415. Here, the training process 500 may derive the probability for any (Pr(y|x)) by eliminating all possible alignments using a forward-backward algorithm.

[0060] 6 is a schematic diagram of an exemplary two-stage training process 500, 500b for training the RNN-T model 200. The training process 500b obtains a set of training samples 400, each including a corresponding sequence of raw speech segments 405 and a corresponding sequence of ground truth transcriptions 415 for the sequence of raw speech segments 405. Each ground truth transcription in the corresponding sequence of ground truth transcriptions 415 may include a start time 414 and an end time 416 of the corresponding raw speech segment 405.

[0061] During the first stage of the two-stage training process 500b, a concatenator 610 concatenates two or more groups of consecutive raw speech segments 405 from the sequence of raw speech segments 405 to form a respective long training utterance for each concatenated group 405G of consecutive raw speech segments 405, where each concatenated group 405G of consecutive raw speech segments 405 is paired with a corresponding group in the sequence of ground truth transcriptions 415. For each long training utterance, the first stage of the two-stage training process processes the concatenated group 405G of consecutive raw speech segments using the speech recognition model 200 to predict a speech recognition hypothesis 232 (represented by the probabilities 222 output from the decoder 520) for each long training utterance. A logarithmic loss term module 630 then computes a logarithmic loss term between the predicted speech recognition hypothesis 232 for each long training utterance and the ground truth transcription y * Based on the corresponding group of 415, (i.e., using Eq. (2)) the logarithmic loss term L ll 537. The first stage of the two-stage training process 500b trains the speech recognition model 200 using the logarithmic loss term 537 to initialize the speech recognition model 200.

[0062] During the second stage, for each training sample 400 in the set of training samples 400 generated by the training sample generator 150 (FIG. 1), the training process 500b processes the corresponding sequence of speech segments 405 using the RNN-T model 200 initialized by the first training stage to obtain one or more speech recognition hypotheses 522, 522a-522n. The MWER loss term module 640 then determines the respective number of word errors for each speech recognition hypothesis 522 relative to the corresponding sequence of the ground truth transcription 415. Here, the MWER loss term module 640 may determine (i.e., using Equation (5)) an MWER loss term 535 that represents the average number of word errors in the one or more speech recognition hypotheses 522. In some examples, the MWER loss term module 640 receives an N-best list of speech recognition hypotheses 522 that correspond to the top N speech recognition hypotheses 522 of the beam search for the corresponding training utterance. Thus, the MWER loss term module 640 may apply equation (5) to determine / calculate an MWER loss term 535 indicating the average number of word errors in the top-N hypotheses 522 of the beam search. Based on the MWER loss term 535 output by the MWER loss term module 640 for each training sample 400, the training process 500b trains the RNN-T model 200 to minimize the MWER based on the respective number of word errors identified for each speech recognition hypothesis obtained for the corresponding training utterance. In some examples, the second training stage of the training process 500b updates the parameters of the decoder 520 based on the MWER loss term 535 while keeping the parameters of the encoder 210 fixed.

[0063] 7 is a flowchart of an example arrangement of operations for a computer-implemented method 700 for training a speech recognition model 200 (e.g., an RNN-T model 200). At operation 702, the method 700 includes obtaining a set of training samples 400. Each training sample 400 in the set of training samples 400 includes a corresponding sequence of speech segments 405 corresponding to a training utterance and one or more ground truth transcriptions y for each speech segment in the sequence of speech segments 405. * 415 and a corresponding sequence of one or more ground truth transcriptions y* 415. * 415 includes the start time 414 and end time 416 of the corresponding audio segment 405 .

[0064] For each training sample 400 in the set of training samples, the method 700 performs operations 704, 706, and 708. In operation 704, the method 700 includes processing the corresponding sequence of speech segments 405 using the speech recognition model 200 to obtain one or more speech recognition hypotheses 522 for the training utterance. In operation 706, the method 700 performs one or more ground truth transcriptions y for each speech recognition hypothesis 522 obtained for the training utterance 410. * The word error l(y * ,y). At operation 708, method 700 includes training speech recognition model 200 to minimize WER based on the respective number of word errors l(y*,y) identified for each speech recognition hypothesis 522 obtained for training utterance 410. For example, method 700 may train model 200 using MWER loss term 535 calculated using Equation 5.

[0065] 8 is a flowchart of another example arrangement of operations for a computer-implemented method 800 for training a speech recognition model 200 (e.g., an RNN-T model 200) using long training utterances. At operation 802, the method 800 includes obtaining a set of training samples 400. Each training sample 400 in the set of training samples 400 includes a corresponding sequence of one or more raw speech segments 405 and a ground truth transcription y for the sequence of raw speech segments 405. * 415 and the corresponding sequence of ground truth transcriptions y* 415. * 415 includes the start time 414 and end time 416 of the corresponding live audio segment.

[0066] At operation 804, the method 800 includes concatenating two or more groups of consecutive raw speech segments 405G from the sequence of raw speech segments 405 to form a respective long training utterance for each concatenated group of consecutive raw speech segments 405G.

[0067] During the first training stage, for each long training utterance, the method 800 performs operations 806 and 808. In operation 806, the method 800 includes processing the concatenated group of consecutive raw speech segments 405G using the speech recognition model 200 to predict speech recognition hypotheses 232 for each long training utterance 614.

[0068] At operation 808, the method 800 generates a predicted speech recognition hypothesis 232 and a ground truth transcription y for each long training utterance. * Based on the corresponding groups of 415, the logarithmic loss L ll 535 to train the speech recognition model 200.

[0069] During an optional second training stage, indicated by the dotted lines, method 800 optionally performs operations 810, 812, and 814. For each short training sample 400 in the set of short training samples 400, in operation 810, method 800 includes processing the corresponding sequence of raw speech segments 405 using the speech recognition model 200 initialized by the first training stage to obtain one or more speech recognition hypotheses 522 (e.g., an N-best list of hypotheses 522, each corresponding to a respective candidate transcription for the utterance). Then, in operation 812, method 800 performs a ground truth transcription y * The word error l(y * , y) for each speech recognition hypothesis 522. In operation 814, the method 800 determines the number of word errors l(y * , y), training the speech recognition model 200 to minimize the word error rate based on the respective numbers of y and y.

[0070] 9 is a schematic diagram of an exemplary computing device 900 that may be used to implement the systems and methods described herein. Computing device 900 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality are merely exemplary and do not limit the implementation of the inventions described and / or claimed herein.

[0071] Computing device 900 includes a processor 910 (i.e., data processing hardware) that may be used to implement data processing hardware 12 and / or 62, a memory 920 (i.e., memory hardware) that may be used to implement memory hardware 14 and / or 64, a storage device 930 (i.e., memory hardware) that may be used to implement memory hardware 14 and / or 64, a high-speed interface / controller 940 that connects to memory 920 and a high-speed expansion port 950, and a low-speed interface / controller 960 that connects to a low-speed bus 970 and storage device 930. Each of the components 910, 920, 930, 940, 950, and 960 are interconnected using various buses and may be mounted on a common motherboard or mounted in any other manner as appropriate. The processor 910 processes instructions executed within the computing device 900, including instructions stored in memory 920 or on storage device 930, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 980 coupled to a high-speed interface 940. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memory, as appropriate. Multiple computing devices 900 may also be connected (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system), with each device providing a portion of the required operations.

[0072] The memory 920 stores information non-transiently within the computing device 900. The memory 920 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 920 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 900. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disk or tape.

[0073] The storage device 930 can provide mass storage for the computing device 900. In some implementations, the storage device 930 is a computer-readable medium. In various different implementations, the storage device 930 may be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory, or other similar solid-state memory device, or a device in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 920, the storage device 930, or memory on the processor 910.

[0074] The high-speed controller 940 manages bandwidth-intensive operations for the computing device 900, and the low-speed controller 960 manages lower-bandwidth intensive operations. Such duty allocation is merely exemplary. In some implementations, the high-speed controller 940 is coupled to the memory 920, coupled to the display 980 (e.g., via a graphics processor or accelerator), and coupled to a high-speed expansion port 950 that may accept various expansion cards (not shown). In some implementations, the low-speed controller 960 is coupled to the storage device 930 and to a low-speed expansion port 990. The low-speed expansion port 990 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., via a network adapter.

[0075] Computing device 900 may be implemented in several different forms, as shown, For example, computing device 900 may be implemented as a standard server 900a, or multiple times in a group of such servers 900a, or as a laptop computer 900b, or as part of a rack server system 900c.

[0076] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, and which may be coupled to receive and transmit data and instructions from a storage system, at least one input device, and at least one output device.

[0077] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0078] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0079] The processes and logic flows described herein can be performed by one or more programmable processors, also known as data processing hardware, which execute one or more computer programs to perform functions by processing input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuitry, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random-access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more storage devices, such as magnetic, magneto-optical, or optical disks, for storing data and is operably coupled to receive data from, transfer data to, or both of these devices. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0080] To enable user interaction, one or more aspects of the present disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display), monitor, or touch screen, for displaying information to the user, and possibly a keyboard and pointing device, e.g., a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to enable user interaction; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can send and receive documents to and from devices used by the user, and interact with the user by, for example, sending web pages to a web browser on the user's client device in response to a request received from the web browser.

[0081] Unless expressly stated to the contrary, "or" refers to an inclusive or, not an exclusive or. For example, "A, B, or C" refers to any combination or subset of A, B, or C, such as (1) A only, (2) B only, (3) C only, (4) A and B, (5) A and C, (6) B and C, or (7) A, B, and C. Similarly, the phrase "at least one of A or B" is intended to refer to any combination or subset of A and B, such as (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Furthermore, the phrase "at least one of A and B" is intended to refer to any combination or subset of A and B, such as (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0082] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0083] 10 User Devices 12 Data Processing Hardware 14 Memory Hardware 16, 16a Audio Capture Device 16, 16b Audio output device 19 responses 40 Network 50 Digital Assistant Applications 70 Remote Computing Devices, Remote Systems 72 Data Processing Hardware 100 Speaking Environment 104 users 106 utterances 107 User Interface Generator 108 Audio Subsystem 110 Acoustic Frame 118 ASR System 120 Transcription 120a, 120b Voice recognition results 150 Training Sample Generator 170 User Interface Generator 180 Lisco Ara 200 RNN-T model 210 Encoder Network 220 Joint Network 222 Probability Z i 230 Final softmax layer 232 Speech Recognition Hypothesis 300 Prediction Network, Decoder Network 232, 232a-232n non-blank symbols 232 Speech Recognition Hypothesis 302 Multi-head Attention Mechanism 302A~302H Head 302B~302H Other heads 304 Shared Embedding Matrix 306, 306a-306n Embedded Representation 308, 308Aa~308An position vectors 310 Weight Layer 312, 312Aa~312An Weighted embedding representation 316 Weighted Average Module 318A~318H Output 322 head average module 324 output 326 Projection layer 328 projected output 400, 400a~400n training samples 405 Audio Segments 405G Concatenated Group 410 Training Utterances 414 Start Time 415 Ground Truth Transcription * 416 End Time 420 audio segments 500, 500a, 500b Training process 520 decoder 522, 522a-522n Speech recognition hypothesis 530 Loss Term Module 535 MWER Loss Lmwer 537 Logarithmic Loss L u 610 Concatenator 614 Long training utterances 630 Logarithmic Loss Term Module 640 MWER loss term module 700, 800 Computer implementation method 900 computing devices 900a Server 900b laptop computer 900c Rack Server System 910 processor 920 memory 930 Storage Devices 940 High-Speed ​​Interface / Controller 950 High-Speed ​​Expansion Port 960 Low-Speed ​​Interface / Controller 970 Slow Bus 980 display 990 Low-Speed ​​Expansion Port

Claims

1. 1. A computer-implemented method (700) for training a speech recognition model (200) to recognize long-form speech, which, when executed on data processing hardware (910), causes the data processing hardware (910) to perform operations, including: obtaining a set of training samples (400), each training sample (400) in the set of training samples (400) including a corresponding sequence of a plurality of speech segments (405) corresponding to a training utterance and a corresponding sequence of a plurality of ground truth transcriptions (415) for the sequence of the plurality of speech segments (405), each pair of adjacent speech segments (405) in the sequence of the plurality of speech segments (405) being separated by a non-speech segment, and each ground truth transcription (415) in a corresponding sequence of speech segments (405) includes a start time (414) and an end time (416) of each corresponding speech segment (405) separated by the non-speech segments, and each training sample (400) is generated by concatenating a group of speech segments (405) each paired with a corresponding ground truth transcription (415) to form a training utterance of a particular length using the start time (414) and end time (416); For each training sample (400) in the set of training samples (400), processing the corresponding sequences of the plurality of speech segments (405) using the speech recognition model (200) to obtain one or more speech recognition hypotheses (522) for the training utterance; for each speech recognition hypothesis (522) obtained for the training utterance, determining a respective number of word errors relative to a corresponding sequence of the plurality of ground truth transcriptions (415); training the speech recognition model (200) to minimize a word error rate based on the respective number of word errors identified for each speech recognition hypothesis (522) obtained for the training utterance; A computer-implemented method (700) comprising:

2. processing the sequence of speech segments (405) to obtain one or more speech recognition hypotheses (522) includes processing the sequence of speech segments (405) to obtain an N-best list of speech recognition hypotheses (522); 2. The computer-implemented method of claim 1, wherein training the speech recognition model to minimize a word error rate comprises training the speech recognition model to minimize a word error rate based on an average of the respective numbers of word errors identified for each speech recognition hypothesis in an N-best list of the speech recognition hypotheses.

3. 10. The computer-implemented method of claim 1, wherein each audio segment in the sequence of audio segments has an average duration of at least six seconds.

4. 10. The computer-implemented method of claim 1, wherein the speech recognition model comprises a transducer-based speech recognition model having an encoder network, a prediction network, and a joint network.

5. 5. The computer-implemented method of claim 4, wherein the encoder network comprises multiple multi-head attention layers.

6. The prediction network (300) performs, at each of a plurality of time steps following an initial time step: receiving as input the sequence of non-blank symbols (232) output by the final softmax layer (230); For each non-blank symbol (232) in the sequence of non-blank symbols (232) received as input at the corresponding time step: generating embedded representations (306) of the corresponding non-blank symbols (232) using the shared embedding matrix (304); assigning respective position vectors (308) to the corresponding non-blank symbols (232); weighting the embedded representations (306) in proportion to the similarity between the embedded representations (306) and the respective position vectors (308); generating as output a single embedding vector (350) for the corresponding time step, the single embedding vector (350) being based on a weighted average (324) of the weighted embedding representations (318); The joint network (320) performs, at each of the plurality of time steps following the initial time step: receiving as input the single embedding vector (350) generated as output from the prediction network (300) at the corresponding time step; generating a probability distribution (222) over the possible speech recognition hypotheses (522) at the corresponding time step.

7. 7. The computer-implemented method of claim 6, wherein the prediction network ties the dimensions of the shared embedding matrix to the dimensions of an output layer of the joint network.

8. Data processing hardware (910); and memory hardware (920) in communication with the data processing hardware (910) and storing instructions that, when executed on the data processing hardware (910), cause the data processing hardware (910) to perform operations, the operations including: obtaining a set of training samples (400), each training sample (400) in the set of training samples (400) including a corresponding sequence of a plurality of speech segments (405) corresponding to a training utterance and a corresponding sequence of a plurality of ground truth transcriptions (415) for the sequence of the plurality of speech segments (405), each pair of adjacent speech segments (405) in the sequence of the plurality of speech segments (405) being separated by a non-speech segment, and each ground truth transcription (415) in a corresponding sequence of speech segments (405) includes a start time (414) and an end time (416) of each corresponding speech segment (405) separated by the non-speech segments, and each training sample (400) is generated by concatenating a group of speech segments (405) each paired with a corresponding ground truth transcription (415) to form a training utterance of a particular length using the start time (414) and end time (416); For each training sample (400) in the set of training samples (400), processing the corresponding sequences of the plurality of speech segments (405) using a speech recognition model (200) to obtain one or more speech recognition hypotheses (522) for the training utterance; for each speech recognition hypothesis (522) obtained for the training utterance, determining a respective number of word errors relative to a corresponding sequence of the plurality of ground truth transcriptions (415); training the speech recognition model (200) to minimize a word error rate based on the respective number of word errors identified for each speech recognition hypothesis (522) obtained for the training utterance; A system (100) including:

9. processing the sequence of speech segments (405) to obtain one or more speech recognition hypotheses (522) includes processing the sequence of speech segments (405) to obtain an N-best list of the speech recognition hypotheses (522); 9. The system of claim 8, wherein training the speech recognition model to minimize a word error rate comprises training the speech recognition model to minimize a word error rate based on an average of the respective numbers of word errors identified for each speech recognition hypothesis in an N-best list of the speech recognition hypotheses.

10. 9. The system of claim 8, wherein each audio segment in the sequence of audio segments has an average duration of at least 6 seconds.

11. 9. The system of claim 8, wherein the speech recognition model comprises a transducer-based speech recognition model comprising an encoder network, a prediction network, and a joint network.

12. 12. The system of claim 11, wherein the encoder network comprises multiple multi-head attention layers.

13. The prediction network (300) performs, at each of a plurality of time steps following an initial time step: receiving as input the sequence of non-blank symbols (232) output by the final softmax layer (230); For each non-blank symbol (232) in the sequence of non-blank symbols (232) received as input at the corresponding time step: generating embedded representations (306) of the corresponding non-blank symbols (232) using the shared embedding matrix (304); assigning respective position vectors (308) to the corresponding non-blank symbols (232); weighting the embedded representations (306) in proportion to the similarity between the embedded representations (306) and the respective position vectors (308); generating as output a single embedding vector (350) for the corresponding time step, the single embedding vector (350) being based on a weighted average (324) of the weighted embedding representations (318); The joint network (320) performs, at each of the plurality of time steps following the initial time step: receiving as input the single embedding vector (350) generated as output from the prediction network (300) at the corresponding time step; generating a probability distribution (222) over the possible speech recognition hypotheses (522) at the corresponding time step.

14. 14. The system of claim 13, wherein the prediction network ties the dimensions of the shared embedding matrix to the dimensions of an output layer of the joint network.

15. 1. A computer-implemented method (800) for training a speech recognition model (200) to recognize long-form speech, comprising: obtaining a set of training samples (400), each training sample (400) in the set of training samples (400) comprising a corresponding sequence of a plurality of raw speech segments (405) and a corresponding sequence of a plurality of ground truth transcriptions (415) for the sequence of the plurality of raw speech segments (405), each pair of adjacent raw speech segments (405) in the sequence of the plurality of raw speech segments (405) being separated by a non-speech segment, and each ground truth transcription (415) in the corresponding sequence of the plurality of ground truth transcriptions (415) comprising a start time (414) and an end time (416) of each corresponding raw speech segment (405) separated by the non-speech segment; concatenating two or more groups of consecutive raw speech segments (405) from the sequence of the plurality of raw speech segments (405) by using the start time (414) and end time (416) to form training utterances of a particular length, thereby forming a respective long training utterance for each concatenated group (405G) of consecutive raw speech segments (405); During a first training stage, for each long training utterance, processing the concatenated group (405G) of consecutive raw speech segments (405) using the speech recognition model (200) to predict a speech recognition hypothesis (522) for the each long training utterance; and training the speech recognition model (200) using a logarithmic loss (537) based on the predicted speech recognition hypotheses (522) and corresponding groups (405G) of ground truth transcriptions (415) for each of the long training utterances.

16. During the second training stage, for each training sample (400) in the set of training samples (400): processing the corresponding sequences of the plurality of raw speech segments (405) using the speech recognition model (200) initialized by the first training stage to obtain one or more speech recognition hypotheses (522); for each speech recognition hypothesis (522), determining a respective number of word errors relative to a corresponding sequence of the plurality of ground truth transcriptions (415); training the speech recognition model (200) to minimize a word error rate based on the respective number of word errors identified for each speech recognition hypothesis (522); 16. The computer-implemented method (800) of claim 15, further comprising:

17. processing the sequence of the plurality of raw speech segments (405) to obtain one or more speech recognition hypotheses (522) comprises processing the sequence of the plurality of raw speech segments (405) to obtain an N-best list of speech recognition hypotheses (522); 17. The computer-implemented method of claim 16, wherein training the speech recognition model to minimize a word error rate comprises training the speech recognition model to minimize a word error rate based on an average of the respective numbers of word errors identified for each speech recognition hypothesis in an N-best list of the speech recognition hypotheses.

18. 17. The computer-implemented method of claim 16, wherein the speech recognition model comprises a transducer-based speech recognition model comprising an encoder network and a decoder, the decoder comprising a prediction network and a joint network.

19. 20. The computer-implemented method of claim 18, wherein training the speech recognition model to minimize a word error rate during the second training stage comprises fine-tuning parameters of the decoder initialized by the first training stage without fine-tuning parameters of the encoder network initialized by the first training stage.

20. Data processing hardware (910); and memory hardware (920) in communication with the data processing hardware (910) and storing instructions that, when executed on the data processing hardware (910), cause the data processing hardware to perform operations, the operations including: obtaining a set of training samples (400), each training sample (400) in the set of training samples (400) including a corresponding sequence of a plurality of raw speech segments (405) and a corresponding sequence of a plurality of ground truth transcriptions (415) for the sequence of the plurality of raw speech segments (405), each pair of adjacent raw speech segments (405) in the sequence of the plurality of raw speech segments (405) being separated by a non-speech segment, and each ground truth transcription (415) in the corresponding sequence of the plurality of ground truth transcriptions (415) including a start time (414) and an end time (416) of each corresponding raw speech segment separated by the non-speech segment; concatenating two or more groups of consecutive raw speech segments (405) from the sequence of the plurality of raw speech segments (405) by using the start time (414) and end time (416) to form training utterances of a particular length, thereby forming a respective long training utterance for each concatenated group (405G) of consecutive raw speech segments (405); During the first training phase, for each long training utterance, processing the concatenated group (405G) of consecutive raw speech segments (405) using a speech recognition model (200) to predict a speech recognition hypothesis (522) for each of the long training utterances; training the speech recognition model (200) using a logarithmic loss (537) based on the predicted speech recognition hypotheses (522) and corresponding groups of ground truth transcriptions (415) for each of the long training utterances; A system (100) including:

21. The operations include, during a second training stage, for each training sample (400) in the set of training samples (400): processing the corresponding sequence of the plurality of raw speech segments (405) using the speech recognition model (200) initialized by the first training stage to obtain one or more speech recognition hypotheses (522); for each speech recognition hypothesis (522), determining a respective number of word errors for a corresponding sequence of the plurality of ground truth transcriptions (415); training the speech recognition model (200) to minimize a word error rate based on the respective number of word errors identified for each speech recognition hypothesis (522); 21. The system (100) of claim 20, further comprising:

22. processing the sequence of raw speech segments (405) to obtain one or more speech recognition hypotheses (522) includes processing the sequence of raw speech segments (405) to obtain an N-best list of speech recognition hypotheses (522); 22. The system of claim 21, wherein training the speech recognition model to minimize a word error rate comprises training the speech recognition model to minimize a word error rate based on an average of the respective numbers of word errors identified for each speech recognition hypothesis in an N-best list of the speech recognition hypotheses.

23. 22. The system of claim 21, wherein the speech recognition model comprises a transducer-based speech recognition model comprising an encoder network and a decoder, the decoder comprising a prediction network and a joint network.

24. 24. The system of claim 23, wherein training the speech recognition model to minimize a word error rate during the second training stage comprises fine-tuning parameters of the decoder initialized by the first training stage without fine-tuning parameters of the encoder network initialized by the first training stage.

Images