Improving speech recognition with speech synthesis-based model adaptation

Abstract

A method (400) for training a speech recognition model (200) includes obtaining sample utterances of synthetic speech (306) in a target domain, obtaining transcribed utterances of non-synthetic speech (304) in the target domain, and pre-training the speech recognition model with the sample utterances of synthetic speech in the target domain to arrive at an initial state for warm-start training. After pre-training the speech recognition model, the method also includes warm-start training the speech recognition model with the transcribed utterances of non-synthetic speech in the target domain to teach the speech recognition model to learn to recognize real / human speech in the target domain.

Description

[Technical Field]

[0001] The present disclosure relates to improving speech recognition with speech synthesis-based model adaptation. [Background technology]

[0002] Automatic speech recognition (ASR), the process of receiving audio input and transcribing it into text, is a highly important technology used in mobile devices and other devices. Generally, automatic speech recognition attempts to provide an accurate transcription of what a person said by receiving audio input (e.g., a spoken utterance) and transcribing the audio input into text. Based on the ongoing development of deep neural networks, modern ASR models continue to improve in both accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay between a user's utterance and transcription). However, one challenge in developing deep learning-based ASR models is that the parameters of the ASR model tend to overfit to the training data, which causes the ASR model to have difficulty generalizing to unseen data when the training data is not sufficiently large. As a result, training an ASR model with a relatively large training data set improves the accuracy of the ASR model. To increase the volume of training data used to train an ASR model, synthetic speech and / or data augmentation speech can be incorporated. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations. The operations include obtaining sample utterances of synthetic speech in a target domain and obtaining transcribed utterances of non-synthetic speech in the target domain. The operations also include pre-training a speech recognition model with the sample utterances of synthetic speech in the target domain to arrive at an initial state for warm-start training. After pre-training the speech recognition model, the operations also include warm-start training the speech recognition model with the transcribed utterances of non-synthetic speech in the target domain to teach the speech recognition model to learn to recognize real / human speech in the target domain.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, obtaining sample utterances of the synthetic speech in the target domain includes generating, using a text-to-speech (TTS) system, each sample utterance of the synthetic speech based on a corresponding transcription of an utterance received as input at the TTS system. The corresponding transcription may be sampled from a language model. The TTS system may be trained with transcribed audio samples in the target domain and / or trained with out-of-domain audio samples different from the target domain. The TTS system may be further configured to at least one of vary voice characteristics across the utterances of the synthetic speech or vary prosodic / style qualities across the utterances of the synthetic speech.

[0005] In some examples, the operations also include applying data augmentation to one or more of the sample utterances of the synthetic speech, where the applied data augmentation may include at least one of adding noise, adding reverberation, or manipulating timing. The speech recognition model may include a frame alignment-based transducer model, such as a recurrent neural network transducer (RNN-T) model. The speech recognition model may include an alignment-based encoder-decoder model.

[0006] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining sample utterances of synthetic speech in a target domain and obtaining transcribed utterances of non-synthetic speech in the target domain. The operations also include pre-training a speech recognition model with the sample utterances of synthetic speech in the target domain to arrive at an initial state for warm-start training. After pre-training the speech recognition model, the operations also include warm-start training the speech recognition model with the transcribed utterances of non-synthetic speech in the target domain to teach the speech recognition model to learn to recognize real / human speech in the target domain.

[0007] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, obtaining sample utterances of the synthetic speech in the target domain includes generating, using a text-to-speech (TTS) system, each sample utterance of the synthetic speech based on a corresponding transcription of an utterance received as input at the TTS system. The corresponding transcription may be sampled from a language model. The TTS system may be trained with transcribed audio samples in the target domain and / or trained with out-of-domain audio samples different from the target domain. The TTS system may be further configured to at least one of vary voice characteristics across the utterances of the synthetic speech or vary prosodic / style qualities across the utterances of the synthetic speech.

[0008] In some examples, the operations also include applying data augmentation to one or more of the sample utterances of the synthetic speech, where the applied data augmentation may include at least one of adding noise, adding reverberation, or manipulating timing. The speech recognition model may include a frame alignment-based transducer model, such as a recurrent neural network transducer (RNN-T) model. The speech recognition model may include an alignment-based encoder-decoder model.

[0009] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram of an exemplary speech recognition system including a speech recognition model. [Figure 2A] FIG. 1 is a schematic diagram of a recurrent neural network transducer (RNN-T) model architecture. [Figure 2B] 1 is a schematic diagram of an attention-based encoder-decoder (AED) model architecture. [Figure 3] FIG. 1 is a schematic diagram of an exemplary training process for promoting a speech recognition model to learn consistent predictions for both non-synthetic and synthetic speech. [Figure 4] 1 is a flowchart of an example sequence of operations for a method of training a speech recognition model. [Figure 5] FIG. 1 is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0011] Like numbers in the various drawings indicate like elements.

[0012] Automated speech recognition has made tremendous progress with the introduction of sequence-to-sequence (Seq2Seq) models that map from audio to character sequences. At the same time, text-to-speech (TTS) or speech synthesis systems have successfully applied Seq2Seq models to obtain state-of-the-art, natural-sounding, lifelike synthetic speech that may be indistinguishable from human speech to the human ear.

[0013] One challenge in developing deep learning-based ASR models is that their parameters tend to overfit to the training data, causing the ASR model to have difficulty generalizing to unseen data when the training data is not sufficiently large. Therefore, training an ASR model with a relatively large training dataset improves the accuracy of the ASR model. For example, using machine learning or other statistical methods, an ASR model can be trained with a training dataset containing approximately 10,000 hours of transcribed speech. However, the performance of the ASR model deteriorates when the domain associated with the training data is distinct from the domain in which the ASR model is deployed during inference. For example, training an ASR model with transcribed speech in a domain associated with videoconferencing is relatively ineffective at recognizing speech related to voice search queries, and vice versa.

[0014] Synthetic speech has the potential to drastically limit the amount of labeled human speech required to train an ASR model, while also providing flexibility for moving ASR models between different domains. Overall, the use of synthetic speech has been shown to affect ASR training differently from human speech, despite examples of state-of-the-art synthetic speech that is indistinguishable from human speech. This gap between synthetic and human speech stems from the discrepancy in synthesized speech data from human speech data that arises from the difficult one-to-many mapping problem that TTS systems attempt to solve. That is, while the ensemble quality of available synthetic speech is very high, synthetic speech exhibits much less variability than human speech, as well as minimal speech stuttering. As a result, training an ASR model exclusively with synthetic speech data presents challenges in generalizing to real speech utterances during inference.

[0015] Implementations herein are directed to using synthetic speech to train an ASR model to recognize speech in a target domain to maintain accuracy of the ASR model when a large amount of transcribed speech (e.g., non-synthetic speech) in the target domain for training the ASR model is not available or is relatively uncommon. More specifically, implementations herein are directed to pre-training an ASR model to achieve a pre-computed reference state, and then using the synthetic speech in the target domain to warm-start training the ASR model from the pre-computed reference state using available transcribed non-synthetic speech in the target domain. Additional implementations include applying data augmentation techniques, such as synthesizing diverse realizations of training utterances by varying synthesized speaker characteristics, to promote robustness to speaker differences. The techniques described herein are particularly useful when a relatively small amount of transcribed human speech in the target domain is available.

[0016] 1 illustrates an automated speech recognition (ASR) system 100 that implements an ASR model 200 that resides on a user device 102 of a user 104 and / or on a remote computing device 201 (e.g., one or more servers of a distributed system executing in a cloud computing environment) that communicates with the user device 102. While the user device 102 is shown as a mobile computing device (e.g., a smartphone), the user device 102 may correspond to any type of computing device, such as, but not limited to, a tablet device, a laptop / desktop computer, a wearable device, a digital assistant device, a smart speaker / display, a smart appliance, an in-vehicle infotainment system, or an Internet of Things (IoT) device, and is equipped with data processing hardware 111 and memory hardware 113.

[0017] The user device 102 includes an audio subsystem 108 configured to receive utterances 106 spoken by the user 104 (e.g., the user device 102 may include one or more microphones for recording the spoken utterances 106) and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the ASR system 100. In the illustrated example, the user speaks each utterance 106 in natural English for the phrase "What is the weather in New York City?", and the audio subsystem 108 converts the utterance 106 into a corresponding acoustic frame 110 for input to the ASR system 100. The ASR model 200 then receives the acoustic frame 110 corresponding to the utterance 106 as input and generates / predicts a corresponding transcription 120 (e.g., a recognition result / hypothesis) of the utterance 106 as output. In the illustrated example, the user device 102 and / or the remote computing device 201 also execute a user interface generator 107 configured to present a representation of a transcription 120 of the utterance 106 to the user 104 of the user device 102. In some configurations, the transcription 120 output from the ASR system 100 is processed by a natural language understanding (NLU) module executing on the user device 102 or the remote computing device 201, for example, to execute user commands. Additionally or alternatively, a text-to-speech system (e.g., executing on any combination of the user device 102 or the remote computing device 201) may convert the transcription into synthesized speech for audible output by another device. For example, the original utterance 106 may correspond to a message the user 104 is sending to a friend, where the transcription 120 is converted into synthesized speech for audible output to the friend, who should hear the message conveyed in the original utterance 106.

[0018] 2A and 2B, the ASR model 200 may include an end-to-end (E2E) sequence-to-sequence model, such as a frame-alignment-based transducer model 200a (FIG. 2A) or an attention-based encoder-decoder (AED) model 200b (FIG. 2B). The ASR model 200 may provide E2E speech recognition by integrating acoustic, pronunciation, and language models into a single neural network, without requiring a lexicon or a separate text normalization component. Various structures and optimization mechanisms can increase accuracy and reduce model training time. The ASR model 200

[0019] Referring to FIG. 2A , an exemplary frame-alignment-based transducer model 200a includes a recurrent neural network transducer (RNN-T) model architecture that adheres to latency constraints associated with interactive applications. The use of the RNN-T model architecture is exemplary, and the frame-alignment-based transducer model 200a may include other architectures, such as transformer transducer and conformal transducer model architectures, among others. The RNN-T model 200a offers a small computational footprint and uses fewer memory requirements than traditional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device 102 (e.g., no communication with a remote server is required). The RNN-T model 200a includes an encoder network 210, a prediction network 220, and a collaboration network 230. The encoder network 210 is generally similar to an acoustic model (AM) in traditional ASR systems and includes a recurrent network consisting of stacked long short-term memory (LSTM) layers. For example, the encoder may generate a sequence of d-dimensional feature vectors (e.g., acoustic frame 110 (FIG. 1)) x=(x1, x2,..., x T ), where

[0020]

number

[0021] and at each time step, a higher-level representation is generated.

[0022]

number

[0023] ,...,

[0024]

number

[0025] It is shown as follows.

[0026] Similarly, the prediction network 220 is also an LSTM network, which, like a language model (LM), computes the sequence of non-empty symbols output so far by the final softmax layer 240, i.e., y 0 , ..., y ui-1 , a dense representation

[0027]

number

[0028] Finally, using an RNN-T model architecture, the representations produced by the encoder and prediction networks 210, 220 are combined by a collaborative network 230. The prediction network 220 may be replaced with an embedding lookup table to improve latency by outputting a looked-up sparse embedding instead of processing a dense representation. The collaborative network then

[0029]

number

[0030] , which is a distribution over the next output symbol. In other words, at each output step (e.g., time step), the collaborative network 230 generates a probability distribution over possible speech recognition hypotheses. Here, a "possible speech recognition hypothesis" corresponds to a set of output labels, each representing a symbol / character in a specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters of the English alphabet, and one label specifying a space. Thus, the collaborative network 230 may output a set of values ​​indicating the likelihood of each occurrence of a given set of output labels. This set of values ​​may be a vector and may indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels may include wordpieces and / or entire words in addition to or instead of graphemes. The output distribution of the collaborative network 230 may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output y of the collaborative network 230 i may contain 100 different probability values, one for each output label. The probability distributions can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and / or words) in the beam search process (e.g., by softmax layer 240) to determine transcription 120.

[0031] The softmax layer 240 may utilize any technique for selecting the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200a at the corresponding output step. In this manner, the RNN-T model 200a does not make a conditional independence assumption; rather, the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model 200a assumes that the output symbols are independent of future acoustic frames 110, which allows the RNN-T model to be utilized in a streaming manner.

[0032] In some examples, the encoder network 210 of the RNN-T model 200a consists of eight 2,048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. The prediction network 220 may have two 2,048-dimensional LSTM layers, each also followed by a 640-dimensional projection layer. Finally, the joint network 230 may also have 640 hidden units. The softmax layer 240 may consist of a unified word piece or grapheme set generated using all unique word pieces or graphemes in multiple training datasets 301 (FIG. 3).

[0033] 2B, an exemplary AED model 200b associated with a Listen, Attend and Spell (LAS) model architecture provides a single neural network that includes a listener encoder module 211, which is similar to a traditional acoustic model, an attendant module 221, which acts as an alignment model, and a decoder 231, which is similar to a language model in traditional systems. Specifically, the listener encoder module 211 receives input features (e.g., acoustic frame 110 (FIG. 1)), i.e., x, and converts them into a higher-level feature representation, i.e., h enc The encoded feature representation, i.e., h encThis process of generating f can be done for each of several input frames, each representing a different input time step. These time steps are denoted below by the subscript u. Thus, given a set of frames {f,f,f,...f u}, the corresponding set of encoded outputs {h1,h2,h3,...h u} is possible.

[0034] The output of the listener encoder module 211 is passed to the attendant module 221, which, like the dynamic time warping (DTW) alignment module, enc Which encoder features in i In some examples, the attendant module 221 is referred to herein as an attendant neural network or attendant 221. The attendant 221 generates a context output c for each of a plurality of output steps i. i Each context output vector c i For u, the attender 221 can compute attention based on the encoding for one or more input steps u, e.g., the encoding for the current input step as well as the encoding for the previous input step. For example, the attender 221 can compute attention based on the set of all encoder outputs of an utterance, e.g., the set {h1, h2, h3, ... h u} throughout, attention context output c i The attention context vector may be a vector that represents a weighted summary of the current and previous encodings for a frame (e.g., part) of the utterance being recognized.

[0035] Finally, the output of the attender 221 is passed to the decoder 231, which uses the attention context (e.g., context vector or attention distribution) output by the attender 221, i.e., c i , as well as the previous prediction, i.e., y iThe decoder output is the previous unit, i.e. {y i-1 ,...,y0}, and the current subword unit, i.e., y, given the input, i.e., x. i Probability distribution over y, i.e., P(y i |y i-1 ,...,y0,x). Thus, at each output step, decoder 231 generates a probability distribution over the possible speech recognition hypotheses. Similar to RNN-T model 200a described above with reference to FIG. 2A, a "possible speech recognition hypothesis" corresponds to a set of output symbols, each representing a symbol / character in the specified natural language.

[0036] Although not shown, the ASR model 200 may include a softmax layer that receives the output of the decoder 231. In some implementations, the softmax layer is separate from the decoder 231 and receives the output from the decoder 231, i.e., y i , and the output of the softmax layer is then used in the beam search process to select the orthographic elements. In some implementations, the softmax layer is integrated with the decoder 231, so that the output y i represents the output of the softmax layer.

[0037] The decoder 231 and / or associated softmax layer may be trained to output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels. This set of values ​​may be a vector, indicating a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels may include wordpieces and / or entire words in addition to, or instead of, graphemes. The output distribution of the decoder 231 and / or softmax layer may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the decoder output y i , or output y i The output of a softmax layer that receives and processes σ may include 100 different probability values, one for each output label. The probability distributions can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and / or words) in the beam search process to determine transcription.

[0038] Referring to FIG. 3 , an exemplary training process 300 is shown for initially pre-training the ASR model 200 to achieve a pre-calculated reference state from a random state, and then warm-start training the model to teach the ASR model 200 from the pre-calculated reference state to learn to accurately recognize speech in a target domain. As used herein, a target domain may refer to the type of speech that the ASR model 200 recognizes during inference when utilized. For example, a target domain may include conference speech, voice search speech, sports broadcaster speech, news anchor speech, or navigation command speech, to name a few. A target domain may also correspond to a specific language, such as a low-resource language for which transcribed human speech is not readily available. In some examples, the ASR model 200 includes a multilingual ASR model. The training process 300 may be performed on the remote computing device 201 of FIG. 1 . For example, the remote computing device 201 includes data processing hardware 510 (FIG. 5) and memory hardware 520 (FIG. 5) that communicates with the data processing hardware and stores instructions that, when executed on the data processing hardware 510, cause the data processing hardware 510 to perform the operations of the training process 300.

[0039] In the illustrated example, the training process 300 includes a pre-training stage in which the ASR model 200 is pre-trained from a random state using sample utterances of the synthetic speech 306 in the target domain to arrive at an initial state for warm-start training. This initial state is associated with a pre-computed reference state associated with the target domain. Thus, the pre-training stage uses the sample utterances of the synthetic speech 306 to adapt the model for warm-start training.

[0040] The text-to-speech (TTS) system 330 may generate each sample utterance of the synthetic speech 306 by performing TTS conversion on a corresponding transcription 320 of the utterance. Here, each transcription 320 corresponds to input text received by the TTS system 330 for conversion to synthetic audio including the corresponding utterance of the synthetic speech 306 in the target domain. Thus, each transcription 320 includes a sequence of text in the target domain. The sequence of text may include graphemes or phonemes. The transcription 320 may be sampled from a language model trained to generate text utterances in the target domain. The TTS system 330 may apply a speaker embedding, i.e., z, when converting the transcription 320 to obtain synthetic speech with a particular speaking style and prosody associated with the speaker embedding. The TTS system 330 may apply multiple different speaker embeddings z, each associated with a different speaker characteristic of the resulting utterance of the synthetic speech 306 being synthesized. Similarly, the TTS system 330 may vary the prosody and other production qualities of the synthesized utterances.

[0041] In some examples, the training process first trains the TTS system 330 using available transcribed audio samples. In some examples, the available audio samples used to train the TTS system 330 include in-domain audio samples associated with the target domain. In other examples, the available audio samples used to train the TTS system 330 include out-of-domain audio samples distinct from the target domain. In these examples, the TTS system 330 generates utterances of the synthetic speech 306 in the target domain for input to the ASR model 200 during the pre-training phase, even though the TTS system 330 is trained with out-of-domain transcribed audio samples. The TTS system 330 may, in some examples, be trained with in-domain and out-of-domain variants.

[0042] In some examples, the training process 300 applies data augmentation to at least one of the sample utterances of the synthetic speech 306. Data augmentation may include, but is not limited to, adding noise, manipulating timing (e.g., stretching), or adding reverberation to the corresponding speech representation. Data augmentation may apply different synthesized recording conditions to the synthetic speech 306.

[0043] During the pre-training phase, the ASR model 200 receives as input the synthesized speech

[0044]

number

[0045] Each utterance in 306 is received as a sequence of features / vectors (e.g., acoustic frames 110 in FIG. 1) and, as output, for each of a plurality of output steps, a corresponding synthesized speech of the utterance.

[0046]

number

[0047] 306. For simplicity, the term "synthetic speech recognition result 311" is used to refer to the synthetic speech recognition result 311.

[0048]

number

[0049] may be used interchangeably to refer to a first probability distribution 311 over the possible synthetic speech recognition hypotheses (y) for the corresponding utterance of 306.

[0050] The training process 300 outputs a transcription 320 and a synthesized speech as output at each of a number of output steps.

[0051]

number

[0052] a first probability distribution 311 over the possible synthetic speech recognition hypotheses (y) for the corresponding utterances of 306 and a synthetic speech loss term (J tts (θ)) 344. In particular, the transcription 320 corresponding to the input text that is provided to the TTS system 330 to generate a corresponding utterance of the synthetic speech 306 also serves as the ground truth label / transcription. In the illustrated example, the training process 300 receives the transcription 320 and the first probability distribution 311 as ground truth and generates a first supervised loss term including the synthetic speech loss term (J tts Implements a supervised loss term module 340 configured to output a synthetic speech loss term (J tts (θ) 344 can be calculated by the following formula:

[0053]

number

[0054] where y*, z are the synthesized speech representations generated by the TTS module 330 based on the transcription (y*) and speaker embedding (z).

[0055]

number

[0056] Shows 306.

[0057] The supervised loss term module 340 calculates the synthetic speech loss term (J tts(θ)) 344 may be returned to the ASR model 200. For example, the training process 300 may train the ASR model 200 using a stochastic optimization algorithm, such as stochastic gradient descent, to train the ASR model 200 through backpropagation. Here, the stochastic optimization algorithm uses loss terms 344 to define respective loss functions (e.g., cross-entropy loss functions) based on the difference between the actual output of the neural network (e.g., synthetic speech recognition results 311) and the desired output (e.g., ground truth transcription 320 of the corresponding utterance). For example, the loss functions are calculated for a batch of training examples and then differentiated with respect to each weight in the ASR model 200.

[0058] Once the pre-training phase is complete and an initial state is reached, the training process 300 may perform a warm-start training phase to fine-tune the pre-trained model using available transcribed utterances of non-synthesized speech 304 in the target domain. Each utterance of non-synthesized speech 304 corresponds to an actual speech uttered by a human. In some examples, the training process 300 applies data augmentation to at least one of the utterances of non-synthesized speech 304. Data augmentation may include, but is not limited to, adding noise, manipulating timing (e.g., stretching), or adding reverberation to the corresponding speech representation. Each transcribed utterance of non-synthesized speech 304 is associated with a corresponding ground truth transcription 321.

[0059] During the warm-start training phase, the ASR model 200 receives as input each utterance of non-synthetic speech (x) 304 as a sequence of features / vectors (e.g., acoustic frames 110 of FIG. 1 ) and generates as output, for each of a plurality of output steps, a second probability distribution 312 over the possible non-synthetic speech recognition hypotheses (y) for the corresponding utterance of non-synthetic speech (x) 304. For simplicity, the term “non-synthetic speech recognition result 312” may be used to refer to the second probability distribution 312 over the possible non-synthetic speech recognition hypotheses (y) for the corresponding utterance of non-synthetic speech (x) 304.

[0060] The training process 300 then calculates, for output by the ASR model 200 at each of a plurality of output steps, a non-synthetic speech loss term (J) based on the ground truth transcription 321 and the non-synthetic speech recognition result 312 (y) for the corresponding non-synthetic speech representation (x) 304 of the corresponding utterance. real In the illustrated example, the training process 300 receives the ground truth transcription 321 and the second probability distribution 312 and generates a second supervised loss term (J real Implements a supervised loss term module 340 configured to output a non-synthetic speech loss term (J real (θ) 342 can be calculated by the following formula:

[0061]

number

[0062] The supervised loss term module 340 calculates the non-synthetic speech loss term (J real (θ)) 342 to the ASR model 200. For example, the training process 300 may train the ASR model 200 using a stochastic optimization algorithm, such as stochastic gradient descent, to train the ASR model 200 through backpropagation. Here, the stochastic optimization algorithm uses loss terms 342 to define respective loss functions (e.g., cross-entropy loss functions) based on the difference between the actual output of the neural network (e.g., the non-synthetic speech recognition results 312) and the desired output (e.g., the ground truth transcription 321 of the corresponding utterance). For example, the loss functions are calculated for a batch of training examples and then differentiated with respect to each weight in the ASR model 200.

[0063] Advantageously, the ASR model 200 can be initially pre-trained with a potentially infinite number of utterances of synthetic speech in the target domain to adapt the ASR model 200 to learn to recognize real / human speech in the target domain. In particular, the techniques herein are particularly advantageous when there are few available transcribed utterances of non-synthetic speech in the target domain. Thus, the ASR model 200 can be quickly trained in a new target domain without having to rely on transcribed non-synthetic speech in the domain, because synthetic speech can be generated on-the-fly from unpaired text to train the ASR model 200 in the target domain. In one exemplary use case, the accuracy of the ASR model 200 can be drastically improved for recognizing speech in natural languages, where transcribed non-synthetic speech is difficult to obtain, by increasing the number of speech training examples from synthetic speech generated from unpaired text during the pre-training phase.

[0064] 4 is a flowchart of an example sequence of operations for a method 400 of training a speech recognition model 200. The ASR model 200 may include an attention-based encoder-decoder (AED) model 200b (FIG. 2B) or a frame-alignment-based transducer model 200a (FIG. 2A). At operation 402, the method 400 includes obtaining a sample utterance of synthesized speech 306 in a target region and obtaining a transcribed utterance of unsynthesized speech 304 in the target region.

[0065] At operation 404, method 400 includes pre-training speech recognition model 200 with sample utterances of synthetic speech 306 in the target domain to arrive at an initial state for warm-start training. After pre-training speech recognition model 200, method 400 also includes warm-start training speech recognition model 200 with transcribed utterances of non-synthetic speech 304 in the target domain to teach speech recognition model 200 to learn to recognize real / human speech in the target domain at operation 406.

[0066] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0067] Non-transitory memory may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transitory memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0068] 5 is a schematic diagram of an exemplary computing device 500 that can be used to implement the systems and methods described herein. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the implementation of the invention as described and / or claimed herein.

[0069] Computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 connecting to memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 connecting to a low-speed bus 570 and storage device 530. Each of components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be mounted on a common motherboard or in other manners as desired. Processor 510 (i.e., data processing hardware 111 or data processing hardware 201) can process instructions for execution within computing device 500, including instructions stored in memory 520 or on storage device 530, for displaying graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 coupled to high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memory, as desired. Also, multiple computing devices 500 may be connected, each providing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0070] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0071] The storage device 530 is capable of providing mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as memory 520, the storage device 530, or memory on the processor 510.

[0072] The high-speed controller 540 manages bandwidth-intensive operations for the computing device 500, while the low-speed controller 560 manages less bandwidth-intensive operations. Such allocation of duties is merely exemplary. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 550, which may receive various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and a low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), but may also be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, etc., or to a network device, such as a switch or router, for example, through a network adapter.

[0073] The computing device 500 may be implemented in several different forms, as shown in the figure, such as a standard server 500a, or in a group of servers 500a, such as laptop computers 500b, or many times as part of a rack server system 500c.

[0074] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuitry, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or translatable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0075] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0076] The processes and logic flows described herein can be implemented by one or more programmable processors, also referred to as data processing hardware, that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be implemented by special-purpose logic circuitry, such as FPGAs (field-programmable gate arrays) and ASICs (application-specific integrated circuits). Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, e.g., magnetic, magneto-optical, or optical disks, for storing data, or is operably coupled to a mass storage device to receive data from, transfer data to, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0077] To enable user interaction, one or more aspects of the present disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touch screen, for displaying information to the user, and optionally a keyboard and pointing device, e.g., a mouse or trackball, for allowing the user to provide input to the computer. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0078] Although several implementations have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0079] 100 Automated Speech Recognition (ASR) Systems 102 User Devices 107 User Interface Generator 110 Acoustic Frame 111 Data Processing Hardware 113 Memory Hardware 200 ASR models, speech recognition models, models 200a Frame alignment based transducer model, RNN-T model 200b Attention-based Encoder-Decoder Model, AED Model 201 Remote Computing Devices, Data Processing Hardware 210 Encoder Network, Encoder 211 Listener Encoder Module 220 Prediction Network 221 Attendant Model, Attendant 230 Collaborative Network 231 decoder 240 Final softmax layer, softmax layer 300 training processes 304 Non-synthetic Speech 306 Synthetic Speech 311 First probability distribution, synthetic speech recognition results 312 Second probability distribution, non-synthetic speech recognition results 320 Transcription 321 Ground Truth Transcription 330 Text-to-Speech (TTS) Systems 340 Supervised Loss Term Module 342 Non-synthetic speech loss terms 344 Synthetic Speech Loss Terms 500 computing devices 500a Standard Server, Server 500b laptop computer 500c Rack Server System 510 Processors, Data Processing Hardware, Components 520 Memory, Memory Hardware, and Components 530 Storage devices, components 540 High-Speed ​​Interface / Controller, Components 550 High-Speed ​​Expansion Port, Components 560 Low-Speed ​​Interface / Controller, Components 570 Slow Bus 580 Display 590 Low-Speed ​​Expansion Port

Claims

1. A computer-implemented method (400) executed by data processing hardware (510) to cause the data processing hardware (510) to perform operations, the operations comprising: obtaining a sample utterance of a synthesized speech (306) in a target region; obtaining a transcribed utterance of the non-synthesized speech (304) in the target region; pre-training a speech recognition model (200) with only the sample utterances of the synthetic speech (306) in the target domain to arrive at an initial state for warm-start training; pre-training the speech recognition model (200), and then warm-start training the pre-trained speech recognition model (200) with the transcribed utterances of non-synthetic speech (304) in the target domain to teach the speech recognition model (200) to learn to recognize real / human speech in the target domain; A method (400) comprising:

2. 10. The method of claim 1, wherein obtaining the sample utterances of the synthetic voice in the target region includes generating, using a text-to-speech (TTS) system, each sample utterance of the synthetic voice based on a corresponding transcription of the utterance received as input at the TTS system.

3. The method of claim 2, wherein the corresponding transcription is sampled from a language model.

4. 3. The method of claim 2, wherein the TTS system is trained with transcribed audio samples in the target region.

5. 3. The method of claim 2, wherein the TTS system is trained with audio samples outside the target domain.

6. 3. The method of claim 2, wherein the TTS system is configured to vary voice characteristics across the utterances of the synthesized voice.

7. 3. The method of claim 2, wherein the TTS system is configured to vary prosodic / stylistic qualities across the utterances of the synthesized voice.

8. 10. The method of claim 1, wherein the operations further comprise applying data augmentation to one or more of the sample utterances of the synthesized speech.

9. 10. The method (400) of claim 8, wherein the applied data augmentation comprises at least one of adding noise, adding reverberation, or manipulating timing.

10. The method of claim 1, wherein the speech recognition model includes a frame alignment-based transducer model.

11. 11. The method of claim 10, wherein the frame alignment based transducer model comprises a recurrent neural network transducer (RNN-T) model.

12. 12. The method (400) of any one of claims 1 to 11, wherein the speech recognition model (200) comprises an alignment-based encoder-decoder model (200b).

13. data processing hardware (510); and memory hardware (520) in communication with the data processing hardware (510), the memory hardware (520) storing instructions that, when executed on the data processing hardware (510), cause the data processing hardware (510) to perform operations, the operations including: obtaining a sample utterance of a synthesized speech (306) in a target region; obtaining a transcribed utterance of the non-synthesized speech (304) in the target region; pre-training a speech recognition model (200) with only the sample utterances of the synthetic speech (306) in the target domain to arrive at an initial state for warm-start training; and warm-start training the pre-trained speech recognition model (200) with the transcribed utterances of non-synthetic speech (304) in the target domain, after pre-training the speech recognition model (200), to teach the speech recognition model (200) to learn to recognize real / human speech in the target domain. A system (100) including:

14. 14. The system of claim 13, wherein obtaining the sample utterances of the synthetic voice in the target region includes generating, using a text-to-speech (TTS) system, each sample utterance of the synthetic voice based on a corresponding transcription of the utterance received as input at the TTS system.

15. The system (100) of claim 14, wherein the corresponding transcription (320) is sampled from a language model.

16. 15. The system of claim 14, wherein the TTS system is trained with transcribed audio samples in the target region.

17. 15. The system of claim 14, wherein the TTS system is trained with audio samples outside the target domain.

18. 15. The system of claim 14, wherein the TTS system is configured to vary voice characteristics across the utterances of the synthesized voice.

19. 15. The system of claim 14, wherein the TTS system is configured to vary prosodic / stylistic qualities across the utterances of the synthesized voice.

20. 14. The system (100) of claim 13, wherein the operations further include applying data augmentation to one or more of the sample utterances of the synthesized speech (306).

21. 21. The system (100) of claim 20, wherein the applied data augmentation comprises at least one of adding noise, adding reverberation, or manipulating timing.

22. 14. The system (100) of claim 13, wherein the speech recognition model (200) includes a frame alignment-based transducer model (200a).

23. 23. The system (100) of claim 22, wherein the frame alignment based transducer model (200a) comprises a recurrent neural network transducer (RNN-T) model (200a).

24. 24. The system (100) of any one of claims 13 to 23, wherein the speech recognition model (200) comprises an alignment-based encoder-decoder model (200b).

Images