Speech recognition using non-spoken text and speech synthesis

Abstract

To provide a method for providing speech recognition using unspoken text and speech synthesis.SOLUTION: A method 500 for simultaneously training generative adversarial network (GAN)-based text-to-speech (TTS) models and speech recognition models includes the steps of: acquiring a plurality of training text utterances; generating, for output by a GAN-based TTS model, a synthetic speech representation of corresponding training text utterances; determining, using an adversarial discriminator, an adversarial loss term indicating an amount of acoustic noise imbalance in a non-synthetic speech representation relative to a corresponding synthetic speech representation of the corresponding training text utterances; and updating parameters of the GAN-based TTS model parameters based on the adversarial loss term.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] TECHNICAL FIELD This disclosure relates to speech recognition using non-spoken text and speech synthesis. [Background technology]

[0002] Automatic speech recognition (ASR), the process of receiving audio input and transcribing it into text, is a highly important technology used in mobile and other devices. Generally, automatic speech recognition attempts to provide an accurate transcription of what a person said by receiving audio input (e.g., a spoken utterance) and transcribing the audio input into text. Based on the ongoing development of deep neural networks, modern ASR models continue to improve in both accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay between a user's utterance and transcription). However, one challenge in developing deep learning-based ASR models is that the parameters of the ASR model tend to overfit to the training data, which causes the ASR model to have difficulty generalizing to unseen data when the training data is not sufficiently large. As a result, training an ASR model with a relatively large training dataset improves the accuracy of the ASR model. Synthetic speech and / or data augmented speech can be incorporated to increase the volume of training data used to train the ASR model, as well as to increase linguistic diversity when speech is synthesized from unspoken text that can be easily and inexpensively obtained for immediate training of the ASR model. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present disclosure provides a method for simultaneously training a generative adversarial network (GAN)-based text-to-speech (TTS) model and a speech recognition model. The method includes acquiring, by data processing hardware, a plurality of training text utterances, where a first portion of the plurality of training text utterances includes unspoken text utterances and a second portion of the plurality of training text utterances includes transcriptions of a set of spoken training utterances. Each unspoken text utterance is not paired with any corresponding spoken utterance. Each spoken training utterance, including a corresponding transcription, is paired with a corresponding non-synthesized speech representation of the corresponding spoken training utterance. For each of the plurality of output steps for each training text utterance of the plurality of training text utterances, the method also includes generating, by the data processing hardware, a synthetic speech representation of the corresponding training text utterance for output by the GAN-based TTS model, and determining, by the data processing hardware, using a GAN adversarial discriminator, an adversarial loss term indicative of an amount of acoustic noise imbalance in one of the non-synthetic speech representations selected from the set of spoken training utterances relative to the corresponding synthetic speech representation of the corresponding training text utterance. The method also includes updating, by the data processing hardware, parameters of the GAN-based TTS model based on the adversarial loss term determined at each of the plurality of output steps for each training text utterance of the plurality of training text utterances. The method also includes training, by the data processing hardware, a speech recognition model with the synthetic speech representation generated at each of the plurality of output steps for each training text utterance of the plurality of training text utterances.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, training the speech recognition model further includes training the speech recognition model with training utterance batches, where each training utterance batch includes a portion of a generated synthetic speech representation of a corresponding training text utterance and a portion of a non-synthesized speech representation in the set of spoken training utterances, and a ratio of the portion of the generated synthetic speech representation to the portion of the non-synthesized speech representation in each training utterance batch increases over time during training of the speech recognition model.

[0005] In implementations in which the speech recognition model is trained with training utterance batches, the method may also optionally include generating, by the data processing hardware, during training of the speech recognition model with each training utterance batch, a respective non-synthesized speech loss term for each non-synthesized speech representation in the portion of the non-synthesized speech representation for the corresponding training utterance batch, and applying, by the data processing hardware, an adjustable loss weight to each respective non-synthesized speech loss term generated by the speech recognition model, wherein the adjustable weight has a magnitude proportional to the ratio of the portion of the generated speech representation to the portion of the non-synthesized speech representation in the corresponding training utterance batch.

[0006] Furthermore, while training the speech recognition model with each training utterance batch, the method may also include, in each of a plurality of output steps for each generated synthetic speech representation among the portion of generated synthetic speech representations associated with the corresponding training utterance batch, determining, by the data processing hardware, a first probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation for output by the speech recognition model, and generating, by the data processing hardware, a synthetic speech loss term based on the first probability distribution over the possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation and the corresponding training text utterance from which the corresponding synthetic speech representation is generated. At the same time, the method includes, in each of a plurality of output steps for each non-synthesized speech expression in the portion of non-synthesized speech expressions associated with the corresponding training utterance batch, determining, by the data processing hardware, a second probability distribution over possible non-synthesized speech recognition hypotheses for the corresponding non-synthesized speech expression for output by the speech recognition model, and generating, by the data processing hardware, a non-synthesized speech loss term based on the second probability distribution over the possible non-synthesized speech recognition hypotheses for the corresponding non-synthesized speech expression and a transcription in the set of spoken training utterances paired with the corresponding non-synthesized speech expression.

[0007] In some examples, in each of the multiple outputting steps for each training text utterance of the multiple training text utterances, one of the non-synthesized speech representations selected from the set of spoken training utterances includes a randomly selected non-synthesized speech representation from the set of spoken training utterances when the corresponding training text utterance includes one of the non-spoken text utterances in the first portion of the multiple training text utterances, or a non-synthesized speech representation from the set of spoken training utterances paired with a corresponding one of the transcriptions when the corresponding training text utterance includes one of the transcriptions in the second portion of the multiple training text utterances. In additional examples, each training text utterance is conditioned on a utterance embedding selected from a set of utterance embeddings and a speaker embedding selected from a set of speaker embeddings. Each speaker embedding in the set of speaker embeddings represents a speaker characteristic, and each utterance embedding in the set of utterance embeddings represents an intended prosody. Each training text utterance may be represented by a corresponding phoneme sequence, each utterance embedding in the set of utterance embeddings may be extracted from a corresponding one of the transcriptions in the set of spoken training utterances by a variational autoencoder (VAE), and each speaker embedding in the set of speaker embeddings may be extracted from a corresponding one of the non-synthesized phonetic representations in the set of spoken training utterances.

[0008] In some implementations, the method also includes, prior to determining the adversarial loss term at each of the plurality of output steps for each training text utterance of the plurality of training text utterances, applying, by the data processing hardware, data augmentation to the corresponding generated synthetic speech representation of the corresponding training text utterance. In these implementations, the applied data augmentation may include at least one of adding noise, adding reverberation, or manipulating timing. In additional implementations, the method also includes, at each of the plurality of output steps for each training text utterance of the plurality of training text utterances, generating, by the data processing hardware, a reference synthetic speech representation of the corresponding training text utterance for output by the reference TTS model, and determining, by the data processing hardware, a consistency loss term based on the synthetic speech representation of the corresponding training text utterance output by the GAN-based TTS model and the reference synthetic speech representation of the corresponding training text utterance output by the reference TTS model. wherein updating the parameters of the GAN-based TTS model is based on the adversarial loss term and the consistency loss term determined at each of the multiple output steps for each training text utterance of the multiple non-speech training utterances, and the parameters of the reference TTS model may remain fixed at each of the multiple output steps for each training text utterance of the multiple training text utterances.

[0009] The method may also include executing, by the data processing hardware, a non-spoken text selection process to obtain non-spoken text utterances among the first portion of the plurality of training text utterances, the text selection process being configured to: obtain a corpus of non-spoken text utterances; for each non-spoken text utterance in the corpus of non-spoken text utterances, determine a first probability associated with the non-spoken text utterance appearing in a domain-specific language model, the domain-specific language model being trained with each transcription in the set of spoken training utterances; determine a second probability associated with the non-spoken text utterance appearing in a background language model, the background language model being trained with every non-spoken text utterance in the corpus of non-spoken text utterances; and determine a score based on the first probability, the second probability, and the number of words appearing in the corresponding non-spoken text utterance. The text selection process is further configured to select a non-spoken text utterance in the corpus of non-spoken text utterances that has an N-best utterance score as the non-spoken text utterance in the first portion of the plurality of training text utterances.

[0010] In some examples, training the speech recognition model includes, at each of a plurality of output steps for each training text utterance of the plurality of training text utterances, encoding, by the speech recognition model, a synthetic speech representation of the corresponding training text utterance output by the GAN-based TTS model; encoding, by the speech recognition model, one of the non-synthetic speech representations selected from the set of spoken training utterances; determining, using a separate adversarial discriminator, a separate adversarial loss term between the encoded synthetic speech representation and the encoded one of the non-synthetic speech representations; and updating parameters of the speech recognition model based on the separate adversarial loss term determined at each of the plurality of output steps for each training text utterance of the plurality of training text utterances. In these examples, at each of the plurality of output steps for each training text utterance of the plurality of training text utterances, the corresponding text utterance and the separate adversarial discriminator may each be conditioned on the same speaker embedding selected from a set of speaker embeddings. Each speaker embedding in the set of speaker embeddings represents a speaker characteristic. Furthermore, one of the non-synthesized speech representations selected from the set of spoken training utterances may include a randomly selected non-synthesized speech representation from the set of spoken training utterances when the corresponding training text utterance includes one of the non-spoken text utterances in the first portion of the plurality of training text utterances, and a non-synthesized speech representation from the set of spoken training utterances paired with a corresponding one of the transcriptions when the corresponding training text utterance includes one of the transcriptions in the second portion of the plurality of training text utterances.

[0011] Another aspect of the present disclosure provides a system for simultaneously training a generative adversarial network (GAN)-based text-to-speech (TTS) model and a speech recognition model. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining a plurality of training text utterances, a first portion of the plurality of training text utterances including unspoken text utterances, and a second portion of the plurality of training text utterances including a transcription of a set of spoken training utterances. Each unspoken text utterance is not paired with any corresponding spoken utterance. Each spoken training utterance including a corresponding transcription is paired with a corresponding unsynthesized speech representation of the corresponding spoken training utterance. For each of the plurality of output steps for each training text utterance of the plurality of training text utterances, the operations also include generating a synthetic speech representation of the corresponding training text utterance for output by the GAN-based TTS model and determining, with a GAN adversarial discriminator, an adversarial loss term indicative of an amount of acoustic noise imbalance in one of the non-synthetic speech representations selected from the set of spoken training utterances relative to the corresponding synthetic speech representation of the corresponding training text utterance. The operations also include updating parameters of the GAN-based TTS model based on the adversarial loss term determined at each of the plurality of output steps for each training text utterance of the plurality of training text utterances.

[0012] This aspect may include one or more of the following optional features: In some implementations, training the speech recognition model further includes training the speech recognition model with training utterance batches, where each training utterance batch includes a portion of a generated synthetic speech representation of a corresponding training text utterance and a portion of a non-synthesized speech representation in the set of spoken training utterances, and a ratio of the portion of the generated synthetic speech representation to the portion of the non-synthesized speech representation in each training utterance batch increases over time during training of the speech recognition model.

[0013] In implementations in which the speech recognition model is trained with training utterance batches, the operations may also optionally include, during training of the speech recognition model with each training utterance batch, generating a respective non-synthesized speech loss term for each non-synthesized speech expression in the portion of the non-synthesized speech expression for the corresponding training utterance batch, and applying an adjustable loss weight to each respective non-synthesized speech loss term generated by the speech recognition model, where the adjustable weight has a magnitude proportional to the ratio of the portion of the generated speech expression to the portion of the non-synthesized speech expression in the corresponding training utterance batch.

[0014] Additionally, while training the speech recognition model with each training utterance batch, the operations may also include, for output by the speech recognition model, determining a first probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech expression at each of a plurality of output steps for each generated synthetic speech expression in the portion of generated synthetic speech expressions associated with the corresponding training utterance batch, and generating a synthetic speech loss term based on the first probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech expression and the corresponding training text utterance from which the corresponding synthetic speech expression is generated. Simultaneously, the operations may include, for output by the speech recognition model, determining a second probability distribution over possible non-synthesized speech recognition hypotheses for the corresponding non-synthesized speech expression at each of a plurality of output steps for each non-synthesized speech expression in the portion of non-synthesized speech expressions associated with the corresponding training utterance batch, and generating a non-synthesized speech loss term based on the second probability distribution over possible non-synthesized speech recognition hypotheses for the corresponding non-synthesized speech expression and a transcription in the set of spoken training utterances paired with the corresponding non-synthesized speech expression.

[0015] In some examples, in each of the multiple outputting steps for each training text utterance of the multiple training text utterances, one of the non-synthesized speech representations selected from the set of spoken training utterances includes a randomly selected non-synthesized speech representation from the set of spoken training utterances when the corresponding training text utterance includes one of the non-spoken text utterances in the first portion of the multiple training text utterances, or a non-synthesized speech representation from the set of spoken training utterances paired with a corresponding one of the transcriptions when the corresponding training text utterance includes one of the transcriptions in the second portion of the multiple training text utterances. In additional examples, each training text utterance is conditioned on a utterance embedding selected from a set of utterance embeddings and a speaker embedding selected from a set of speaker embeddings. Each speaker embedding in the set of speaker embeddings represents a speaker characteristic, and each utterance embedding in the set of utterance embeddings represents an intended prosody. Each training text utterance may be represented by a corresponding phoneme sequence, each utterance embedding in the set of utterance embeddings may be extracted from a corresponding one of the transcriptions in the set of spoken training utterances by a variational autoencoder (VAE), and each speaker embedding in the set of speaker embeddings may be extracted from a corresponding one of the non-synthesized phonetic representations in the set of spoken training utterances.

[0016] In some implementations, the operations also include applying data augmentation to the corresponding generated synthetic speech representation of the corresponding training text utterance prior to determining an adversarial loss term at each of the multiple output steps for each training text utterance of the multiple training text utterances. In these implementations, the applied data augmentation may include at least one of adding noise, adding reverberation, or manipulating timing. In additional implementations, the operations also include generating a reference synthetic speech representation of the corresponding training text utterance for output by a reference TTS model at each of the multiple output steps for each training text utterance of the multiple training text utterances, and determining a consistency loss term based on the synthetic speech representation of the corresponding training text utterance output by the GAN-based TTS model and the reference synthetic speech representation of the corresponding training text utterance output by the reference TTS model. Here, updating a parameter of the GAN-based TTS model is based on the adversarial loss term and the consistency loss term determined at each of the multiple output steps for each training text utterance of the multiple non-speech training utterances. The parameters of the reference TTS model may remain fixed in each of the multiple output steps for each training text utterance of the multiple training text utterances.

[0017] The operations may also include performing a non-spoken text selection process to obtain non-spoken text utterances among the first portion of the plurality of training text utterances, the text selection process being configured to: obtain a corpus of non-spoken text utterances; for each non-spoken text utterance in the corpus of non-spoken text utterances, determine a first probability associated with the non-spoken text utterance appearing in a domain-specific language model, the domain-specific language model being trained with each transcription in the set of spoken training utterances; determine a second probability associated with the non-spoken text utterance appearing in a background language model, the background language model being trained with every non-spoken text utterance in the corpus of non-spoken text utterances; and determine a score based on the first probability, the second probability, and the number of words appearing in the corresponding non-spoken text utterance. The text selection process is further configured to select a non-spoken text utterance in the corpus of non-spoken text utterances having the N best utterance scores as the non-spoken text utterance in the first portion of the plurality of training text utterances.

[0018] In some examples, training the speech recognition model includes, at each of a plurality of output steps for each training text utterance of the plurality of training text utterances, encoding, by the speech recognition model, a synthetic speech representation of the corresponding training text utterance output by the GAN-based TTS model; encoding, by the speech recognition model, one of the non-synthetic speech representations selected from the set of spoken training utterances; determining, using a separate adversarial discriminator, a separate adversarial loss term between the encoded synthetic speech representation and the encoded one of the non-synthetic speech representations; and updating parameters of the speech recognition model based on the separate adversarial loss term determined at each of the plurality of output steps for each training text utterance of the plurality of training text utterances. In these examples, at each of the plurality of output steps for each training text utterance of the plurality of training text utterances, the corresponding text utterance and the separate adversarial discriminator may each be conditioned on the same speaker embedding selected from a set of speaker embeddings. Each speaker embedding in the set of speaker embeddings represents a speaker characteristic. Furthermore, one of the non-synthesized speech representations selected from the set of spoken training utterances may include a randomly selected non-synthesized speech representation from the set of spoken training utterances when the corresponding training text utterance includes one of the non-spoken text utterances in the first portion of the plurality of training text utterances, and a non-synthesized speech representation from the set of spoken training utterances paired with a corresponding one of the transcriptions when the corresponding training text utterance includes one of the transcriptions in the second portion of the plurality of training text utterances.

[0019] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0020] [Figure 1]FIG. 1 is a schematic diagram of an exemplary speech recognition system including a speech recognition model. [Figure 2A] FIG. 1 is a schematic diagram of a recurrent neural network transducer (RNN-T) model architecture. [Figure 2B] 1 is a schematic diagram of an attention-based encoder-decoder (AED) model architecture. [Figure 3A] FIG. 1 is a schematic diagram of an exemplary training process for training a generative adversarial network (GAN)-based text-to-speech (TTS) model and a speech recognition model in parallel. [Figure 3B] FIG. 2 is a schematic diagram of an exemplary training process for promoting the speech recognition model of FIG. 1 to learn consistency predictions for both non-synthetic and synthetic speech. [Figure 3C] FIG. 2 is a schematic diagram of an exemplary training process for facilitating domain-invariant learning by the speech recognition model of FIG. 1. [Figure 4] FIG. 2 is a schematic diagram of an example selection process for selecting non-spoken text utterances for training the GAN-based TTS model and the speech recognition model of FIG. [Figure 5] 1 is a flowchart of an example sequence of operations for a method of training a generative adversarial network (GAN)-based text-to-speech (TTS) model and a speech recognition model in parallel. [Figure 6] FIG. 1 is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0021] Like numbers in the various drawings indicate like elements.

[0022] Automated speech recognition has made tremendous progress with the introduction of sequence-to-sequence (Seq2Seq) models, which map from audio to character sequences. One challenge in developing end-to-end (E2E) deep learning-based ASR models is that the parameters of the ASR model tend to overfit to the training data, causing the ASR model to have difficulty generalizing to unseen data when the training data is not sufficiently large. Furthermore, E2E ASR models tend to perform less well than traditional speech recognition systems for long and noisy speech audio. As a result, training an ASR model with a relatively large training dataset of transcribed speech data improves the accuracy of the ASR model.

[0023] Text-to-speech (TTS) or speech synthesis systems have successfully applied Seq2Seq models to obtain state-of-the-art, natural-sounding, synthetic speech that may be indistinguishable from human speech to the human ear. Advantageously, non-spoken text utterances, or text-only data, can be easily and inexpensively acquired to generate synthetic speech for improving ASR model training. For example, non-spoken text utterances can be used to increase the volume of a training dataset, and non-spoken text utterances can increase the linguistic diversity of the training data without the drawbacks of having to obtain transcribed speech (e.g., human speech audio and corresponding transcriptions). While the aggregate quality of synthetic speech produced by modern TTS systems is quite high, synthetic speech exhibits much less noise variation and minimal speech stuttering than non-synthetic (real / human) speech. As a result, training ASR models exclusively on synthetic speech data presents challenges in generalizing to real speech utterances during inference.

[0024] Implementations herein are directed to combining generative adversarial networks (GANs) with data augmentation techniques to increase the acoustic diversity of synthetic speech produced by a TTS model for training an ASR model in parallel. As will become apparent, parameters of a GAN-based TTS model can be updated to produce synthetic speech representations that have acoustic similarity to non-synthetic speech representations recorded under hostile acoustic environments. Here, the synthetic speech may drastically limit the amount of labeled human speech required to train the model while also providing flexibility to move the ASR model between different domains.

[0025] Furthermore, data augmentation techniques may be applied to the synthetic speech representations output from the GAN-based TTS model to closely match the acoustic noise imbalance associated with real / human speech. Data augmentation techniques may include adding / injecting noise (e.g., via multi-style training (MTR)), adding reverberation, and / or manipulating timing (e.g., via Spectral Augmentation). Implementations herein are further directed to generating contrastive language models for selecting non-spoken text utterances for use in ASR training. That is, the use of contrastive language models can improve the efficiency of large-scale non-spoken text utterance training.

[0026] 1 illustrates an automated speech recognition (ASR) system 100 that implements an ASR model 200 that resides on a user device 102 of a user 104 and / or on a remote computing device 201 (e.g., one or more servers of a distributed system executing in a cloud computing environment) that communicates with the user device 102. Although the user device 102 is shown as a mobile computing device (e.g., a smartphone), the user device 102 may correspond to any type of computing device, such as, but not limited to, a tablet device, a laptop / desktop computer, a wearable device, a digital assistant device, a smart speaker / display, a smart appliance, an in-vehicle infotainment system, or an Internet of Things (IoT) device.

[0027] The user device 102 includes an audio subsystem 108 configured to receive utterances 106 spoken by the user 104 (e.g., the user device 102 may include one or more microphones for recording the spoken utterances 106) and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the ASR system 100. In the illustrated example, the user speaks each utterance 106 in natural English for the phrase "What is the weather in New York City?", and the audio subsystem 108 converts the utterance 106 into a corresponding acoustic frame 110 for input to the ASR system 100. The ASR model 200 then receives the acoustic frame 110 corresponding to the utterance 106 as input and generates / predicts a corresponding transcription (e.g., recognition result / hypothesis) 120 of the utterance 106 as output. In the illustrated example, the user device 102 and / or the remote computing device 201 also execute a user interface generator 107 configured to present a representation of a transcription 120 of the utterance 106 to the user 104 of the user device 102. In some configurations, the transcription 120 output from the ASR system 100 is processed by a natural language understanding (NLU) module executing on the user device 102 or the remote computing device 201, for example, to execute user commands. Additionally or alternatively, a text-to-speech system (e.g., executing on any combination of the user device 104 or the remote computing device 201) may convert the transcription into synthesized speech for audible output by another device. For example, the original utterance 106 may correspond to a message the user 104 is sending to a friend, where the transcription 120 is converted into synthesized speech for audible output to the friend, who should hear the message conveyed in the original utterance 106.

[0028] 2A and 2B, the ASR model 200 may include an end-to-end (E2E) sequence-to-sequence model, such as a frame-alignment-based transducer model 200a (FIG. 2A) or an attention-based encoder-decoder (AED) model 200b (FIG. 2B). The ASR model 200 may provide E2E speech recognition by integrating acoustic, pronunciation, and language models into a single neural network, without requiring a lexicon or a separate text normalization component. Various structures and optimization mechanisms can increase accuracy and reduce model training time.

[0029] Referring to FIG. 2A , an exemplary frame alignment-based transducer model 200a includes a recurrent neural network transducer (RNN-T) model architecture that adheres to latency constraints associated with interactive applications. The RNN-T model 200a offers a small computational footprint and uses fewer memory requirements than traditional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device 102 (e.g., no communication with a remote server is required). The RNN-T model 200a includes an encoder network 210, a prediction network 220, and a collaboration network 230. The encoder network 210 is generally similar to an acoustic model (AM) in traditional ASR systems and includes a recurrent network consisting of stacked long short-term memory (LSTM) layers. For example, the encoder receives a sequence of d-dimensional feature vectors (e.g., acoustic frames 110 ( FIG. 1 )) x=(x1, x2,..., x T ), where

[0030]

number

[0031] and at each time step, a higher-level representation is generated.

[0032]

number

[0033] It is shown as follows.

[0034] Similarly, the prediction network 220 is also an LSTM network, which, like a language model (LM), computes the sequence of non-empty symbols output so far by the final softmax layer 240, i.e., y 0 , ..., y ui-1 , a dense representation

[0035]

number

[0036] Finally, using an RNN-T model architecture, the representations produced by the encoder and predictor networks 210, 220 are combined by a joint network 230. The joint network then

[0037]

number

[0038] , which is a distribution over the next output symbol. In other words, at each output step (e.g., time step), the collaborative network 230 generates a probability distribution over possible speech recognition hypotheses. Here, a "possible speech recognition hypothesis" corresponds to a set of output labels, each representing a symbol / character in a specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters of the English alphabet, and one label specifying a space. Thus, the collaborative network 230 may output a set of values ​​indicating the likelihood of each occurrence of a given set of output labels. This set of values ​​may be a vector and may indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels may include wordpieces and / or entire words in addition to or instead of graphemes. The output distribution of the collaborative network 230 may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output y of the collaborative network 230 i may contain 100 different probability values, one for each output label. The probability distributions can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and / or words) in the beam search process (e.g., by softmax layer 240) to determine transcription 120.

[0039] The softmax layer 240 may use any technique for selecting the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the model 200a in the corresponding output step. In this way, the RNN-T model 200a does not make a conditional independence assumption; rather, the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model 200a assumes that the output symbols are independent of future acoustic frames 110, which allows the RNN-T model to be used in a streaming manner.

[0040] In some examples, the encoder network 210 of the RNN-T model 200a consists of eight 2,048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. The prediction network 220 may have two 2,048-dimensional LSTM layers, each also followed by a 640-dimensional projection layer. Finally, the collaboration network 230 may also have 640 hidden units. The softmax layer 240 may consist of a unified word piece or grapheme set generated using all unique word pieces or graphemes in multiple training text utterances 302 (FIG. 3A).

[0041] 2B, an exemplary AED model 200b associated with a Listen, Attend and Spell (LAS) model architecture provides a single neural network that includes a listener encoder module 211, which is similar to a traditional acoustic model, an attendant model 221 that acts as an alignment model, and a decoder 231, which is similar to a language model in traditional systems. Specifically, the listener encoder module 211 receives input features (e.g., acoustic frame 110 (FIG. 1)), i.e., x, and converts them into a higher-level feature representation, i.e., h enc The encoded feature representation, i.e., h encThis process of generating f can be done for each of several input frames, each representing a different input time step. These time steps are denoted below by the subscript u. Thus, given a set of frames {f,f,f,...f u}, the corresponding set of encoded outputs {h1,h2,h3,...h u} is possible.

[0042] The output of the encoder 211 is passed to an attendant model 221, which, like the dynamic time warping (DTW) alignment module, enc Which encoder features in i In some examples, the attender 221 is referred to herein as an attender neural network or attention module 221. The attender 221 generates a context output c for each of a plurality of output steps i. i Each context output vector c i For u, the attender 221 can compute attention based on the encoding for one or more input steps u, e.g., the encoding for the current input step as well as the encoding for the previous input step. For example, the attender 221 can compute attention based on the set of all encoder outputs of an utterance, e.g., the set {h1, h2, h3, ... h u} throughout, attention context output c i The attention context vector may be a vector that represents a weighted summary of the current and previous encodings for a frame (e.g., part) of the utterance being recognized.

[0043] Finally, the output of the attender 221 is passed to the decoder 231, which uses the attention context (e.g., context vector or attention distribution) output by the attender 221, i.e., c i , as well as the previous prediction, i.e., y iThe decoder output is the previous unit, i.e. {y i-1 ,...,y0}, and the current subword unit, i.e., y, given the input, i.e., x. i Probability distribution over y, i.e., P(y i |y i-1 ,...,y0,x). Thus, at each output step, decoder 231 generates a probability distribution over the possible speech recognition hypotheses. Similar to RNN-T model 200a described above with reference to FIG. 2A, a "possible speech recognition hypothesis" corresponds to a set of output symbols, each representing a symbol / character in the specified natural language.

[0044] Although not shown, model 200 may include a softmax layer that receives the output of decoder 231. In some implementations, the softmax layer is separate from decoder 231 and receives the output from decoder 231, i.e., y i , and the output of the softmax layer is then used in the beam search process to select the orthographic elements. In some implementations, the softmax layer is integrated with the decoder 231, so that the output y i represents the output of the softmax layer.

[0045] The decoder 231 and / or associated softmax layer may be trained to output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels. This set of values ​​may be a vector, indicating a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels may include wordpieces and / or entire words in addition to, or instead of, graphemes. The output distribution of the decoder 231 and / or softmax layer may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the decoder output y i , or output y i The output of a softmax layer that receives and processes σ may include 100 different probability values, one for each output label. The probability distributions can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and / or words) in a beam search process to determine transcription 120.

[0046] 3A-3C illustrate exemplary training processes 300, 300a-300c, for jointly training a generative adversarial network (GAN)-based text-to-speech (TTS) model 310 and a speech recognition model 200. The training process 300 may be executed on the remote computing device 201 of FIG. 1. The speech recognition model 200 may be referred to as an ASR model 200. Training the GAN-based TTS model 310 facilitates learning by the model 310 to produce synthetic speech expressions 306 that have acoustic similarities to non-synthesized speech expressions 304 recorded under a hostile acoustic environment. Here, the synthetic speech may drastically limit the amount of labeled human speech required to train the ASR model 200 while also providing flexibility in translating the ASR model 200 across different domains. Furthermore, the training process 300 facilitates the ASR model 200 learning consistency predictions for each of non-synthetic speech (e.g., real / human speech), synthetic speech resulting from transcribed speech, and synthetic speech resulting from non-spoken text utterances that have not been paired with any corresponding audio / speech.

[0047] 3A , the exemplary training process 300a first obtains a plurality of training text utterances 302, 302A-N, which the GAN-based TTS model 310 converts into a synthetic speech representation 306 for training the ASR model 200. A first portion of the plurality of training text utterances 302 includes an unspoken text utterance 302a, and a second remaining portion of the plurality of training text utterances 302 includes a transcription 302b from a set of spoken training utterances 305, 305a-n. Here, each unspoken text utterance 302a is text-only data, i.e., unpaired data such that the unspoken text utterance 302 is not paired with any corresponding spoken utterance or audible sound. Meanwhile, each spoken training utterance 305 in the set of spoken training utterances 305 includes a corresponding transcription 302b paired with a corresponding unsynthesized speech representation 304 of the corresponding spoken training utterance 305. For example, each non-synthesized speech representation 304 may be manually transcribed by a human listener. Thus, each transcription 302b may correspond to a transcribed version of the corresponding non-synthesized speech representation 304, such that each spoken training utterance 305 includes a respective "paired data" 302b, 304.

[0048] The set of spoken training utterances 305 may be stored in a data store XX on memory hardware of the system (e.g., remote computing device 201 of FIG. 1 ). In some examples, each spoken training utterance 305 in the set of spoken training utterances 305 is associated with a particular domain for training the ASR model 200. For example, the ASR model 200 may be trained to recognize speech in domains such as music, navigation, weather, career fields, education training, as well as language-specific and / or multilingual domains.

[0049] Each training text utterance 302 input to the GAN-based TTS model 310 may be conditioned on a speaker embedding, i.e., z, and an utterance embedding, i.e., u, for conversion to synthetic speech 306 having a particular speaking style associated with the speaker embedding, i.e., z, and an intended prosody associated with the utterance embedding, i.e., u. The utterance embedding, u, may be selected from a set of utterance embeddings, each representing an intended prosody. For example, each utterance embedding, u, may encode prosodic features of syllable duration, pitch contour, and our energy contour. In some examples, when the corresponding training text utterance 302 includes a transcription 302b in a spoken training utterance 305, the utterance embedding, u, is extracted as a latent feature from the corresponding non-synthetic speech representation 304 using a variational autoencoder (VAE). The VAE may incorporate a hierarchical structure. In a further example, when the corresponding training text utterance 302 includes an unspoken text utterance 302a, the unspoken text utterance 302a is conditioned on a randomly assigned utterance embedding u previously extracted from one of the unsynthesized speech representations 304 in the set of training utterances spoken using the VAE.

[0050] Each speaker embedding, i.e., z, may include a D vector previously extracted from the set of spoken training utterances 305 using a separately trained speaker encoder (not shown), where the D vector may be extracted from the unsynthesized speech representation 304 of the spoken training utterance 305 using a speaker encoder neural network, such that the D vector encodes the speaker characteristics of the speaker who spoke the spoken training utterance 305 into a fixed-length vector. Thus, each speaker embedding z may be selected from a set of speaker embeddings z previously extracted and stored (e.g., in the data store 202). Each speaker embedding z in the set of speaker embeddings represents the speaker characteristics of the speaker who spoke the corresponding spoken training utterance 305. A previously extracted speaker embedding z may be randomly assigned to each training text utterance 302. In some examples, when the corresponding training text utterance 302 includes a transcription 302b in the spoken training utterance 305, the transcription 302b is conditioned on a speaker embedding z previously extracted from the corresponding unsynthesized speech representation 304.

[0051] In some implementations, the TTS model 310 includes an encoder 312, a decoder 314, and a postnet 316 that cooperate to process the training text utterances 302 to generate time-domain audio waveforms, which are audio waveforms that define the amplitude of an audio signal over time.

[0052] The encoder 312 may be an encoder neural network 312 configured to receive the training text utterances 302 as a sequence of characters and generate a fixed-length context vector 313 for each Mel-frequency spectrogram 315 that the decoder 314 subsequently generates. In some examples, the characters of the training text utterances 302 include phonemes based on a pronunciation model. For example, each training text utterance 302 may be represented as a sequence of phoneme inputs based on an American English pronunciation model. Here, the fixed-length context vector 313 (e.g., phoneme encoding) defines features that appear at specific positions in the sequence of characters (e.g., a sequence of phonemes). The features model the context in which each character in the sequence of characters appears in each training text utterance 302.

[0053] In some configurations, the encoder neural network 312 includes one or more convolutional layers followed by a bidirectional long short-term memory ("LSTM") layer. Each convolutional layer can be followed by batch normalization and a rectified linear unit ("ReLU"), and the bidirectional LSTM layer can be configured to process the hidden features generated by the final convolutional layer to generate a sequential feature representation of the sequence of characters. The sequential feature representation represents the local structure of the sequence of characters around a particular character. The sequential feature representation can include a sequence of feature vectors.

[0054] The encoder neural network 312 may also include an attention network. The attention network may be configured to receive sequential feature representations from another component of the encoder neural network 312, e.g., a bidirectional LSTM layer, and process the sequential feature representations to generate a fixed-length context vector 313 for each output step of the decoder 314. That is, the attention network may generate a fixed-length context vector 313 for each frame of the mel-frequency spectrogram 315 that the decoder 314 subsequently generates. A frame is a unit of the mel-frequency spectrogram that is based on a small portion of the input signal, e.g., a 10-millisecond sample of the input signal.

[0055] In some implementations, the decoder 314 includes a decoder neural network 314 configured to receive as input the fixed-length context vectors 313 generated by the encoder neural network 312 and to generate as output for each fixed-length context vector 313 a corresponding frame of a Mel-frequency spectrogram 315. A Mel-frequency spectrogram is a frequency-domain representation of a sound. The Mel-frequency spectrogram emphasizes lower frequencies that are critical to speech intelligibility and de-emphasizes high frequencies that are dominated by fricatives and other noise bursts and generally do not need to be modeled with high fidelity. In some implementations, the Mel-frequency spectrograms 315 generated by the decoder neural network 314 have a frame length of 12.5 milliseconds.

[0056] The postnet 316 may be a component of the decoder neural network 314 and may be configured to refine the acoustic characteristics of the mel-frequency spectrogram 315 generated by the decoder neural network 314. In some examples, the postnet 316 may be a convolutional postnet 316 with one or more convolutional layers that processes the predicted mel-frequency spectrogram 315 for each output step and predicts a residual to be added to the predicted mel-frequency spectrogram 315. Each convolutional layer except the final convolutional layer may be followed by batch normalization and tanh activation. The convolutional layers may be regularized using dropout with probability 0.5. The residual is added to the predicted mel-frequency spectrogram 315 to produce a resulting synthetic speech representation 306 for each corresponding training text utterance 302.

[0057] 3A , the GAN-based TTS model 310 used to train the ASR model 200 is pre-trained to be capable of converting input training text utterances 302 into corresponding synthetic speech representations 306. However, the implementation herein is directed to a training process 300a that fine-tunes the parameters of the post-net 316 to “noisify” the resulting synthetic speech representations 306 to match the acoustic noise imbalance found in non-synthetic speech representations 304 spoken by real humans in a hostile acoustic environment. Thus, the pre-trained GAN-based TTS model 310 runs as a pseudo-inference stage during the training process 300a to generate / predict the corresponding synthetic speech representation 306, and the training process 300a aims to fine-tune the post-net 316 parameters to produce a synthetic speech representation 306 with an acoustic that matches that of the non-synthetic speech representation 304. The shading of the PostNet 316, GAN adversarial discriminator 318 in the GAN-based TTS model 310, and the ASR model 200 shown in FIG. 1 indicates that these are the only components trained by the training process 300a.

[0058] In each of a plurality of output steps for each training text utterance 302 of the plurality of training text utterances 302, the training process 300a generates a synthetic speech representation 306 of the corresponding training text utterance 302 for output by the GAN-based TTS model 310 and uses an adversarial discriminator 318 to determine an adversarial loss term 320 indicative of the amount of acoustic noise imbalance in the non-synthesized speech representation 304 relative to the corresponding synthetic speech representation 306 of the corresponding training text utterance 302. Here, the adversarial discriminator 318 compares the acoustic noise characteristics of the non-synthesized speech representation 304 with the synthetic speech representation 306 output by the GAN-based TTS model 310 in the output step. In other words, the adversarial discriminator 318 is tasked with analyzing the synthetic speech representation 306 in general realism by distinguishing between the non-synthesized speech representation 304 (e.g., actual / human audible speech) and the synthetic speech representation 306. The training process 300a is configured to update parameters of the GAN-based TTS model 310 based on an adversarial loss term 320 determined at each of a plurality of output steps for each training text utterance 302 of a plurality of training text utterances. Specifically, the training process 300a updates parameters of the PostNet 316 by backpropagating the adversarial loss term 320 through the PostNet 316 to teach the PostNet 316 to make the resulting synthesized speech representations 306 acoustically similar to the non-synthesized speech representations 304 in the set of spoken training utterances 305.

[0059] At each output step, the non-synthesized speech representation 304 input to the adversarial discriminator 318 for comparison with the synthetic speech representation 306 may be selected from the set of spoken training utterances 305. For example, if the synthetic speech representation 306 input to the adversarial discriminator 318 is generated from a training text utterance 302 that includes one of the non-spoken text utterances 302a, the non-synthesized speech representation 304 is randomly selected from the set of spoken training utterances 305 for comparison in the adversarial discriminator 318 because the non-spoken text utterance 302a is not initially paired with any corresponding spoken audio. On the other hand, when the synthesized speech representation 306 input to the adversarial discriminator 318 is generated from a training text utterance 302 that includes one of the transcriptions 302b of the spoken training utterance 305, the training process 300a may select a non-synthesized speech representation 304 from the set of spoken training utterances 305 that is paired with the corresponding transcription 302b for comparison in the adversarial discriminator 318.

[0060] In some examples, the training process 300a further applies data augmentation to the synthetic speech representation 306 generated by the GAN-based TTS model 310 prior to determining the adversarial loss term 320 in each of the multiple output steps. The purpose of data augmentation is to regularize the training of the ASR model 200 on the synthetic speech representation 306 by further facilitating the generation of synthetic speech with audio that matches the non-synthetic speech in an adversarial acoustic environment, thereby preventing overfitting due to acoustic mismatches between the non-synthetic and synthetic speech representations 304, 306. In some examples, the postnet 316 in the TTS model 310 injects background noise into the synthetic speech representation to match the acoustics of the non-synthetic speech representation 304 of the spoken training utterance 305. In other words, data augmentation enables the TTS model 310 to generate synthetic speech representations 306 that are expected to be consistent with the non-synthetic speech representation 304. Data augmentation may include at least one of adding / injecting noise, adding reverberation, or manipulating the timing of the synthetic speech representation 306. One data augmentation technique includes using multi-style training (MTR) to inject various environmental noises into the synthetic speech representation 304. Another data augmentation technique that the training process 300a may apply in addition to, or instead of, MTR includes using spectral augmentation (SpecAugment) to make the acoustics of the synthetic speech representation 306 more closely resemble the adversarial acoustics of the unsynthesized speech representation 304 of the spoken training utterance 305. Combined, MTR and SpecAugment may inject noise into the synthetic speech representation 304 by inserting a random external noise source in front of and tiling it over the representation 304 in time, and filtering the noise-injected synthetic speech representation 304 prior to training the ASR model 200.

[0061] In an additional implementation, in each of a plurality of output steps for each training text utterance 302, the training process 300a generates a reference synthesized speech representation 306 of the corresponding training text utterance 302 for output by the reference TTS model 322.ref , generating a synthetic speech representation 306 of the corresponding training text utterance 302 output by the GAN-based TTS model 310 and a reference synthetic speech representation 306 of the same corresponding training text utterance 302 output by the reference TTS model 322. ref 3A illustrates a consistency loss term 324 determined based on the synthesized speech representation 306 output by the GAN-based TTS model 310 and the reference synthesized speech representation 306 output by the reference TTS model 322 for the same training text utterance 302 at a given time step. ref and receiving two speech representations 306, 306 ref As with the adversarial loss term 320, the training process 300a is configured to update the parameters of the GAN-based TTS model 310 based on the consistency loss term 324 determined at each of the multiple output steps for each training text utterance 302 of the multiple training text utterances. Specifically, the training process 300a updates the parameters of the post-net 316 by back-propagating both the adversarial loss term 320 and the consistency loss term (e.g., MSE loss) 324 through the post-net 316 to teach the post-net 316 to make the resulting synthetic speech representations 306 acoustically similar to the non-synthesized speech representations 304 in the set of spoken training utterances 305.

[0062] By backpropagating the consistency loss term 324 through the postnet 316 of the GAN-based TTS model 310, the training process 300a is constrained to produce synthetic speech representations 306 output by the GAN-based TTS model 310 that preserve their linguistic information. Otherwise, updating the parameters of the postnet 316 to promote acoustic diversity based solely on the adversarial loss 320 may result in a loss of linguistic diversity across the synthetic speech representations 306. The use of non-spoken text utterances 302a promotes increased linguistic diversity to enable training of the ASR model 200 with unknown words and sequences not present in the spoken training utterances 305. Notably, the parameters of the reference TTS model 322 remain fixed, allowing the reference TTS model 322 to consistently generate synthetic speech representations 306 that preserve linguistic diversity. The reference TTS model 322 may include the same architecture as the GAN-based TTS model 310. For example, when the training process 300a initializes, the TTS model 310 may be a replica trained on the same training dataset.

[0063] In some examples, the first portion of the plurality of training text utterances 302, including the non-spoken text utterances 302a used to jointly train the GAN-based TTS model 310 and the ASR model 200, is selected to optimize the training process 300 of FIGS. 3A-3C . That is, the non-spoken text utterances are easy and inexpensive to obtain, and some non-spoken text utterances 302a are more suitable for training the ASR model 200 than others. In some examples, the non-spoken text utterances 302a are selected to be associated with the domain in which the ASR model 200 is being trained. In this example, if the ASR model 200 is being trained to recognize speech in the musical domain, the non-spoken text utterances 302a used in medical terminology would not be suitable for use in training the ASR model 200. FIG. 4 provides an exemplary non-spoken text selection process 400 for obtaining the non-spoken text utterances 302a in the first portion of the plurality of training text utterances 302. Specifically, the non-spoken text selection process 400 is configured to select the non-spoken text utterances 302a from a corpus of non-spoken text 402. The corpus of non-spoken text 402 includes a large number of non-spoken text utterances 302 from across a large range of domains and includes much greater linguistic diversity than the transcriptions 302b in the set of spoken training utterances 305. As mentioned above, the set of spoken training utterances 305 may be domain-specific in that it relates to the same domain for which the ASR model 200 is trained. The corpus of non-spoken text 402 may be stored in the same or a different data store 202 as the spoken training utterances 305. The corpus of non-spoken text 402 may change dynamically to incorporate new non-spoken text utterances 302a.Simply using all the unspoken text utterances 302 a in the unspoken text corpus 402 ( FIG. 4 ) is not feasible for the following reasons: i) for each sentence, the speech modality requires much more memory to be encoded than the text, thereby making it impractical to convert all the text in the corpus 402; ii) converting between speech and text modalities in TTS inference and ASR training is also computationally intensive; and iii) the vast amount of difference between the transcripts 302 b in the spoken training utterances 305 and the unspoken text utterances 302 a in the unspoken text corpus 402 requires an intelligent strategy to balance their contributions.

[0064] The non-spoken text selection process 400 aims to select a subset of available non-spoken text utterances 302 a from the non-spoken text corpus 402 as data for TTS synthesis. Specifically, the process 400 aims to improve the match between the selected subset of available non-spoken text utterances 302 a and the target domain, which reduces the computational resources required to utilize large amounts of non-domain-specific data. Thus, the process 400 reduces computational and memory costs by selecting the non-spoken text utterances 302 a that best match the particular domain that the ASR model 200 is trained to learn.

[0065] During the first stage (Stage 1), the non-spoken text selection process 400 assembles two language models 410, 420 to enable contrastive selection of non-spoken text utterances 302a. Here, a domain-specific language model (LM) 410 is trained with each transcript 302b in the set of spoken training utterances 305. The set of spoken training utterances 305 is assumed to belong to the specific domain for which the ASR model 200 is trained. Meanwhile, a background LM 420 is trained with each non-spoken text utterance 302a in the entire non-spoken text corpus 402. As mentioned above, the non-spoken text corpus 402 spans many different domains. In some examples, the first stage uses n-gram language model training to assemble the two language models 410, 420. In other examples, the first stage uses neural network language model training to assemble the two language models 410, 420.

[0066] During the second state (Stage 2), the unspoken text selection process 400 calculates a first probability associated with each word in the unspoken text utterance 302a that appears in the domain-specific LM 410, i.e.,

[0067]

number

[0068] and determining a second probability associated with each word in the non-spoken text utterance 302a that appears in the background LM 420, i.e.,

[0069]

number

[0070] The two control LMs 410, 420 are used to evaluate each non-spoken text utterance 302a in the non-spoken text corpus 402 by determining: Then, for each non-spoken text utterance 302a in the non-spoken text corpus 402, the process 400 determines a score, S, in a scorer 430 based on the first probability, the second probability, and the number of words, #(w), that appear in the corresponding non-spoken text utterance 302a. For example, the score S for each non-spoken text utterance 302a may be calculated as follows:

[0071]

number

[0072] After determining the scores, the non-spoken text selection process 400 selects the non-spoken text utterances 302a with the N best utterance scores S as these non-spoken text utterances 302a best match the particular domain. Instead of using the non-spoken text selection process 400, the non-spoken text utterances 302a may alternatively be sampled from a well-trained language model, such as a large maximum entropy language model, using an available text corpus 402. Under the assumption that this model learns the distribution of the data, a certain number of text utterances 302a may be sampled therefrom for non-spoken text.

[0073] 3B, training process 300b trains ASR model 200 with synthesized speech representations 306 generated in each of a plurality of output steps for each non-spoken training text utterance 302 of a plurality of non-spoken text utterances 302 (e.g., selected by non-spoken text selection process 400 of FIG. 4). Training process 300b also trains ASR model 200 in each of a plurality of output steps for each non-synthesized speech representation 304 in set of spoken training utterances 305. Synthesized speech representations 306 include unpaired synthesized speech representations 306a and paired synthesized speech representations 306b. The unpaired synthetic speech representation 306a includes TTS audio converted by the GAN-based TTS model 310 from the unspoken text utterance 302a (i.e., the portion of the training text utterance 302 that is not paired with any corresponding spoken utterance), and the paired synthetic speech representation 306b includes TTS audio converted by the GAN-based TTS model 310 from the transcription 302b in the set of spoken training utterances 305 (i.e., the portion of the training text utterance 302 that is paired with the corresponding unsynthesized speech representation 304 of the spoken training text utterance 305).

[0074] Thus, each paired synthetic speech representation 306b is paired with a corresponding non-synthetic speech representation 304 of the same corresponding spoken training utterance 305, and the transcription 302b initially paired with the non-synthetic speech representation 304 of the corresponding training utterance 305 serves as both (1) an input to the GAN-based TTS model 310 to generate the paired synthetic speech representation 306b of the corresponding training utterance 305, and (2) a ground truth transcription 302b for the training process 300b in generating supervised loss terms 342, 344b between the ground truth transcription 302b and each of the non-synthetic speech recognition hypotheses 311 output by the ASR model 200 and the paired synthetic speech recognition hypotheses 312b output by the ASR model 200. Meanwhile, the training process 300b uses each unspoken text utterance 302a converted into a corresponding unpaired synthetic speech representation 306a by the GAN-based TTS model 310 to generate a supervised loss term 344b between the corresponding unspoken text utterance 302a and the unpaired synthetic speech recognition hypotheses 312a output by the ASR model 200.

[0075] The ASR model 200 receives as input a sequence of features / vectors (e.g., Mel-frequency spectrogram frames) (e.g., acoustic frames 110 of FIG. 1 ) for each corresponding spoken training utterance 305 as an unsynthesized speech representation (x) 304, and generates as output, for each of a plurality of output steps, a first probability distribution 311 over the possible unsynthesized speech recognition hypotheses (y) for the corresponding unsynthesized speech representation (x) 304 of the corresponding spoken training utterance 305. For simplicity, the term “unsynthesized speech recognition result 311” may be used to refer to the first probability distribution 311 over the possible unsynthesized speech recognition hypotheses (y) for the corresponding unsynthesized speech representation (x) 304. The ASR model 200 receives as input a sequence of features / vectors (e.g., Mel-frequency spectrogram frames) (e.g., acoustic frames 110 of FIG. 1 ) for each corresponding spoken training utterance 305 as an unsynthesized speech representation (x) 304 of the ....

[0076]

number

[0077] 306, 306a-b also receive as an output a sequence of features / vectors (e.g., Mel-frequency spectrogram frames) (e.g., acoustic frames 110 of FIG. 1) and, for each of a plurality of output steps, a corresponding synthesized speech representation of the corresponding utterance.

[0078]

number

[0079] 306. For simplicity, the term "synthetic speech recognition result 312" refers to the corresponding synthetic speech representation

[0080]

number

[0081] may be used interchangeably to refer to a second probability distribution 312 over the possible synthetic speech recognition hypotheses (y) for 306. Finally, the ASR model 200 receives as input a synthetic speech representation for each non-spoken text utterance 302a.

[0082]

number

[0083] 306a also receives as an output a sequence of features / vectors (e.g., Mel-frequency spectrogram frames) (e.g., acoustic frames 110 of FIG. 1) and, as an output, for each of a plurality of output steps, a corresponding synthesized speech representation of the corresponding utterance.

[0084]

number

[0085] 306. For simplicity, the term "synthetic speech recognition result 312b" is used to refer to the corresponding synthetic speech representation

[0086]

number

[0087] may be used interchangeably to refer to a second probability distribution 312b over the possible synthetic speech recognition hypotheses (y) for 306.

[0088] Similar to training a conventional end-to-end sequence-to-sequence ASR model, the training process 300b generates, for output by the ASR model 200 at each of a plurality of output steps, a non-synthetic speech loss term (J) based on the ground truth transcription 302b and the non-synthetic speech recognition result 311(y) for the corresponding non-synthetic speech representation (x) 304 of the corresponding spoken training utterance 305. real In the illustrated example, the training process 300b receives the ground truth transcription 120b and the first probability distribution 311 and generates a first supervised loss term (J real Execute a supervised loss term module 340 configured to output a loss term (θ) 342.

[0089] Moreover, the training process 300b may, at each of a plurality of output steps, generate a synthetic speech loss term (J) based on a corresponding one of the unspoken text utterance 302a or the ground truth transcription 302b. tts (θ)) 344, a second supervised loss term including 344a-b and the corresponding synthetic speech representation of the corresponding utterance

[0090]

number

[0091] 306, 306a-b) and generates a second probability distribution 312, 312a-b over the possible synthetic speech recognition hypotheses (y) for the unspoken text utterance 302a. In the illustrated example, an unpaired synthetic speech loss term 344a is associated with each unpaired synthetic speech representation 306a and the corresponding unspoken text utterance 302a, and a paired synthetic speech loss term 344b is associated with each paired synthetic speech representation 306b and the corresponding ground truth transcript 302b. In the illustrated example, the training process 300b receives the unspoken text utterance 120a, the ground truth transcript 120b, and the second probability distributions 312, 312a-b, and generates the synthetic speech loss terms (J tts Executes a supervised loss term module 340 configured to output (θ) 344, 344a-b.

[0092] The supervised loss term module 340 calculates the non-synthetic speech loss term (J real (θ))342, unpaired synthesized speech loss term (J tts (θ))344a, and the pairwise synthesis speech loss term (J tts (θ)) 344b may be fed back to the ASR model 200. For example, the training process 300b may train the ASR model 200 using a stochastic optimization algorithm, such as stochastic gradient descent, to train the model 200 through backpropagation. Here, the stochastic optimization algorithm uses the loss terms 342, 344a, 344b to define respective loss functions (e.g., cross-entropy loss functions) based on the difference between the actual outputs of the neural network (e.g., the non-synthetic and synthetic speech recognition results 311, 312a, 312b) and the desired outputs (e.g., the unspoken text utterance 302a and the ground truth transcription 302b). For example, the loss functions are calculated for a batch of training examples and then differentiated with respect to each weight in the model 200. In batch training, the non-synthetic speech loss term (J real (θ)) 342 may correspond to the average loss obtained for each batch of unsynthesized speech representations 304, and the synthetic speech loss term (J ttsEach of (θ) 344, 344a-b may correspond to the average loss obtained for a respective batch of synthetic speech representations 306a, 306b. Furthermore, model 200 may be trained in parallel on corresponding batches of unsynthesized and synthetic speech representations 304, 306a, 306b, such that supervised loss term module 340 outputs corresponding unsynthesized and synthetic speech loss terms 342, 344a, 344b in parallel.

[0093] In some configurations, the training process 300b, for each training utterance pairing 304, 306b of a corresponding non-synthesized speech representation 304 and a paired synthetic speech representation 306b of the same corresponding spoken training utterance 305, calculates a consistency loss term (J) for the corresponding training utterance pairing 304, 306b based on a first probability distribution 311 over the possible non-synthesized speech recognition hypotheses and a second probability distribution 312b over the possible synthetic speech recognition hypotheses. cons For example, the training process 300b may utilize a consistency loss term module 350 configured to receive corresponding unsynthesized and synthesized speech recognition results 311, 312b output by the ASR model 200 at each of a plurality of output steps and to determine a consistency loss term 352 for the corresponding training utterance pairings 304, 306b at the output steps.

[0094] In some examples, the training process 300b calculates a Kullback-Leibler divergence (D) between a first probability distribution 311 over the possible non-synthetic speech recognition hypotheses and a second probability distribution 312b over the possible synthetic speech recognition hypotheses associated with the corresponding pairwise synthetic speech representation 306b. KL) to determine a consistency loss term 352. Thus, the consistency loss term 352 determined for each training utterance pairing 304, 306b provides an “unsupervised” loss term that is independent of the accuracy of the ASR model 200 (e.g., independent of the supervised loss terms 342, 344) and may therefore be utilized to update the parameters of the ASR model 200 to promote consistency between the non-synthetic and synthetic speech representations 304, 306b of the same spoken training utterance 305. In other words, the consistency loss term 352 allows the ASR model 200 to learn to behave similarly regardless of whether the spoken training utterance 305 belongs to non-synthetic or synthetic speech, e.g., to make consistency predictions for both non-synthetic (e.g., real / human) and synthetic (e.g., synthetic) speech of the same spoken training utterance 305. In the illustrated example, training process 300b is configured to output corresponding non-synthetic and synthetic speech loss terms 342, 344a-b from supervised loss term module 340 and, in parallel, output consistency loss term 352 from unsupervised loss term module 350.

[0095] In some examples, the training process 300b uses batch training to train the ASR model 200 by training the ASR model 200 on multiple training utterance batches. In these examples, each training utterance batch includes a portion of generated synthetic speech representations 306 and a portion of unsynthesized speech representations 304 in a set of spoken training utterances 305. The portion of generated synthetic speech representations 306 may be further divided into unpaired synthetic speech representations 306a and paired synthetic speech representations 306b, where each paired synthetic speech representation 306b in a given training utterance batch may be paired with a corresponding one of the unsynthesized speech representations 304 of the same corresponding spoken training utterance 305. Each training utterance batch may include a greater proportion of generated synthetic speech representations 306 than unsynthesized speech representations.

[0096] The data augmentation techniques described above may be applied to the synthetic speech representations 306 so that their acoustics match those of the non-synthetic speech representations 304 in a hostile acoustic environment. Additionally, the training process 300b may shuffle the non-synthetic and synthetic speech representations 304, 306 within each training batch of utterances.

[0097] In some implementations, the training process 300b uses curriculum batch training, in which the ratio of the portion of generated synthetic speech representations 306 to the portion of non-synthesized speech representations in each training utterance batch increases over time while training the speech recognition model. In these implementations, the ratio increases by increasing the number of unpaired synthetic speech representations 306a generated in each training batch, i.e., from unspoken text utterances 302a. The training process 300b trains the ASR model 200 with each training utterance batch, and the training process 300b generates, in a supervised loss term module 340, a respective non-synthesized speech loss term 342 for each non-synthesized speech representation 304 in the portion of non-synthesized speech representations for the training utterance batch.

[0098] In some examples, the supervised loss term module 340 applies an adjustable weight to each respective non-synthesized loss term 342 having a magnitude proportional to the ratio of the portion of the generated speech representation to the portion of the non-synthesized speech representation in the corresponding training batch of utterances. Similarly, the training process 300b further generates a respective synthetic speech loss term 344a, 344b for each synthetic speech representation 306a, 306b in the corresponding training batch of utterances in the supervised loss term module 340. As with the non-synthesized loss term 342, the supervised loss term module 340 may further apply an adjustable weight to each respective non-synthesized speech loss term 342b having a magnitude proportional to the ratio of the portion of the generated speech representation to the portion of the non-synthesized speech representation in the corresponding training batch of utterances. Thus, during curriculum batch training, in which the ratio of the generated synthetic speech representations 306 to the unsynthesized speech representations in each training batch gradually increases in each subsequent training utterance batch, the weights applied to the loss terms 342, 344b associated with the training utterance pairings 304, 306b also gradually increase in each subsequent training utterance batch to allow overfitting by the model 200. Similarly, adjustable weights having scales proportional to this scale may also be applied to the consistency loss term 352 in each training utterance batch. In particular, the supervised loss term 350 may apply a fixed weight to each respective unpaired unsynthesized speech loss term 342b during curriculum batch training, regardless of the increasing proportion of unpaired synthetic speech representations 306a in each subsequent training utterance batch.

[0099] 3C , in some implementations, a training process 300 c applies conditional GAN-based domain-invariant training to train the ASR model 200. The training process 300 c may apply domain-invariant training in addition to or instead of the technique described in FIG. 3A for fine-tuning the parameters of the postnet 316 of the GAN-based TTS model 310 based on adversarial and consistency loss terms 320, 324. In the illustrated example, in each of multiple output steps for each training text utterance 302 of the multiple training text utterances 302 of FIG. 1 , the encoder 410 in the ASR model 200 (e.g., the encoder 210 of FIG. 2A or the encoder 211 of FIG. 2B ) converts the synthetic speech representation 306 of the corresponding training text utterance 302 output by the TTS model 310 into a corresponding encoded synthetic speech representation TTS. enc (e.g., synthetic embedding) and one of the non-synthesized speech representations 304 selected from a set of spoken training utterances 305 is encoded into a corresponding non-synthesized speech representation Real enc For the output step when the corresponding training text utterance 302 includes one of the unspoken text utterances 302a, one of the unsynthesized speech representations 304 input to the encoder 410 is randomly selected from the set of spoken training utterances 305, because the unspoken text utterance 302a is not initially paired with any spoken utterance. In contrast, for the output step when the corresponding training text utterance 302 includes one of the transcriptions 302b in the set of spoken training utterances 305, one of the unsynthesized speech representations 304 may include the unsynthesized speech representation 304 in the set of spoken training utterances 305 that is paired with a corresponding one of the transcriptions 302b.

[0100] The training process 300c uses an adversarial discriminator 358 to generate the encoded synthetic speech representation TTS. enc and the encoded non-synthetic speech representation Real encand determine an adversarial loss term 360 between z and . The training process 300c then applies domain-invariant training by updating the parameters of the ASR model 200 based on the determined adversarial loss term 360 at each of a plurality of output steps for each training text utterance 302 of the plurality of training text utterances 302. In some implementations, at each of the plurality of output steps for each training text utterance 302, the corresponding training text utterance 302 and the adversarial discriminator 358 are conditioned on the same speaker embedding z as the corresponding training text utterance 302. Conditioning the corresponding training text utterance 302 on the respective speaker embedding z was discussed above with respect to FIG. 3A .

[0101] A decoder 420 (e.g., decoder / softmax 240 of FIG. 2A or decoder 231 of FIG. 2B) decodes the synthetic speech representation 306 and the non-synthetic speech representation 304 into corresponding probability distributions 311, 312 as described above with reference to FIG. 2B. The shading of the encoder 410 and the separate adversarial discriminator 358 of the GAN in the ASR model 200 shown in FIG. 3C indicates that these are the only components trained with the domain-invariant training aspects applied by the training process 300c.

[0102] 5 is a flowchart of an example sequence of operations for a method 500 of jointly training a generative adversarial network (GAN)-based text-to-speech (TTS) model 310 and a speech recognition model 200. The flowchart may be described with reference to the training processes 300, 300a-c, of FIGS. 3A-3C. At operation 502, the method 500 obtains a plurality of training text utterances 302, 302A-N. A first portion of the plurality of training text utterances 302 includes unspoken text utterances 302a that are not paired with any corresponding spoken utterance. The unspoken text utterances 302a may be selected from a large unspoken text corpus 402, for example, using the unspoken text selection process 400 described in FIG. 4. A remaining second portion of the plurality of training text utterances 302 includes transcriptions 302b in a set of spoken training utterances 305. Each spoken training utterance 305 in the set includes a corresponding transcription 302 b paired with a corresponding non-synthesized speech representation 304 of the corresponding spoken training utterance 305 .

[0103] For each of a plurality of output steps for each training text utterance of the plurality of training text utterances 302, the method 500 generates a synthetic speech representation 306 of the corresponding training text utterance 302 for output by the GAN-based TTS model 310 in operation 504, and in operation 506, the method 500 uses a GAN adversarial discriminator 318 to determine an adversarial loss term 320 indicative of the amount of acoustic noise imbalance in one of the unsynthesized speech representations 304 selected from the set of spoken training utterances 305 relative to the corresponding synthetic speech representation 306 of the corresponding training text utterance 302. For an unpaired synthetic speech representation 306a derived from a first portion of the training text utterances 302, including the unspoken text utterance 302a, the selected one of the unsynthesized speech representations 304 is randomly selected. For a paired synthesized speech representation 306b derived from a second portion of the training text utterance 302 that includes a transcription 302b in the set of spoken training utterances 305, the selected one of the non-synthesized speech representations 304 is the non-synthesized speech representation 304 in the set of spoken training utterances 305 that is paired with the corresponding transcription 302b.

[0104] At operation 508, the method 500 includes updating parameters of the GAN-based TTS model 310 based on the adversarial loss term 320 determined at each of the multiple output steps for each training text utterance 302 of the multiple training text utterances 302. Here, updating the parameters of the GAN-based TTS model 310 may include updating / fine-tuning parameters of a postnet 316 of the TTS model 310 to produce a synthetic speech representation 306 with an acoustic that matches that of the non-synthetic speech representation. In particular, data augmentation may be applied to each synthetic speech representation 306 output from the TTS model 510 to further add noise to match the acoustics of the non-synthetic speech representation 304. A reference synthetic speech representation 306 may be compared to the synthetic speech representation 306 to determine an MSE loss 324 to constrain the postnet 316 to maintain language imbalance across the synthetic speech representation 306. refAt operation 510, the method 500 trains the speech recognition model 200 with the synthesized speech representations 306 generated in each of the plurality of output steps for each training text utterance 302 of the plurality of training text utterances.

[0105] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0106] Non-transitory memory may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transitory memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0107] 6 is a schematic diagram of an exemplary computing device 600 that can be used to implement the systems and methods described herein. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the implementation of the invention as described and / or claimed herein.

[0108] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be mounted on a common motherboard or in other manners as desired. Processor 610 can process instructions for execution within computing device 600, including instructions stored in memory 620 or on storage device 630, for displaying graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 680 coupled to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memory, as desired. Also, multiple computing devices 600 may be connected, each providing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0109] The memory 620 stores information non-transiently within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0110] The storage device 630 is capable of providing mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as memory 620, the storage device 630, or memory on the processor 610.

[0111] The high-speed controller 640 manages bandwidth-intensive operations for the computing device 600, while the low-speed controller 660 manages less bandwidth-intensive operations. Such allocation of duties is merely exemplary. In some implementations, the high-speed controller 640 is coupled to the memory 620, the display 680 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 650, which may receive various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), but may also be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, etc., or to a network device, such as a switch or router, for example, through a network adapter.

[0112] The computing device 600 may be implemented in several different forms, as shown in the figure, such as a standard server 600a, or in a group of servers 600a, such as laptop computers 600b, or many times as part of a rack server system 600c.

[0113] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuitry, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or translatable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0114] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0115] The processes and logic flows described herein can be implemented by one or more programmable processors, also referred to as data processing hardware, that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be implemented by special-purpose logic circuitry, such as FPGAs (field-programmable gate arrays) and ASICs (application-specific integrated circuits). Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, e.g., magnetic, magneto-optical, or optical disks, for storing data, or is operably coupled to a mass storage device to receive data from, transfer data to, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0116] To enable user interaction, one or more aspects of the present disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touch screen, for displaying information to the user, and optionally a keyboard and pointing device, e.g., a mouse or trackball, for allowing the user to provide input to the computer. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0117] Although several implementations have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0118] 100 Automated Speech Recognition (ASR) Systems 102 User Devices 107 User Interface Generator 200 ASR models, speech recognition models, models 201 Remote Computing Devices 202 Datastore 210 Encoder Network, Encoder 211 Listener encoding module, listener encoder module, encoder 220 Prediction Network 221 Attendant Model, Attendant 230 Collaborative Network 231 decoder 240 Final softmax layer, softmax layer, decoder / softmax 310 Generative Adversarial Network (GAN) Based Text-to-Speech (TTS) Model, Model, TTS Model 312 Encoder, Encoder Neural Network 314 Decoder, Decoder Neural Network 316 Postnet 318 Adversarial Discriminator 322 Reference TTS Model 326 Consistency Module 340 Supervised Loss Term Module 350 Consistency Loss Term Module 358 Adversarial Discriminator 402 Corpus, Non-Speech Text Corpus, Text Corpus 410 Language Model, Domain-Specific Language Model (LM), Encoder 420 Language Model, Background LM, Decoder 430 Scorer 600 computing devices 600a Standard Server, Server 600b laptop computer 600c Rack Server System 610 Processor, component 620 Memory, Components 630 Storage Devices, Components 640 High-Speed ​​Interface / Controller, Components 650 High-Speed ​​Expansion Port, Components 660 Low-Speed ​​Interface / Controller, Components 670 Slow Bus 680 display 690 Low-Speed ​​Expansion Port

Claims

1. 1. A computer-implemented method for training a speech recognition model, the method, when executed on data processing hardware, causing the data processing hardware to perform operations, the operations comprising: obtaining a plurality of non-spoken text utterances associated with a target domain that the speech recognition model is trained to learn; obtaining a set of spoken training utterances, each spoken training utterance including a corresponding transcription paired with a corresponding non-synthesized speech representation of the spoken training utterance; For each non-spoken text utterance, conditioning the unspoken text utterance with a randomly assigned speaker embedding from a set of speaker embeddings, each speaker embedding in the set of speaker embeddings representing a speaker characteristic; generating a synthetic speech representation of the corresponding non-spoken text utterance conditioned on the randomly assigned speaker embedding as output from a pre-trained text-to-speech (TTS) model configured to receive the corresponding non-spoken text utterance as input; training the speech recognition model with the synthetic speech representations generated as output from the pre-trained TTS model and the non-synthetic speech representations in the set of spoken training utterances, the speech recognition model including a frame alignment based transducer model; Including, the pre-trained TTS model is further configured to receive the corresponding transcription as input and output a synthetic speech representation of the corresponding transcription, and training the speech recognition model further includes training the speech recognition model with the synthetic speech representation of the corresponding transcription produced as output from the pre-trained TTS model.

2. 2. The method of claim 1 , wherein each speaker embedding in the set of speaker embeddings is extracted from a corresponding one of the non-synthesized speech representations in the set of spoken training utterances and represents the speaker characteristics of the speaker who spoke the corresponding spoken training utterance.

3. The method of claim 1 , wherein each non-spoken text utterance is represented by a corresponding sequence of phonemes.

4. The method of claim 1 , wherein each synthetic speech representation generated as output from the pre-trained TTS model is represented by a sequence of Mel-frequency spectrogram frames.

5. The method of claim 1 , wherein each non-synthesized speech representation in the set of spoken training utterances is represented by a sequence of Mel-frequency spectrogram frames.

6. 10. The method of claim 1, wherein the speech recognition model including the frame alignment based transducer model includes a recurrent neural network transducer (RNN-T) model.

7. The pre-trained TTS model 1. An encoder neural network, comprising: receiving as input each non-spoken text utterance as a sequence of phonemes; As output, it produces a sequence of context vectors an encoder neural network configured as follows: a decoder neural network, receiving as input each context vector in the sequence of context vectors produced as output by the encoder neural network; For each context vector, generate the corresponding frame in the sequence of Mel-frequency spectrogram frames as output. A decoder neural network constructed as follows:

2. The method of claim 1, comprising:

8. The operations further include, during training the speech recognition model: In each of a plurality of output steps for each synthetic speech representation generated as output from the pre-trained TTS model, determining a first probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation for output by the speech recognition model; generating a synthetic speech loss term based on the first probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation and the corresponding unspoken text utterance from which the corresponding synthetic speech representation is generated; In each of a plurality of output steps for each non-synthesized speech representation, determining a second probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation for output by the speech recognition model; generating a non-synthetic speech loss term based on the second probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation and a transcription in the set of spoken training utterances paired with the corresponding non-synthetic speech representation; 2. The method of claim 1, comprising:

9. The operation further comprises: The method of claim 1 , comprising conditioning each non-spoken text utterance with an utterance embedding selected from a set of utterance embeddings, each utterance embedding in the set representing an intended prosody.

10. 10. The method of claim 9, wherein each utterance embedding in the set of utterance embeddings is extracted from a corresponding one of the non-synthesized speech representations in the set of spoken training utterances by a variational autoencoder (VAE).

11. 1. A system for training a speech recognition model, comprising: Data Processing Hardware and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations comprising: obtaining a plurality of non-spoken text utterances associated with a target domain that the speech recognition model is trained to learn; obtaining a set of spoken training utterances, each spoken training utterance including a corresponding transcription paired with a corresponding non-synthesized speech representation of the spoken training utterance; For each non-spoken text utterance, conditioning the unspoken text utterance with a randomly assigned speaker embedding from a set of speaker embeddings, each speaker embedding in the set of speaker embeddings representing a speaker characteristic; generating a synthetic speech representation of the corresponding non-spoken text utterance conditioned on the randomly assigned speaker embedding as output from a pre-trained text-to-speech (TTS) model configured to receive the corresponding non-spoken text utterance as input; training the speech recognition model with the synthetic speech representations generated as output from the pre-trained TTS model and the non-synthetic speech representations in the set of spoken training utterances, the speech recognition model including a frame alignment based transducer model; Including, the pre-trained TTS model is further configured to receive the corresponding transcription as input and output a synthetic speech representation of the corresponding transcription, and training the speech recognition model further includes training the speech recognition model with the synthetic speech representation of the corresponding transcription produced as output from the pre-trained TTS model.

12. 12. The system of claim 11, wherein each speaker embedding in the set of speaker embeddings is extracted from a corresponding one of the non-synthesized speech representations in the set of spoken training utterances and represents the speaker characteristics of the speaker who spoke the corresponding spoken training utterance.

13. The system of claim 11 , wherein each non-spoken text utterance is represented by a corresponding sequence of phonemes.

14. 12. The system of claim 11, wherein each synthetic speech representation generated as output from the pre-trained TTS model is represented by a sequence of Mel-frequency spectrogram frames.

15. 12. The system of claim 11, wherein each non-synthesized speech representation in the set of spoken training utterances is represented by a sequence of Mel-frequency spectrogram frames.

16. 12. The system of claim 11, wherein the speech recognition model including the frame alignment based transducer model includes a recurrent neural network transducer (RNN-T) model.

17. The pre-trained TTS model 1. An encoder neural network, comprising: receiving as input each non-spoken text utterance as a sequence of phonemes; As output, it produces a sequence of context vectors an encoder neural network configured as follows: a decoder neural network, receiving as input each context vector in the sequence of context vectors produced as output by the encoder neural network; For each context vector, generate the corresponding frame in the sequence of Mel-frequency spectrogram frames as output. A decoder neural network constructed as follows: The system of claim 11 , comprising:

18. The operations further include, during training the speech recognition model: In each of a plurality of output steps for each synthetic speech representation generated as output from the pre-trained TTS model, determining a first probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation for output by the speech recognition model; generating a synthetic speech loss term based on the first probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation and the corresponding unspoken text utterance from which the corresponding synthetic speech representation is generated; In each of a plurality of output steps for each non-synthesized speech representation, determining a second probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation for output by the speech recognition model; generating a non-synthetic speech loss term based on the second probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation and a transcription in the set of spoken training utterances paired with the corresponding non-synthetic speech representation; The system of claim 11 , comprising:

19. The operation further comprises:

12. The system of claim 11, comprising conditioning each non-spoken text utterance with an utterance embedding selected from a set of utterance embeddings, each utterance embedding in the set representing an intended prosody.

20. 20. The system of claim 19, wherein each utterance embedding in the set of utterance embeddings is extracted from a corresponding one of the non-synthesized speech representations in the set of spoken training utterances by a variational autoencoder (VAE).

Images