Self-training wavenet for text-to-speech

Abstract

The method (300) of self-training a WaveNet includes receiving a plurality of recorded speech samples (242) and training a first autoregressive neural network (210) using the plurality of recorded speech samples. The trained first autoregressive neural network is configured to output (212) synthesized speech as an audible representation of a text input. The method further includes generating a plurality of synthesized speech samples (252) using the trained first autoregressive neural network. The method additionally includes training a second autoregressive neural network (220) using the plurality of synthesized speech samples from the trained first autoregressive neural network, and extracting the trained second autoregressive neural network as a feedforward neural network (230).

Description

TECHNICAL FIELD

[0001] The present disclosure relates to self-training WaveNets for text-to-speech. BACKGROUND

[0002] Speech synthesis systems use text-to-speech (TTS) models to generate speech from text input. The generated / synthesized speech should accurately convey the message (intelligibility) while sounding like human speech with the intended prosody (expressiveness) (naturalness). While traditional concatenative and parametric synthesis models can provide intelligible speech, recent advances in neural modeling of speech have significantly improved the naturalness and fidelity of synthesized speech. However, even with these advances, the accuracy of these neural network models often depends on the corpus of training examples available to teach the neural network model how to synthesize speech. Thus, when there is a limited number of training examples, the neural network model lacks the precision of speech synthesis that users of the speech synthesis system expect or even require. This can be especially true as speech synthesis systems (e.g., personal assistants) become more integrated in everyday human-machine interactions. SUMMARY

[0003] One aspect of the present disclosure provides a method of self-training a WaveNet. The method includes receiving, at data processing hardware, a plurality of recorded speech samples, and training, by the data processing hardware, a first autoregressive neural network using the plurality of recorded speech samples. The trained first autoregressive neural network is configured to output synthesized speech as an audible representation of text input. The method further includes generating, by the data processing hardware, a plurality of synthesized speech samples using the trained first autoregressive neural network. The method additionally includes training, by the data processing hardware, a second autoregressive neural network using the plurality of synthesized speech samples from the trained first autoregressive neural network, and extracting, by the data processing hardware, the trained second autoregressive neural network as a feedforward neural network.

[0004] Another aspect of the present disclosure provides another method of self-training a WaveNet. The method includes receiving, at data processing hardware, a plurality of recorded speech samples, and training, by the data processing hardware, an autoregressive neural network using the plurality of recorded speech samples. The trained autoregressive neural network is configured to output synthesized speech as an audible representation of text input. The method further includes generating, by the data processing hardware, a plurality of synthesized speech samples using the trained autoregressive neural network. The method additionally includes extracting, by the data processing hardware, the trained autoregressive neural network as a feedforward neural network based on the plurality of synthesized speech samples.

[0005] Yet another aspect of the present disclosure provides a self-training WaveNet system. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving a plurality of recorded speech samples and training a first autoregressive neural network using the plurality of recorded speech samples. The trained first autoregressive neural network is configured to output synthesized speech as an audible representation of text input. The operations further include generating a plurality of synthesized speech samples using the trained first autoregressive neural network. The operations additionally include training a second autoregressive neural network using the plurality of synthesized speech samples from the trained first autoregressive neural network and extracting the trained second autoregressive neural network as a feedforward neural network.

[0006] Embodiments of any of the present disclosure can include one or more of the following optional features. In some embodiments, the second autoregressive neural network comprises a different autoregressive neural network than the first autoregressive neural network or the same autoregressive neural network as the first autoregressive neural network. When the first and second autoregressive neural networks are the same, training the second subregressive neural network using the plurality of synthesized speech samples comprises retraining the first autoregressive neural network using the plurality of synthesized speech samples. In some examples, the plurality of recorded speech samples comprises a respective number of recorded speech samples and the plurality of synthesized speech samples comprises a respective number of synthesized speech samples, the respective number of recorded speech samples being less than the respective number of synthesized speech samples. In these examples, the respective number of synthesized speech samples can be at least one time greater than the respective number of recorded speech samples.

[0007] In some configurations, extracting the trained second autoregressive neural network as a feedforward neural network comprises training the feedforward neural network based on a probability distribution from the trained second autoregressive neural network. In these configurations, training the feedforward neural network comprises optimizing a loss function between the feedforward neural network and the second autoregressive neural network based on a Kullback-Leibler (KL) divergence. The loss function can comprise a weighted sum of the KL divergence, a mean squared error, a phoneme classification error, and a contrastive loss between the feedforward neural network and the second autoregressive neural network.

[0008] In some embodiments, the feedforward neural network is configured to output synthesized speech without knowledge of one or more previous synthesized speech outputs. Each of the second autoregressive neural network and the feedforward network can comprise a plurality of dilated residual blocks, where each dilated residual block comprises a dilated convolutional layer. The feedforward network can comprise a plurality of inverse autoregressive flows (IAFs). When the feedforward network comprises a plurality of IAFs, each IAF flow of the plurality of IAF flows comprises one or more dilated residual blocks, where each dilated residual block comprises a dilated convolutional layer.

[0009] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the detailed description below. Other aspects, features, and advantages will become apparent from the detailed description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a schematic diagram of an example speech environment.

[0011] Figure 2 is a schematic diagram of an example self-training model of a speech synthesis system for Figure 1 a speech environment.

[0012] Figure 3 and 4 is a flowchart of an example arrangement of operations of a method for self-training WaveNet for text-to-speech.

[0013] Figure 5 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0014] The same reference numbers in the various drawings indicate the same elements. DETAILED DESCRIPTION

[0015] Deep neural networks have increasingly been used to improve the ability of computing devices to understand natural speech. However, in addition to understanding natural speech, people often interact with computing devices expecting the computing devices (e.g., speech-enabled devices) to generate synthetic speech in response. For example, a user of a speech-enabled device submits a query to the speech-enabled device, or requests that the speech-enabled device generate speech based on some form of text. To output a response to the user, the speech-enabled device employs a speech synthesis system or a text-to-speech (TTS) system. Over time, speech synthesis has shifted from concatenative or statistical parametric synthesis to synthesis performed by deep neural network models. In this shift, speech synthesis or TTS systems have evolved to produce high-fidelity audio comparable to that of a human.

[0016] WaveNet is a model that has proven to be a popular choice for generating seemingly real speech. WaveNet originally referred to a deep neural network that generated raw audio waveforms. At the time of initial development, WaveNet was a fully probabilistic and autoregressive model, where the predictive distribution of an audio sample depends on the distribution of all previous audio samples (i.e., using ancestor samples). As an autoregressive network, WaveNet used dilated convolutions to model the probability distribution of a speech sample. For example, the convolutional layers of WaveNet had various dilation factors to allow the receptive field to grow exponentially with depth in order to cover thousands of time steps. Through the convolutional layers, WaveNet was able to process its input in parallel; in contrast to models based on recurrent neural networks, WaveNet’s architecture was able to be trained much faster. However, despite WaveNet’s ability to model thousands of raw audio samples, as an autoregressive network, WaveNet proved to be too slow during the inference process for real-time speech synthesis. In contrast to training, which can be performed in parallel, during inference or generating waveforms, the fully autoregressive WaveNet generates the synthesized output in a sequential manner. For example, WaveNet was only able to generate speech at 172 time steps per second. While this speed, which is slower than real-time inference, is acceptable for offline synthesis speech generation, for real-time applications, the fully autoregressive WaveNet proved to be too slow.

[0017] To remedy the slow inference speed, a parallel variant of WaveNet, referred to as Parallel WaveNet, emerged to produce audio at a faster rate than real-time speech while maintaining the high-fidelity and realistic speech manner of the original autoregressive WaveNet. To be able to generate synthesized audio during inference at real-time speech rates, Parallel WaveNet extracted the autoregressive network of the original autoregressive WaveNet as a parallel feed-forward neural network described in van den Oord, Parallel WaveNet: Fast High-Fidelity Speech Synthesis, which is available at https: / / arxiv.org / pdf / 1711.10433.pdfobtained and incorporated herein by reference. Here, the autoregressive network is referred to as a“teacher” network (or autoregressive teacher) because the feedforward network is taught by the autoregressive network; thus, the feedforward network is also referred to as a“student” network or feedforward student network. In other words, Parallel WaveNet uses a trained autoregressive neural network as a teacher network to train a student network. In this regard, Parallel WaveNet leverages the parallel training speed of autoregressive WaveNet, but generates a feedforward network that does not rely on ancestral sampling (e.g., similar to a trained autoregressive teacher network). As a feedforward network, the student network is thereafter able to implicitly infer outputs of previous time steps without the actual knowledge of these outputs. With this teacher-student architecture, Parallel WaveNet is able to generate samples at about 500,000 time steps per second; addressing the speed deficiency of fully autoregressive WaveNet.

[0018] Unfortunately, Parallel WaveNet is not without its drawbacks. One such drawback is that Parallel WaveNet requires a large amount of recorded speech data to produce a high quality student network. For example, a single speaker version of WaveNet has shown to require approximately 24 hours of recorded speech to form a high fidelity student network. Further, other neural vocoders such as single speaker Tacotron have higher fidelity when trained on more utterances (e.g., 25,000 utterances) than when trained on fewer utterances (e.g., 15,000 or even 8,000 utterances). However, there are inevitably situations where a large amount of recorded speech data is not available. When a teacher network trains a student network with a small amount of recorded speech data, the synthesized output can contain artifacts such as static noise, which become more prominent when fewer training samples of the recorded speech region are available.

[0019] Because speech synthesis models do not always have large amounts of recorded speech data, a WaveNet version that generates high-fidelity audio from low-data mechanisms would improve the robustness and / or versatility of WaveNet. Here, to further develop WaveNet to be effective under low-data mechanisms, a self-training model utilizes the high-fidelity audio produced by the autoregressive WaveNet and the faster-than-real-time synthesis capability of the Parallel WaveNet. In other words, under low-data mechanisms, the Parallel WaveNet can be trained on speech synthesized by the autoregressive WaveNet teacher to form a self-training model (i.e., a self-training WaveNet). By generating high-fidelity synthesized speech data from the autoregressive teacher network to train the student network, the self-training WaveNet is able to train the student network when low amounts of recorded speech are available without compromising fidelity. Here, self-training refers to a technique that uses an already trained system to generate outputs on unseen input examples and uses these generated outputs as targets for subsequent training / retraining. Using the self-training method, the high-fidelity synthesized speech samples generated by the trained autoregressive teacher network train (or extract) the feedforward student network. This approach takes advantage of the fact that the autoregressive WaveNet (e.g., the autoregressive teacher network) produces high-quality synthesized examples; thereby allowing the self-training technique to not further degrade the student network in response to the synthesized examples.

[0020] Reference Figure 1 In some embodiments, the speech environment 100 includes a user 10 communicating a spoken utterance 12 to a voice-enabled device 110 (also referred to as a device 110 or a user device 110). The user 10 (i.e., the speaker of the utterance 12) can speak the utterance 12 as a query or a command requesting a response from the device 110. The device 110 is configured to capture sound from one or more users 10 within the speech environment 100. Here, the audio sound can refer to the utterance 12 spoken by the user 10 that is used as an audible query, a command for the device 110, or an audible communication captured by the device 110. The voice-enabled system of or associated with the device 110 can field the query of the command by answering the query and / or causing the command to be executed.

[0021] Here, the device 110 captures an audio signal 202 (also referred to as audio data) of an utterance 12 spoken by the user 10. The device 110 can correspond to any computing device associated with the user 10 and capable of receiving the audio signal 202. Some examples of the user device 110 include, but are not limited to, a mobile device (e.g., a mobile phone, a tablet computer, a laptop computer, etc.), a computer, a wearable device (e.g., a smart watch), a smart appliance and an Internet of Things (IoT) device, a smart speaker, etc. The device 110 includes data processing hardware 112 and memory hardware 114 that is in communication with the data processing hardware 112 and that stores instructions that, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations. In some examples, the device 110 includes one or more application programs (i.e., software applications), where each application program can utilize one or more speech processing systems 140, 150, 200 associated with the device 110 to perform various functions within the application program. For example, the device 110 includes an assistive application program that is configured to communicate synthesized playback audio 154 to the user 10 to assist the user 10 in various tasks.

[0022] The device 110 further includes an audio subsystem having an audio capture device (e.g., a microphone) 116 for capturing spoken utterances 12 within the speech environment 100 and converting the spoken utterances into electrical signals, and a speech output device (e.g., a speaker) 118 for communicating audible audio signals (e.g., synthesized playback signals 154 from the device 110). While the device 110 implements a single audio capture device 116 in the illustrated example, the device 110 can implement an array of audio capture devices 116 without departing from the scope of the present disclosure, whereby one or more audio capture devices 116 in the array can not physically reside on the device 110 but rather be in communication with the audio subsystem (e.g., a peripheral device of the device 110). For example, the device 110 can correspond to a vehicle infotainment system that utilizes an array of microphones located throughout a vehicle.

[0023] Further, the device 110 is configured to communicate with a remote system 130 via the network 120. The remote system 130 can include remote resources 132, such as remote data processing hardware 134 (e.g., a remote server or CPU) and / or remote memory hardware 136 (e.g., a remote database or other storage hardware). The device 110 can utilize the remote resources 132 to perform various functions related to speech processing and / or synthetic playback communication. For example, the device 110 is configured to perform speech recognition using a speech recognition system 140 and / or convert text to speech using a TTS system 150 (e.g., using the self-training model 200). These systems 140, 150, 200 can reside on the device 110 (referred to as on-device systems) or remotely (e.g., on the remote system 130), but in communication with the device 110. In some examples, some of these systems 140, 150, 200 reside locally or on-device, while other systems reside remotely. In other words, any of these systems 140, 150, 200 can be any combination of local or remote. For example, when the systems 140, 150, 200 are fairly large in size or processing requirements, the systems 140, 150, 200 can reside in the remote system 130. However, when the device 110 can support the size or processing requirements of one or more systems 140, 150, 200, the one or more systems 140, 150, 200 can reside on the device 110 using the data processing hardware 112 and / or the memory hardware 114. Optionally, one or more of the systems 140, 150, 200 can reside both locally / on-device and remotely. For example, when a connection to the network 120 between the device 110 and the remote system 130 is available, one or more of the systems 140, 150, 200 can default to performing on the remote system 130, but when the connection is lost or the network 120 is unavailable, the systems 140, 150, 200 instead perform locally on the device 110.

[0024] The speech recognition system 140 receives the audio signal 202 as input and transcribes the audio signal into a transcription 142 as output. Generally, by converting the audio signal 202 into the transcription 142, the speech recognition system 140 allows the device 110 to identify when spoken utterances 12 from the user 10 correspond to a query, a command, or some other form of audio communication. The transcription 142 refers to a sequence of text that the device 110 can subsequently use to generate a response to the query or command. For example, if the user 10 asks the device 110 the question "what will the weather be like today," the device 110 passes the audio signal corresponding to the question "what will the weather be like today" to the speech recognition system 140. The speech recognition system 140 converts the audio signal into a transcription that includes the text "what will the weather be like today?" The device 110 can then use the text or portions of the text to determine a response to the query. For example, to determine the weather for the day (i.e., today), the device 110 passes the text (e.g., "what will the weather be like today?") or identified portions of the text (e.g., "weather" and "today") to a search engine. The search engine can then return one or more search results that the device 110 interprets to generate a response to the user 10.

[0025] In some embodiments, the device 110 or a system associated with the device 110 identifies text 152 that the device 110 will communicate to the user 10 as a response to a query of a spoken utterance 12. The device 110 can then use the TTS system 150 to convert the text 152 into corresponding synthesized playback audio 154 for the device 110 to communicate to the user 10 (e.g., audibly communicate to the user 10) as a response to the query of the spoken utterance 12. In other words, the TTS system 150 receives the text 152 as input and converts the text 152 into synthesized playback audio 154 as output, where the synthesized playback audio 154 is an audio signal that defines an audible rendition of the text 152. Here, the TTS system 150 (or other speech synthesis system) includes a self-training model 200 (e.g., Tacotron 2's self-training model) that utilizes a deep neural network (e.g., a self-training WaveNet) to generate the synthesized playback audio 154. Once generated, the TTS system 150 communicates the synthesized playback audio 154 to the device 110 to allow the device 110 to output the synthesized playback audio 154. For example, the device 110 outputs the synthesized playback audio 154 "today is sunny" at the speaker 118 of the device 110. Figure 2

[0026] Reference is made to Figure 2 ​The TTS system 150 includes a self-training model 200 as a neural vocoder to generate the synthesized playback audio 154. As a neural vocoder, the self-training model 200 can condition on features that encode linguistic and / or prosodic information. Here, linguistic conditioning can include phoneme, syllable, word, phrase, and / or utterance level features. In some examples, these features are derived through text normalization (e.g., static rule-based text normalization) combined with rule-based feature computation. Prosodic conditioning can be provided by an autoencoder such as a hierarchical variational autoencoder. The autoencoder can also use linguistic conditioning to generate per-frame fundamental frequency as output.

[0027] In some embodiments, such as Figure 2 The self-training model 200 includes a first teacher network 210, a second or synthesized teacher network 220, and a synthesized student network 230. Each network 210, 220, 230 can have a dilated residual block as its constituent unit(s). In some examples, the dilated residual block includes one or more convolutional layers. For example, the dilated residual blocks of the networks 210, 220, 230 correspond to ten convolutional layers (e.g., each layer increases by a factor of 2). In some configurations, the teacher networks 210, 220 include three dilated residual blocks, while the student network 230 includes four inverse autoregressive flows (IAFs), where each IAF can have some number of residual blocks. For example, the first, second, and third IAFs each include one dilated block, while the fourth IAF includes three dilated residual blocks.

[0028] As illustrated by Figure 2In contrast, the self-training model 200 is trained without utilizing synthetic speech. Generally, conventional methods of training a Parallel WaveNet are a two-step process. In the first step, an autoregressive teacher network is trained to model a probability distribution for a sample based on previous samples of recorded speech. Here, training can proceed in parallel with the teacher forcing technique since all recorded speech samples are available. During the second step for training a conventional Parallel WaveNet, the training process extracts the density distribution of the autoregressive teacher network as a feedforward student network. Extraction generally refers to a process of training a neural network using a pre-trained network. Using extraction, neurons of the pre-trained network that are less critical to the desired output (e.g., similar to self-weights) can be reduced to form a more compact neural network (i.e., the extracted neural network). When compared to the pre-trained network, the extracted neural network can be more accurate and / or more compact. In other words, when forming the pre-trained network, the pre-trained network can have formed neurons that ultimately result in less influence on the desired output when training of the pre-trained network is complete; thus, the pre-trained network includes neurons that can be removed or modified to reduce any detrimental effects of these neurons or remove unnecessary neurons.

[0029] In some examples, the self-training model 200 includes a first teacher 210 that functions as an autoregressive neural network. Much like a conventional Parallel WaveNet, the first teacher 210 is trained using recorded speech samples 242 from a total amount 240 of recorded speech samples 242, 242a-n (e.g., shown as a database or other type of audio data repository). However, here, the self-training model 200 can be utilized when the number of recorded speech samples 242 is relatively low (i.e., a low-data regime). For example, a low-data regime refers to a total amount 240 of recorded speech samples 242 that is less than twenty-five thousand samples (i.e., a healthy amount of recorded samples 242 for a Parallel WaveNet), but greater than five thousand samples (e.g., an unhealthy amount of recorded samples 242 for a Parallel WaveNet). In some examples, a low-data regime is between five thousand and fifteen thousand recorded speech samples 242. Even though a low-data regime has fewer recorded speech samples 242, the first teacher 210 is trained using the recorded speech samples 242 to form a trained first teacher 210.

[0030] With the trained first teacher 210, the self-training model 200 causes the trained second teacher 210 to generate a total amount 250 of synthetic speech samples 252, 252a-n as output 212. In some embodiments, the generation of synthetic speech samples 252 is a one-time processing task that can be performed offline, which has little to no limit on the number of synthetic speech samples 252 generated by the trained teacher 210. In some configurations, the self-training model 200 uses a trained autoregressive teacher from Parallel WaveNet instead of a feedforward student (e.g., as shown in the dashed box in FIG. 1) to generate synthetic speech samples 252 because autoregressive teacher networks 210 generate synthetic speech samples 252 with higher fidelity than feedforward student networks. Thus, while it is also possible to generate synthetic speech samples with a feedforward student network of Parallel WaveNet, this can compromise the fidelity of the self-training model 200. Figure 2

[0031] In some examples, with the TTS system 150, the self-training model 200 is configured to generate synthetic speech samples 252 from readily available text samples (e.g., text 152 from the TTS system 150). In some configurations, the text samples can be unlabeled text samples. The teacher 210 can generate synthetic speech samples 252 from a text sample corpus that has sufficient phonetic coverage of the language of the synthetic speech samples 252. In some examples, the synthetic speech samples 252 undergo a pruning process to prevent the synthetic student 230 from learning with noisy or corrupted synthetic speech samples 252 (i.e., synthetic speech samples 252 that would be detrimental to learning). For example, to avoid issues with synthetic speech samples 252 that include biases with respect to phoneme distribution, a script selection method can be applied to the dataset of generated synthetic speech samples 252. Additionally or alternatively, during or after the process of generating synthetic speech samples 252, the training process can generate a phoneme alignment of the synthetic speech samples 252. With the phoneme alignment of the synthetic speech samples 252, the pruning process can reject synthetic speech samples 252 that produce a phoneme alignment score that is below a particular threshold (i.e., a minimum allowable alignment score). In other words, the phoneme alignment score can indicate which synthetic speech samples 252 can need to be pruned from the training set of synthetic speech samples 252 that will be used to train the synthetic teacher 220 (or, in some cases, retrain the teacher 210).

[0032] ​In some configurations, once the trained teacher 210 generates the synthetic speech samples 252, the model training process uses the synthetic speech samples 252 to train a synthetic teacher 220. For example, the model training process uses a training dataset that includes the synthetic speech samples 252 that survived the pruning process. Here, the training process for the synthetic teacher 220 is the same as the training process for the first teacher 210, except that the training process uses synthetic speech samples 252 instead of recorded speech samples 242. With this training process, the synthetic teacher 220 can be trained with a large sample set (i.e., synthetic speech samples 252) when compared to a small number of recorded speech samples 252. For example, the teacher 210 generates up to twenty-five to fifty thousand synthetic speech samples, while the low-data mechanism includes a fraction of this amount (e.g., between five to fifteen thousand recorded speech samples 242); thus, even if the first teacher 210 and the synthetic teacher 220 are both autoregressive neural networks, the synthetic teacher 220 is trained on a quantity of speech data that is two or more times larger than the quantity of speech data on which the teacher 210 is trained.

[0033] Optionally, in some examples, the training process trains or extracts a student 230 using the synthetic speech samples 252 without training a synthetic teacher 220. While this approach can work, it can not be ideal. This is particularly true because, in order to generate the synthetic speech samples 252, the trained teacher 210 is trained on only a relatively small number of recorded speech samples 242. This means that training a student 230 immediately likely does not have the level of fidelity that a synthetic student 230 would have if trained by a teacher (e.g., synthetic teacher 220) that has been trained using a larger data mechanism (e.g., synthetic speech samples 252). Additionally or alternatively, the first teacher 210 and the second teacher or synthetic teacher 220 can be the same neural network. In other words, the synthetic teacher 220 is simply the teacher 210 that has been retrained using synthetic speech samples 252 such that the neural network that forms the teacher 210 has subsequently been retrained on a larger corpus of speech samples.

[0034] When the synthetic teacher 220 has been trained, the trained synthetic teacher 230 can then be extracted into a synthetic student 230. This means that the trained synthetic teacher 230 trains the synthetic student 230 according to the probability distribution corresponding to the synthetic speech. The training process by the trained synthetic teacher 220 extracts the synthetic teacher 220 into a feedforward synthetic student 230. As a feedforward neural network, the synthetic student 230, which is very similar to the feedforward student of the Parallel WaveNet, is able to generate an output of synthetic speech (e.g., the synthetic playback audio 154) without needing to know one or more previous synthetic speech outputs during inference (e.g., similar to an autoregressive network). Although the training process is generally described with respect to a single speaker, the training process of the model 200 can be scaled for multiple speakers. For example, when the model 200 is used for multiple speakers, the synthetic teacher 220 extracts a synthetic student 230 for each speaker of the multiple speakers, such that the model 200 for the multiple speakers includes multiple synthetic students 230 (e.g., proportional to the number of speakers).

[0035] In some embodiments, the training process trains the components of the model 200 by optimizing a loss function. For example, to train either teacher model 210, 220, the training process can use a loss function that represents the negative log-likelihood of the predicted mixture distribution. In some examples, the extraction process trains the student 230 using a loss function based on the Kullback-Leibler (KL) divergence. For example, the loss function is based on the KL divergence between the student 230 (e.g., a feedforward neural network) and the synthetic teacher 220 (e.g., an autoregressive neural network trained by the synthetic speech samples 252). In some configurations, the extraction process trains the student 230 with a loss function that is a weighted sum of several different loss functions (or errors), such as the KL divergence (e.g., between the student 230 and the synthetic teacher 220 distribution), the mean squared error (e.g., between the predicted and target signal powers in a moving window), the phoneme classification error, and / or the contrastive loss that maximizes the difference between the KL divergence of the probability distributions between the student 230 and the teacher 220 when obtained with the correct conditioning and the KL divergence of the probability distributions between the student 230 and the teacher 220 when obtained with the incorrect conditioning.

[0036] Figure 3is a flowchart of an example arrangement of operations for a method of self-training a WaveNet. At operation 302, method 300 receives a plurality of recorded speech samples 242. At operation 304, method 300 trains a first autoregressive neural network 210 using the plurality of recorded speech samples 242. The trained first autoregressive neural network 210 is configured to output synthesized speech as an audible representation of a text input. At operation 306, method 300 generates a plurality of synthesized speech samples 252 using the trained first autoregressive neural network 210. At operation 308, method 300 trains a second autoregressive neural network 220 using the plurality of synthesized speech samples 252 from the trained first autoregressive neural network 210. At operation 310, method 300 extracts the trained second autoregressive neural network 220 as a feedforward neural network 230.

[0037] Figure 4 is another flowchart of an example arrangement of operations for another method of self-training a WaveNet. At operation 402, method 400 receives a plurality of recorded speech samples 242. At operation 404, method 300 trains an autoregressive neural network 210 using the plurality of recorded speech samples 242. The trained autoregressive neural network 210 is configured to output synthesized speech as an audible representation of a text input. At operation 406, method 400 generates a plurality of synthesized speech samples 252 using the trained autoregressive neural network 210. At operation 408, method 400 extracts the trained autoregressive neural network 210 as a feedforward neural network 230.

[0038] Figure 5 is a schematic diagram of an example computing device 500 that can be used to implement the systems (e.g., speech recognition system 140, TTS system 150, and / or self-training model 200) and methods (e.g., methods 300, 400) described in this document. The computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the applications described and / or claimed in this document.

[0039] The computing device 500 includes a processor 510 (e.g., data processing hardware), a memory 520 (e.g., memory hardware), a storage device 530, a high-speed interface / controller 540 connecting the memory 520 and the high-speed expansion ports 550, and a low speed interface / controller 560 connecting the low speed bus 570 and the storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 are interconnected using various busses, and can be mounted on a common motherboard or otherwise in other manners as appropriate. The processor 510 can process instructions for execution within the computing device 500, including instructions stored in the memory 520 or on the storage device 530 to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 coupled to the high-speed interface 540. In other implementations, multiple processors and / or multiple buses can be used as appropriate, along with multiple memories and types of memory. Also, multiple computing devices 500 can be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

[0040] The memory 520 stores information non-transitorily within the computing device 500. The memory 520 can be a computer-readable medium, a volatile memory unit(s), or non-volatile memory unit(s). The non-transitory memory 520 can be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by or

[0041] The storage device 530 can provide mass storage for the computing device 500. In some embodiments, the storage device 530 is a computer- readable medium. In various different embodiments, the storage device 530 can be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional embodiments, the computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, for example, the memory 520, the storage device 530, or memory on processor 510.

[0042] The high-speed controller 540 manages bandwidth-intensive operations for the computing device 500, while the low-speed controller 560 manages lower bandwidth-intensive operations. Such allocation of functions is exemplary only. In some embodiments, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports 550, which can accept various expansion cards (not shown). In some embodiments, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

[0043] As shown, the computing device 500 can be implemented using a variety of different forms. For example, the computing device can be implemented as a standard server 500a, or multiple times in a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0044] Various implementations of the systems and techniques described herein can be realized in digital electronic and / or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0045] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer readable medium, devices and / or means used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0046] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical, or optical disks, or a computer program product suitable for storing a computer program and data. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0047] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touchscreen, for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball), through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on the user's client device in response to a request received from a web browser.

[0048] Several embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A method comprising: Receive multiple recorded raw audio waveform samples of the speech at the data processing hardware; The data processing hardware trains a first autoregressive neural network using multiple recorded raw audio waveform samples of the speech. The trained first autoregressive neural network is configured to output the raw audio waveform of the synthesized speech as an audible representation of the text input. The data processing hardware generates multiple raw audio waveform samples of synthesized speech as the output from the first autoregressive neural network after training; The data processing hardware trains a second autoregressive neural network using multiple raw audio waveform samples of the synthesized speech generated as the output of the first autoregressive neural network after training. as well as The data processing hardware extracts the trained second autoregressive neural network into a feedforward neural network.

2. The method according to claim 1, wherein, The second autoregressive neural network includes one of the following: An autoregressive neural network that is different from the first autoregressive neural network; or The same autoregressive neural network as the first autoregressive neural network, such that training the second autoregressive neural network using multiple original audio waveform samples of the synthesized speech includes retraining the first autoregressive neural network using multiple original audio waveform samples of the synthesized speech.

3. The method according to claim 1, wherein, The multiple recorded original audio waveform samples of the speech include a corresponding number of recorded speech samples, and the multiple original audio waveform samples of the synthesized speech include a corresponding number of synthesized speech samples, wherein the corresponding number of recorded speech samples is less than the corresponding number of synthesized speech samples.

4. The method according to claim 3, wherein, The number of synthesized speech samples is at least twice as large as the number of recorded speech samples.

5. The method according to claim 1, wherein, The feedforward neural network is configured to output synthesized speech without knowing one or more previous synthesized speech outputs.

6. The method according to claim 1, wherein, Each of the second autoregressive neural network and the feedforward neural network includes multiple dilated residual blocks, each dilated residual block including a dilated convolutional layer.

7. The method according to claim 1, wherein, The feedforward neural network includes multiple inverse autoregressive streams.

8. The method according to claim 7, wherein, Each of the plurality of inverse autoregressive streams includes one or more dilated residual blocks, and each dilated residual block includes a dilated convolutional layer.

9. A method comprising: Receive multiple recorded voice samples at the data processing hardware; The data processing hardware trains a first autoregressive neural network using the multiple recorded speech samples. The trained first autoregressive neural network is configured to output synthesized speech as an audible representation of the text input. The data processing hardware generates multiple synthetic speech samples using the trained first autoregressive neural network. The data processing hardware trains a second autoregressive neural network using the plurality of synthesized speech samples from the trained first autoregressive neural network; as well as The data processing hardware extracts the trained second autoregressive neural network into a feedforward neural network. Extracting the trained second autoregressive neural network into the feedforward neural network includes optimizing a loss function based on the Kullback-Leibler divergence between the feedforward neural network and the second autoregressive neural network, and training the feedforward neural network based on a probability distribution from the trained second autoregressive neural network. The loss function includes a weighted sum of the Kullback-Leibler divergence, mean squared error, phoneme classification error, and contrast loss between the feedforward neural network and the second autoregressive neural network.

10. A system comprising: Data processing hardware; as well as A memory hardware that communicates with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations including: Multiple recorded raw audio waveform samples of received speech; A first autoregressive neural network is trained using multiple recorded raw audio waveform samples of the speech, and the trained first autoregressive neural network is configured to output the raw audio waveform of the synthesized speech as an audible representation of the text input. Multiple original audio waveform samples are generated as the output of the first autoregressive neural network after training to produce synthesized speech; A second autoregressive neural network is trained using multiple raw audio waveform samples of the synthesized speech generated as the output of the first autoregressive neural network after training; and The trained second autoregressive neural network is extracted as a feedforward neural network.

11. The system according to claim 10, wherein, The second autoregressive neural network includes one of the following: An autoregressive neural network that is different from the first autoregressive neural network; or The same autoregressive neural network as the first autoregressive neural network, such that training the second autoregressive neural network using multiple original audio waveform samples of the synthesized speech includes retraining the first autoregressive neural network using multiple original audio waveform samples of the synthesized speech.

12. The system according to claim 10, wherein, The multiple recorded original audio waveform samples of the speech include a corresponding number of recorded speech samples, and the multiple original audio waveform samples of the synthesized speech include a corresponding number of synthesized speech samples, wherein the corresponding number of recorded speech samples is less than the corresponding number of synthesized speech samples.

13. The system according to claim 12, wherein, The number of synthesized speech samples is at least twice as large as the number of recorded speech samples.

14. The system according to claim 10, wherein, The feedforward neural network is configured to output synthesized speech without knowing one or more previous synthesized speech outputs.

15. The system according to claim 10, wherein, Each of the second autoregressive neural network and the feedforward neural network includes multiple dilated residual blocks, each dilated residual block including a dilated convolutional layer.

16. The system according to claim 10, wherein, The feedforward neural network includes multiple inverse autoregressive streams.

17. The system according to claim 16, wherein, Each of the plurality of inverse autoregressive streams includes one or more dilated residual blocks, and each dilated residual block includes a dilated convolutional layer.

18. A system comprising: Data processing hardware; as well as A memory hardware that communicates with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations including: Receive multiple recorded voice samples; The first autoregressive neural network is trained using the multiple recorded speech samples, and the trained first autoregressive neural network is configured to output synthesized speech as an audible representation of the text input. Multiple synthetic speech samples are generated using the trained first autoregressive neural network. A second autoregressive neural network is trained using the plurality of synthesized speech samples from the first autoregressive neural network after training; and The trained second autoregressive neural network is extracted as a feedforward neural network. Extracting the trained second autoregressive neural network as a feedforward neural network includes optimizing the loss function based on the Kullback-Leibler divergence between the feedforward neural network and the second autoregressive neural network, and training the feedforward neural network based on the probability distribution from the trained second autoregressive neural network. The loss function includes a weighted sum of the Kullback-Leibler divergence, mean squared error, phoneme classification error, and contrast loss between the feedforward neural network and the second autoregressive neural network.

Images