Predicting Spectral Representations for Training Speech Synthesis Neural Networks

Abstract

Methods, systems and apparatuses, including computer programs encoded on a computer storage medium, for training a neural network to perform speech synthesis, one of which includes the steps of: training a first neural network to obtain a training data set for processing spectral representations of audio samples and generating predictions of the audio samples, where after training, the first neural network obtains the spectral representations of the audio samples from a second neural network and generates, for a plurality of audio samples in the training data set, ground-truth spectral representations of the audio samples, processing the ground-truth spectral representations with a third neural network to generate updated spectral representations of the audio samples, and training the first neural network with the updated spectral representations, where the third neural network is configured to generate updated spectral representations similar to the spectral representations generated by the second neural network.

Description

[Technical Field]

[0001] This specification relates to generating audio data using neural networks. [Background technology]

[0002] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to one or more other layers in the network: one or more other hidden layers, the output layer, or both. Each layer of the network generates an output from the received input according to the current values ​​of each set of parameters. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] "Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions" (arXiv: 1712.05884) [Non-patent document 2] "Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling" (arXiv: 2010.04301) [Non-patent document 3] “PixelCNN++: Improving The PixelCNN With Discretized Logistic Mixture Likelihood And Other Modifications”, Salimans et al., arXiv:1701.05517 Summary of the Invention [Means for solving the problem]

[0004] This specification describes a system, implemented as a computer program on one or more computers at one or more locations, configured to train a neural network to generate audio samples using a predicted spectral representation of the audio samples.

[0005] As used herein, a spectral representation of an audio sample is a spectral representation of the frequencies of the audio sample, where the spectrum of frequencies varies over time. For example, at each of a plurality of time points of the audio sample, the spectral representation may include a respective amplitude value for each frequency in a predetermined set of frequencies. As used herein, a spectrogram is a visual representation of the spectral representation of the audio sample. For example, a spectrogram may be a two-dimensional image in which time varies along one axis and frequency varies along another axis, and for each element (e.g., pixel) in the image, the spectrogram may include a visual representation of the amplitude value of the corresponding frequency at the corresponding time, such as a color representing the amplitude value from a range of colors representing the corresponding range of possible amplitude values.

[0006] After the neural network (sometimes referred to herein as a "vocoder" neural network) has been trained, it can be deployed in a speech synthesis system that includes a second neural network (sometimes referred to herein as a "spectral representation" neural network) configured to process a text input to generate a predicted spectral representation of a verbalization (or phoneticization) of the text input. The vocoder neural network can then obtain the predicted spectral representation generated by the spectral representation neural network and process the predicted spectral representation to generate a speech sample of the verbalization of the text input. Thus, the speech synthesis system is configured to accept a system input that includes a conditioned text input that characterizes a text sequence. The speech synthesis system can process the conditioned text input to generate speech data corresponding to the input text, i.e., audio data that characterizes a speaker speaking the input text.

[0007] In some embodiments, a spectral representation neural network may require a significant amount of time and computational resources to train to generate predictive spectral representations for a particular voice or type of voice. That is, in these embodiments, the spectral representation neural network needs to be specifically trained for a particular voice to generate predictive spectral representations for audio samples in which that voice verbalizes a text input. The training process also often requires a significant amount of training data corresponding to the particular voice, i.e., true-valued audio samples of the particular voice verbalizing various text inputs. However, a large corpus of training data is often not available for any small number of voices. Thus, in some such embodiments, the spectral representation neural network is only trained for a few voices (e.g., 3, 5, 10, 20, 50, or 100 voices), i.e., it can only generate predictive spectral representations for audio samples verbalized by these voices.

[0008] On the other hand, a vocoder neural network can be trained broadly across voices. That is, a training system can use a training set containing training examples (e.g., pairs of true-valued audio samples and corresponding spectral representation inputs) corresponding to multiple voices to train the vocoder neural network for multiple voices simultaneously, rather than having to train the vocoder neural network separately for each different voice. Thus, a training set can include training examples corresponding to relatively rare voices, even if the training system only has access to a few such training examples. Indeed, in some embodiments, for example, if a vocoder neural network has been trained on a sufficiently wide variety of voices, the vocoder neural network can be configured, after training is complete, to receive spectral representations corresponding to new voices that were not in the training data set and generate accurate audio samples for the new voices.

[0009] Given a corpus of audio samples, a training system can generate true-valued spectral representations of the audio samples by processing the audio samples, for example, with a bandpass filter or a Fourier transform. However, providing the true-valued spectral representations as training inputs to a vocoder neural network during training can produce suboptimal results after the vocoder neural network is deployed because the vocoder neural network does not receive the true-valued spectral representations at inference time. Instead, the vocoder neural network is generated by a spectral representation neural network. prediction Receive spectral representations. While spectral representation neural networks can be trained to generate highly accurate predicted spectral representations, these predicted spectral representations are not perfect and may have slight differences from the corresponding true spectral representations. When training a neural network, it is important to provide training inputs that match what the neural network will receive during inference.

[0010] Thus, a training system for a vocoder neural network has two competing requirements. First, the vocoder neural network must receive training data that matches the inputs it will receive after deployment; that is, the vocoder neural network must receive spectral representations that are generated by a spectral representation neural network, or that closely resemble the predicted spectral representations generated by a spectral representation neural network. Second, the vocoder neural network must receive training inputs that correspond to a wide variety of speech sounds, which a spectral representation neural network may not be able to generate due to the limitations discussed above.

[0011] Using the techniques described herein, a training system can generate training inputs for a vocoder neural network by processing the true-valued spectral representations of audio samples using a third neural network (sometimes referred to herein as a “perturbation” neural network) to generate updated spectral representations of the audio samples (sometimes referred to herein as a “perturbed spectral representation” neural network). The perturbed spectral representations are similar to the predicted spectral representations generated by the spectral representation neural network and are therefore suitable training inputs for the vocoder neural network. In other words, the perturbation neural network can be configured to process the true-valued spectral representations to generate perturbed spectral representations having the same properties as those generated by the spectral representation neural network. The training system can then perform supervised training of the vocoder neural network using the perturbed spectral representations along with the corresponding true-valued audio samples. After training is complete, the vocoder neural network can be deployed together with the spectral representation neural network as described above; i.e., the perturbation neural network is typically not deployed in a speech synthesis system during inference.

[0012] In some embodiments, the perturbation neural network is explicitly trained to generate perturbed spectral representations that mimic the predicted spectral representations generated by the spectral representation neural network. In some other embodiments, the perturbation neural network is not explicitly trained to generate such perturbed spectral representations. For example, the perturbation neural network can be an autoencoder neural network configured to process the true-valued spectral representations to generate embeddings of the true-valued spectral representations and then use the embeddings to reconstruct the spectral representations. The predicted spectral representations generated by the spectral representation neural network can more closely resemble the reconstructed spectral representations generated by the perturbation neural network than the corresponding true-valued spectral representations, and therefore the reconstructed spectral representations can be used as perturbed spectral representations during training of the vocoder neural network. [Effects of the Invention]

[0013] The subject matter described herein can be implemented in particular embodiments to achieve one or more of the following advantages: Many existing speech synthesis systems are limited to generating audio samples that characterize one or a few predetermined voices. Using the techniques described herein, a training system can train a vocoder neural network to be virtually universal, i.e., capable of generating audio samples that characterize any voice, even one that was not represented in the training data set for the vocoder neural network. In some embodiments, a trained vocoder neural network can be deployed with any one of a number of different spectral representation neural networks (e.g., spectral representation neural networks trained after further training and deployment of the vocoder neural network) to generate high-quality audio samples when receiving spectral representations from any of them.

[0014] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram of an exemplary speech synthesis inference system. [Figure 2] FIG. 1 is a diagram of an exemplary training system. [Figure 3] FIG. 1 is a diagram of an example autoencoder neural network system. [Figure 4] 1 is a flow diagram of an exemplary process for training a neural network to perform speech synthesis. DETAILED DESCRIPTION OF THE INVENTION

[0016] Like reference numbers and designations in the various drawings indicate like elements.

[0017] This specification describes a system for training a neural network to generate audio samples using a predicted spectral representation of the audio samples.

[0018] 1 is a diagram of an example speech synthesis inference system 100 for generating audio samples 122. The speech synthesis inference system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, in which the systems, components, and techniques described below may be implemented.

[0019] The speech synthesis inference system 100 is configured to process a conditioned text input 102 to generate audio samples 122. The conditioned text input 102 represents an input text, and the audio samples 122 represent a speech corresponding to the input text. That is, the audio samples 122 represent a verbalization of the input text represented by the conditioned text input 102.

[0020] The conditioned text input 102 includes a set of input elements representing the input text. In some embodiments, the conditioned text input 102 includes an embedding of the input text at a particular resolution; for example, the set of input elements can be character-level, phoneme-level, or word-level embeddings of the input text. Alternatively, or in addition, the conditioned text input 102 can include linguistic features characterizing the input text. For example, the set of input elements can include a respective vector of linguistic features for each input time step of a set of input time steps corresponding to the input text. As a particular example, the linguistic features for each input time step can include one or more of i) phonemes, ii) the duration of the text at the input time step, or iii) pitch information, such as the pitch at the input time step represented by the logarithmic fundamental frequency, logF, of the input time step.

[0021] The conditioned text input 102 can be obtained from any suitable source. For example, the speech synthesis inference system 100 can be a component of a virtual assistant that interacts with a user to answer questions and / or perform tasks, and the conditioned text input 102 can be a text response generated by the virtual assistant in response to a query issued by the user. The speech synthesis inference system 100 can then generate an audio sample 122 that is a verbalization of the text response and play the audio sample 122 to the user. As another example, the conditioned text input 102 can be text from a book (or other written medium, such as a newspaper or magazine), and the speech synthesis inference system 100 can be configured to generate an audio sample 122 that is a verbalization of the book to be played to the user (e.g., an audiobook corresponding to the book). As another example, the speech synthesis inference system 100 can be a component of a system that receives a text input 102 from a user and provides the user with an audio sample 122 that is a verbalization of the text input 102, e.g., a system that provides accessibility for users who cannot or have difficulty speaking.

[0022] In some embodiments, the speech synthesis inference system 100 also receives as input a noise input 104. For example, the noise input 104 may be randomly sampled from a predetermined distribution, such as a normal distribution. The noise input 104 may ensure variability in the audio samples 122 generated by the speech synthesis inference system 100.

[0023] In some embodiments, the speech synthesis inference system 100 also receives as input an identification of the class 106 to which the audio sample 122 should belong. The class 106 may be a member of a set of possible classes. For example, the class 106 may correspond to a particular speaker (from a set of possible speakers) that the audio sample 122 should also pronounce. That is, the audio sample 122 may represent an input text spoken by a particular speaker. As a particular example, the class 106 may be represented by a one-hot vector, i.e., a vector whose elements correspond to each class from a set of possible classes, with the element corresponding to the class 106 equal to 1 and the rest of the elements equal to zero.

[0024] The audio samples 122 may include a sample of an audio wave at each output time step of a series of output time steps. For example, the audio samples 122 may include an amplitude value of the audio wave for each output time step. In some embodiments, the amplitude values ​​are raw amplitude values, while in other embodiments, the amplitude values ​​are compressed, companded, or both.

[0025] Generally, the series of input time steps and the series of output time steps are characterized by the same time period (i.e., the duration of the vocalization of the input text), such as, for example, 1, 2, 5, or 10 seconds. However, in some cases, there may be a different number of input time steps and output time steps. In a specific example, if the time period is 2 seconds, the conditioning input 102 may include 400 input time steps (resulting in a frequency of 200 Hz), while the audio samples 122 may include 48,000 time steps (resulting in a frequency of 24 kHz). Thus, the speech synthesis inference system 100 may generate audio samples for multiple output time steps (in this case 120) for each input time step.

[0026] Speech synthesis inference system 100 includes two trained neural networks: a spectral representation neural network 110 and a vocoder neural network 120. Speech synthesis inference system 100 is referred to as an "inference" system because spectral representation neural network 110 and generated neural network 120 are already trained, and thus speech synthesis inference system 100 can be deployed, for example, in an authoring environment. That is, the operation of speech synthesis inference system 100 can be performed on a device or system of devices that performs the inference, i.e., receives new input and processes the new input to generate corresponding output.

[0027] In some embodiments, the speech synthesis reasoning system 100 can be deployed to a cloud system, i.e., a distributed computing system having multiple computing nodes, e.g., hundreds or thousands of computing nodes, in one or more locations. In some other embodiments, the speech synthesis reasoning system 100 can be deployed to a user device, e.g., a mobile phone, a tablet computer, a smart device, or other edge computing device.

[0028] The spectral representation neural network 110 is configured to process the conditioned text input 102 (and optionally the noise input 104 and / or the representations of the classes 106 ) to generate a predicted spectral representation 112 of the audio sample 122 .

[0029] The spectral representation 112 may be any suitable representation of the audio samples 122. For example, the spectral representation 112 may be a non-normalized spectral representation of the audio samples, a Mel-frequency spectral representation of the audio samples, or a Mel-frequency Cepstral Coefficients (MFCC) representation of the audio samples.

[0030] For example, the spectral representation neural network 110 may include one or more recurrent neural network layers that iteratively process the sequence of input elements of the conditioned text input 102 over multiple processing time steps. Alternatively, or in addition, the spectral representation neural network 110 may include one or more one-dimensional convolutional neural network layers configured to process the sequence of input elements of the conditioned text input 102 using convolution kernels. Alternatively, or in addition, the spectral representation neural network 110 may include one or more neural network layers, e.g., one or more Transformer neural network layers, that apply a self-attention mechanism to the sequence of input elements of the conditioned text input 102.

[0031] Specific examples of spectral representation neural networks and vocoder neural networks are discussed in more detail in "Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions" (arXiv: 1712.05884), the contents of which are incorporated herein by reference.

[0032] Another specific example of a spectral representation neural network and a vocoder neural network is discussed in more detail in "Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling" (arXiv: 2010.04301), the contents of which are incorporated herein by reference.

[0033] As discussed above, in some embodiments, the spectral representation neural network 110 requires a large amount of training time, significant computational resources, and / or a large number of training examples to learn and generate predictive spectral representations 112 that characterize a given speaker's vocalization of the conditioned text input 102.

[0034] Vocoder neural network 120 is configured to process predicted spectral representation 112 to generate audio samples 122. In some embodiments, vocoder neural network 120 is "universal," i.e., vocoder neural network 120 is configured, via training, to generate audio samples 122 using predicted spectral representation 112 corresponding to any speaker, even if the speaker is not represented in the training data set for vocoder neural network 120.

[0035] In some embodiments, after vocoder neural network 120 generates audio sample 122, speech synthesis inference system 100 can play audio sample 122 to the user, such as by providing the audio sample to an audio system of the user's user device. For example, speech synthesis inference system 100 can be a component of a virtual assistant that interacts with a user to answer questions and / or perform tasks. As a particular example, a user can submit a query, the virtual assistant can generate a text response to the query, and the speech synthesis inference system can generate audio sample 122 that is a verbalization of the text response and play audio sample 122 to the user.

[0036] Alternatively, or in addition, the speech synthesis inference system 100 may provide the audio samples 122 to an external system for storage or further processing.

[0037] In some embodiments, the spectral representation neural network 110 and the vocoder neural network 120 are trained separately. For example, the spectral representation neural network 110 can be trained using training examples, each of which includes i) training conditioned text input and ii) true-valued audio samples. The predicted spectral representations generated from the training conditioned text input for a particular training example can be processed using a reconstruction algorithm, e.g., the Griffin-Lim reconstruction algorithm, to directly generate predicted audio samples. The training system can then use the difference between i) the predicted audio samples generated using the reconstruction and ii) the true-valued audio samples for the particular training example to determine updates to the parameters of the spectral representation neural network 110. As another example, the spectral representation neural network 110 can be trained using training examples, each of which includes i) training conditioned text input and ii) true-valued spectral representations generated from the true-valued audio samples. The training system can then use the difference between i) the predicted spectral representation neural network 110 generated from the training conditioned text input for a particular training example and ii) the true spectral representation of the particular training example to determine updates to the parameters of the spectral representation neural network 110.

[0038] In embodiments in which spectral representation neural network 110 and vocoder neural network 120 are trained separately, vocoder neural network 120 may be trained using training system 200 described below with reference to FIG.

[0039] In some other embodiments, the spectral representation neural network 110 and the vocoder neural network 120 are trained jointly. That is, the training system can obtain training examples, each of which includes i) a training conditioned text input and ii) a ground truth audio sample. The training system can process the training conditioned text input end-to-end using the spectral representation neural network 110 and the vocoder neural network 120 to generate training audio samples and determine the error between the training audio samples and the corresponding ground truth audio samples. The training system can then back-propagate the error through both the vocoder neural network 120 and the spectral representation neural network 110 to determine parameter updates for both neural networks, e.g., using gradient descent.

[0040] In embodiments in which the spectral representation neural network 110 and the vocoder neural network 120 are trained together, the vocoder neural network 120 may then be fine-tuned (i.e., the parameters of the vocoder neural network 120 may be updated) using a training system 200 described below with reference to FIG. 2.

[0041] 2 is a diagram of an example training system 200. Training system 200 is an example of a system in which the systems, components, and techniques described below may be implemented, implemented as computer programs on one or more computers at one or more locations.

[0042] Training system 200 is configured to train vocoder neural network 220 to process spectral representations of audio samples and generate predictions of corresponding audio samples. For example, vocoder neural network 220 can be configured similarly to vocoder neural network 120 described above with reference to Figure 1. Once trained, vocoder neural network 220 can be deployed in a speech synthesis system along with a spectral representation neural network, such as spectral representation neural network 110 shown in Figure 1.

[0043] The training system 200 includes a vocoder neural network 220 , a perturbation neural network 210 , and a parameter update system 230 .

[0044] The perturbation neural network 210 is configured to process the true-value spectral representation 202 of the audio sample to generate a perturbed spectral representation 212 of the audio sample. The training system 200 can then use the perturbed spectral representation 212 as a training input for a vocoder neural network 220.

[0045] As noted above, providing vocoder neural network 220 with inputs during training of vocoder neural network 220 that match or are similar to inputs that vocoder neural network 220 sees during inference can improve training, as measured, for example, by training accuracy or test accuracy after training is complete. Thus, perturbation neural network 210 can be configured to generate perturbed spectral representations 212 that are similar to the predicted spectral representations generated by the spectral representation neural network with which vocoder neural network 220 is deployed.

[0046] In some embodiments, perturbation neural network 210 may be explicitly trained to generate perturbed spectral representations 212 that mimic predicted spectral representations generated by the spectral representation neural network. For example, a training system (e.g., training system 200 or another training system) for perturbation neural network 210 may obtain training examples that each include i) true-valued spectral representations 202 of audio samples representing verbalizations of text input, and ii) predicted spectral representations generated by the spectral representation neural network in response to processing the text input. The training system may then process true-valued spectral representations 202 with perturbation neural network 210 to generate perturbed spectral representations 212, and determine updates to parameters of perturbation neural network 210 according to the error between i) the perturbed spectral representations 212 and ii) the predicted spectral representations generated by the spectral representation neural network.

[0047] In other words, the training system can train the perturbation neural network 210 using training examples that correspond to speech on which the spectral representation neural network was trained. Then, after the perturbation neural network 210 is trained, the perturbation neural network 210 can generate perturbed spectral representations 212 that (i) correspond to speech on which the spectral representation neural network was not trained, but (ii) still resemble the predicted spectral representations that would be generated by the spectral representation neural network. Thus, the vocoder neural network 220 can be trained using a more diverse training set than could be generated using only the spectral representation neural network.

[0048] In some other embodiments, the perturbation neural network 210 is not explicitly trained to generate a perturbed spectral representation 212 that mimics the predicted spectral representation generated by the spectral representation neural network. For example, the perturbation neural network 210 may be an autoencoder neural network that is configured to process the true-valued spectral representation 202 to generate an embedding of the true-valued spectral representation 202, and then use the embedding to reconstruct the true-valued spectral representation 202 (i.e., to generate a reconstructed spectral representation that is a prediction of the true-valued spectral representation 202).

[0049] The reconstructed spectral representation can be considered a perturbed spectral representation 212 because embedding generally cannot encode all of the information in the true-valued spectral representation 202, and therefore the reconstruction process loses information from the true-valued spectral representation 202. The reconstructed spectral representation can be more similar to the predicted spectral representation generated by the spectral representation neural network than the true-valued spectral representation 202 to the predicted spectral representation generated by the spectral representation neural network, and therefore the reconstructed spectral representation can be a more suitable training input for the vocoder neural network 220 than the true-valued spectral representation 202. An exemplary autoencoder neural network that can be used to generate training inputs for a vocoder neural network is discussed in more detail below with reference to FIG. 3.

[0050] The vocoder neural network 220 can process the perturbed spectral representation 212 to generate audio samples 222 and provide the audio samples to a parameter updating system 230. The parameter updating system 230 can also obtain true-valued audio samples 204 that correspond to the true-valued spectral representation 202. That is, the true-valued spectral representation 202 has been generated from the true-valued audio samples 204 by processing the true-valued audio samples 204 with, for example, a bandpass filter or a Fourier transform.

[0051] Thus, training system 200 can generate a large training data set for vocoder neural network 220 by obtaining true-valued audio samples 204 characterizing a variety of different voices, processing the true-valued audio samples 204 to generate corresponding true-valued spectral representations 202, and then processing the true-valued spectral representations 202 with perturbation neural network 210 as described above.

[0052] The parameter updating system 230 can use the difference between the generated audio samples 222 and the true-valued audio samples 204 to determine updates 232 to the vocoder neural network 220. For example, the parameter updating system 230 can determine the L1 or L2 distance, or the square of the L1 or L2 distance, between the generated audio samples 222 and the true-valued audio samples 204. As another example, the parameter updating system 230 can determine the root-mean-square error between the generated audio samples 222 and the true-valued audio samples 204. In another example, the parameter updating system 230 can generate the updates 232 according to a discretized logistic mixture likelihood, as described, for example, in “PixelCNN++: Improving The PixelCNN With Discretized Logistic Mixture Likelihood And Other Modifications,” by Salimans et al., arXiv:1701.05517.

[0053] The parameter updating system 230 can then backpropagate the error through the vocoder neural network 220, for example, using gradient descent, to generate parameter updates 232. The parameter updating system 230 can also use any suitable optimizer, such as, for example, an Adam optimizer or adaptive gradient (AdaGrad).

[0054] In this manner, the training system 200 can train the vocoder neural network 220 to generate audio samples 222 for a wide variety of speakers. For example, the training system can use true-valued audio samples 204 that characterize speakers who speak with high or low voices, speakers who use vocal fry, speakers with various accents, speakers who speak with high or low tempos, speakers who use nasal voices, speakers who whisper, etc. Thus, the vocoder neural network 220 can be trained during inference to better handle new inputs (i.e., spectral representations representing new speech characteristics) that it may receive after deployment.

[0055] In some embodiments, training system 200 uses perturbation neural network 210 to generate perturbed spectral representations 212 for audio samples that do not have corresponding text input. That is, if a text copy of the audio sample is not available, the spectral representation neural network (configured to process the text input) cannot generate predicted spectral representations that can be used as training inputs for vocoder neural network 220. Thus, training system 200 can generate true-valued spectral representations 202 using the audio samples described above and use perturbation neural network 210 to process the true-valued spectral representations 202 and generate training inputs for the vocoder neural network.

[0056] 2, after perturbed spectral representations 212 are generated, rather than providing perturbed spectral representations 212 directly to vocoder neural network 220, training system 200 can store the perturbed spectral representations 212 for training vocoder neural network 220 at a future time. That is, training system 200 can asynchronously train (i) perturbed neural network 210 and (ii) vocoder neural network 220. For example, training system 200 can generate a training set for vocoder neural network 220 by processing multiple true-value spectral representations 202 with the perturbed neural network to generate corresponding perturbed spectral representations, and then train vocoder neural network 220 at a future time using the generated training set.

[0057] In some embodiments, an external system executes perturbation neural network 210 to generate perturbed spectral representation 212 and provides perturbed spectral representation 212 to training system 200 for training vocoder neural network 220. That is, in these embodiments, training system 200 does not include perturbation neural network 210.

[0058] In some embodiments, training system 200 executes on a single device, such as a graphics processing unit (GPU) or a tensor processing unit (TPU). That is, perturbation neural network 210, vocoder neural network 220, and parameter updating system 230 may execute on the same device. In some other embodiments, execution of training system 200 is distributed across multiple devices. For example, perturbation neural network 210 may execute using one or more first devices, and vocoder neural network 220 and parameter updating system 230 may execute using one or more second devices.

[0059] 3 is a diagram of an example autoencoder neural network system 300. The autoencoder neural network system 300 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.

[0060] The autoencoder neural network system 300 is configured to process the true spectral representation 302 of an audio sample to generate a perturbed spectral representation 322 of the audio sample. The autoencoder neural network system 300 can be used by a training system to generate training inputs for a vocoder neural network configured to process the perturbed spectral representation 322 to generate predictions for the corresponding audio sample. For example, the autoencoder neural network system can serve as the perturbation neural network 210 when training the vocoder neural network 220 described above with reference to FIG. 2. The vocoder neural network can then be deployed in a speech synthesis system together with a spectral representation neural network, such as the spectral representation neural network 110 described above with reference to FIG. 1.

[0061] The autoencoder neural network system 300 includes an encoder subnetwork 310 and a decoder subnetwork 320. The encoder subnetwork 310 is configured to process a true-valued spectral representation 302 to generate an embedding 312 of the true-valued spectral representation 302. The decoder subnetwork 320 is configured to process the embedding 312 of the true-valued spectral representation 302 to reconstruct the true-valued spectral representation 302 from the embedding 312 and generate a perturbed spectral representation 322. As used herein, an embedding is an ordered collection of numbers that represent an input in a particular embedding space; for example, the embedding may be a floating-point or other vector of numbers with a fixed dimensionality.

[0062] In some embodiments, the autoencoder neural network system 300 is pre-trained before being deployed in a training system for a vocoder neural network, such as the training system 200 shown in Figure 2. During pre-training, the autoencoder neural network system 300 can be configured to generate perturbed spectral representations 322 such that the difference between the true spectral representation 302 and the corresponding perturbed spectral representation 322 is minimized or reduced below a threshold.

[0063] For example, a training system for the autoencoder neural network system 300 can use the autoencoder neural network system 300 to process training examples including the true spectral representation 302 to generate each perturbed spectral representation 322. For each training example processed during training of the autoencoder neural network system 300, the training system can determine a reconstruction error that characterizes the difference between i) the true spectral representation 302 and ii) the corresponding perturbed spectral representation 322. For example, the reconstruction error can be the L1 or L2 distance between the true spectral representation 302 and the perturbed spectral representation 322, or its square. As another example, the reconstruction error can be the root-mean-square error between the true spectral representation 302 and the perturbed spectral representation 322.

[0064] The training system can then backpropagate the reconstruction error through the autoencoder neural network system 300 to determine updates to the values ​​of the network parameters of the autoencoder neural network system 300, such as using gradient descent. For example, the training system can determine updates to the values ​​of the network parameters of both the encoder subnetwork 310 and the decoder subnetwork 320 (i.e., the encoder subnetwork 310 and the decoder subnetwork 320 can be trained simultaneously). In other words, the training system can train the autoencoder neural network system 300 in an unsupervised manner.

[0065] In some embodiments, the autoencoder neural network system 300 is a variational autoencoder (VAE).

[0066] Generally, the embedding 312 has a lower dimensionality than the true-valued spectral representation 302, while the perturbed spectral representation 322 has the same dimensionality as the true-valued spectral representation 302. For example, the dimensionality of the embedding 312 can be 1 / 10, 1 / 100, or 1 / 1000 the size of the dimensionality of the true-valued spectral representation 302. That is, when the true-valued spectral representation 302 is processed by the encoder subnetwork 310 to generate the embedding 312, there is a loss of information, and therefore the perturbed spectral representation 322 is only an approximation of the original true-valued spectral representation 302. In other words, the perturbed spectral representation 322 is a degraded or "noisy" version of the true-valued spectral representation 302.

[0067] As noted above, the noisy nature of the perturbed spectral representation 322 can be advantageous when using the perturbed spectral representation 322 as a training input for a vocoder neural network. In some embodiments, the autoencoder neural network system 300 is trained independently of the spectral representation neural network, but in some cases, the degradation introduced by the autoencoder neural network system 300 can be similar to the degradation of the predicted spectral representation generated by the spectral representation neural network relative to the true spectral representation. However, unlike the spectral representation neural network, the autoencoder neural network system 300 can process any true spectral representation 302 corresponding to any speaker, and therefore the autoencoder neural network system 300 can generate a diverse training data set for the vocoder neural network.

[0068] 4 is a flow diagram of an exemplary process 400 for training a first neural network to perform speech synthesis. For convenience, process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a training system, such as training system 200 shown in FIG. 2, suitably programmed in accordance with this specification, can perform process 400.

[0069] The first neural network, also referred to herein as a vocoder neural network, is configured to process the spectral representations of the audio samples to generate predictions of the audio samples. For example, the first neural network can be vocoder neural network 120 described above with reference to Figure 1 or can be vocoder neural network 220 described above with reference to Figure 2.

[0070] After the first neural network is trained, it can be deployed in an inference environment, where it obtains predicted spectral representations of new audio samples from a second neural network. The second neural network, also referred to herein as a spectral representation neural network, is configured to process a text input and generate predicted spectral representations of a vocalization of the text input. For example, the second neural network can be the spectral representation neural network 110 described above with reference to FIG. 1.

[0071] The system obtains a training data set containing a number of training samples (step 402).

[0072] For one or more audio samples in the training data set, and optionally for each audio sample in the training data set, the system generates a true-valued spectral representation of the audio sample (step 404).

[0073] For each audio sample in the training data set for which a true-value spectral representation is generated, the system processes the true-value spectral representation of the audio sample using a third neural network to generate an updated spectral representation of the audio sample (step 406). The third neural network, also referred to herein as a perturbation neural network, is configured to process the true-value spectral representation to generate an updated spectral representation that is similar to the spectral representation generated by the second neural network. For example, the third neural network can be the perturbation neural network 210 described above with reference to FIG. 2.

[0074] The system trains a first neural network using the updated spectral representation (step 408). For example, the system may process the updated spectral representation with the first neural network to generate predicted audio samples and use differences between the predicted audio samples and the true audio samples obtained in step 402 to determine parameter updates for the first neural network.

[0075] This specification uses the term "configured" with respect to systems and computer program components. In the case of a system of one or more computers configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when operated, causes the system to perform the operation or action. In the case of one or more computer programs configured to perform a particular operation or action, it means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0076] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, in computer software or firmware tangibly embodied in computer hardware, including the structures disclosed herein and equivalent forms of those structures, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively, or in addition, the program instructions can be encoded on an artificially generated propagated signal, such as, for example, a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to an appropriate receiver apparatus for execution by the data processing apparatus.

[0077] The term "data processing apparatus" refers to data processing hardware and encompasses all types of apparatus, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. An apparatus can also be or further include special-purpose logic circuitry, such as, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). In addition to hardware, an apparatus can optionally include code that creates an execution environment for a computer program, such as, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0078] A computer program, which may also be referred to as or described as a program, software, software application, app, module, software module, script, or code, can be written in any programming language, including compiled or interpreted, or declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program can, but does not necessarily have to, correspond to a file in a file system. A program can be stored in part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files storing one or more modules, subprograms, or portions of code. A computer program can be deployed to run on one or more computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0079] As used herein, the term "database" is used broadly to refer to any collection of data, which need not be structured in any particular way, or at all, and which may be stored on a storage device in one or more locations. Thus, for example, an index database may contain multiple data collections, each of which may be organized and accessed in various ways.

[0080] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine, and in other cases, multiple engines may be installed and operate on the same one or more computers.

[0081] The processes and logic flows described herein may be implemented by one or more programmable computers that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as, for example, an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0082] A computer suitable for running a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit can receive instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes, or is operatively coupled to, one or more mass storage devices for storing data, such as, for example, magnetic, magneto-optical, or optical disks. However, a computer need not have such devices. Furthermore, a computer can be incorporated in another device, such as, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as, for example, a universal serial bus (USB) flash drive, to name a few.

[0083] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM, DVD-ROM disks.

[0084] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, for allowing the user to provide input to the computer. Other types of devices can similarly be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, such as by sending a web page to a web browser on the user's device in response to a request received from the web browser. Furthermore, a computer can interact with a user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0085] A data processing device for implementing a machine learning model may also include, for example, a dedicated hardware accelerator unit for handling the common and computationally intensive part of machine learning training or creation, i.e., the inference, workload.

[0086] The machine learning model can be implemented and deployed using a machine learning framework, such as, for example, the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0087] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components, e.g., a data server, or includes middleware components, e.g., an application server, or includes front-end components, e.g., a client computer having a graphical user interface, a web browser, or includes an app through which a user can interact with embodiments of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs) such as the Internet.

[0088] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, such as HTML pages, to a user device, e.g., to display data to and receive user input from a user acting as a client interacting with the device. For example, data generated on a user device as a result of user interaction can be received from the device by the server.

[0089] In addition to the above-mentioned embodiments, the following embodiments are also innovative.

[0090] Embodiment 1 is a method for training a first neural network configured to process a spectral representation of an audio sample to generate a prediction of the audio sample, wherein after the first neural network is trained, the first neural network is configured to obtain a spectral representation of the audio sample from the second neural network; wherein the second neural network is configured to process the text input to generate a predicted spectral representation of a vocalization of the text input; The method is: obtaining a training data set comprising a plurality of audio samples; for one or more audio samples in the training data set, and optionally for each audio sample in the training data set, generating a true-value spectral representation of the audio samples; and processing the true-value spectral representation of the audio samples with a third neural network to generate updated spectral representations of the audio samples; and training a first neural network using the updated spectral representation; Here, the third neural network is configured (e.g., trained) to process the ground truth spectral representation to generate an updated spectral representation similar to the spectral representation generated by the second neural network. In this embodiment, the predicted spectral representation of the vocalization of the text input generated by the second neural network is provided, or can be provided, to the first neural network as the spectral representation of the audio sample to be processed by the first neural network.

[0091] Example 2 is the method of example 1, wherein the third neural network: obtaining a true-value spectral representation of the audio samples; processing the true-valued spectral representation with an encoder sub-network to generate an embedding of the true-valued spectral representation; processing the embedding of the true-value spectral representation with a decoder sub-network to generate an updated spectral representation of the audio sample; Alternatively, the third neural network may be one that has been trained using training examples that each include i) a ground-truth spectral representation of audio samples representing a vocalization of the text input, and ii) a predicted spectral representation generated by the spectral representation neural network in response to processing the text input. Such a third neural network is configured to process the ground-truth spectral representation to generate an updated spectral representation similar to the spectral representation generated by the second neural network.

[0092] Example 3 is the method of any one of Examples 1 or 2, wherein the first neural network is configured to (e.g., trained to) generate audio samples that characterize any speech.

[0093] Example 4 is the method of Example 3, wherein the second neural network is configured (e.g., trained) only to generate predicted spectral representations corresponding to a predetermined set of sounds. For example, the second neural network may be trained using training data for the predetermined set of sounds.

[0094] Embodiment 5 is the method of any one of embodiments 1 to 4, wherein the third neural network comprises: obtaining a plurality of training examples, each of which comprises: i) a training true spectral representation of the training audio samples; and ii) a training predicted spectral representation of the training audio samples generated by a second neural network; for one or more of the training examples, and optionally for each training example, processing the training true-value spectral representations with a third neural network to generate updated training spectral representations of the training audio samples; and determining an error between the training updated spectral representation and the training predicted spectral representation; generating parameter updates for a third neural network using the determined errors; and The subjects are trained by performing actions including:

[0095] Embodiment 6 is the method of any one of embodiments 1 to 5, Further comprising deploying the first neural network and the second neural network in the creation environment without including the third neural network.

[0096] Embodiment 7 is the method of any one of embodiments 1 to 5, Further comprising developing the first neural network and the second neural network without including the third neural network.

[0097] Embodiment 8 is any one of embodiments 1 to 7, jointly training the first neural network and the second neural network; fine-tuning the first neural network using the plurality of updated spectral representations generated by the third neural network; Further includes:

[0098] Embodiment 9 is the method of any one of embodiments 1 to 7, wherein the step of training the first neural network with the updated spectral representation comprises: jointly training the first neural network and the second neural network; fine-tuning the first neural network using the updated spectral representation produced by the third neural network; Further includes:

[0099] A tenth embodiment is a method for generating audio samples using a first neural network trained using the method of any one of the first to ninth embodiments.

[0100] Embodiment 11 is the method of embodiment 10, The method further includes deploying the first neural network and the second neural network without using the third neural network as a speech synthesis inference system.

[0101] Embodiment 12 is the method of any one of embodiments 10 or 11, providing a text input to a speech synthesis inference system; receiving, as output from the speech synthesis inference system, audio samples representing predictions of speech corresponding to the input text; Further includes:

[0102] Embodiment 13 is a system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the method of any one of embodiments 1 to 12.

[0103] Embodiment 14 is one or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of any one of embodiments 1 to 12.

[0104] While the specification contains details of many specific embodiments, these should not be construed as limitations on the scope of any invention or on the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features that are described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as working in several combinations and initially claimed as such, one or more features from a claimed combination can in some cases be excluded from that combination, and a claimed combination may be directed to subcombinations or variations of subcombinations.

[0105] Similarly, while operations are illustrated in the figures and claimed in a particular order, this should not be construed as requiring such operations to be performed in the particular order shown, or in any sequential order, or that all of the illustrated operations be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0106] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. In one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0107] 100 Speech Synthesis Inference System 102 Conditional Text Input 104 Noise Input 106 Class 110 Spectral Representation Neural Networks 112 Predictive Spectral Representation 120 Vocoder Neural Network 122 audio samples 200 Training System 202 True Spectral Representation 204 True Audio Samples 210 Perturbed Neural Networks 212 Perturbed Spectral Representation 220 Vocoder Neural Network 222 audio samples 230 Parameter Update System 232 Parameter Update 300 Autoencoder Neural Network System 302 True Spectral Representation 310 Encoder Sub-Network 312 Embed 320 Decoder Subnetwork 322 Perturbed Spectral Representation

Claims

1. 1. A computer-implemented method for training a first neural network to process spectral representations of audio samples to generate predictions of the audio samples, the method comprising: wherein after the first neural network has been trained, the first neural network is configured to obtain the spectral representation of the audio sample to be processed from a second neural network; wherein the second neural network is configured to process a text input to generate a predicted spectral representation of a vocalization of the text input; The method comprises: obtaining a training dataset comprising a plurality of audio samples; For one or more audio samples in the training data set, generating a true-value spectral representation of the audio samples; and processing the true-value spectral representation of the audio sample with a third neural network to generate an updated spectral representation of the audio sample; and training the first neural network using the updated spectral representation; wherein the third neural network is configured to process the ground truth spectral representation to generate an updated spectral representation similar to the spectral representation generated by the second neural network; wherein the third neural network is trained using training examples that each include: i) the true spectral representation of input audio samples representing a vocalization of a particular text input; and ii) a predicted spectral representation generated by the second neural network in response to processing the particular text input. method.

2. The third neural network obtaining said true-value spectral representation of input audio samples; processing the true-valued spectral representation with an encoder sub-network to generate an embedding of the true-valued spectral representation; processing the embedding of the true-value spectral representation using a decoder sub-network to generate the updated spectral representation of the input audio samples; 10. The method of claim 1, wherein the neural network is a pre-trained autoencoder neural network that has been trained to perform operations including:

3. The method of claim 1 or 2, wherein the first neural network is configured to generate audio samples that characterize any voice.

4. 4. The method of claim 3, wherein the second neural network is configured only to generate predicted spectral representations corresponding to a predetermined set of sounds.

5. The third neural network obtaining a plurality of training examples, each of which comprises: i) a training true spectral representation of a training audio sample; and ii) a training predicted spectral representation of the training audio sample generated by the second neural network; For one or more of the plurality of training examples, processing the training true spectral representations with the third neural network to generate updated training spectral representations of the training audio samples; and determining an error between the updated training spectral representation and the predicted training spectral representation; generating parameter updates for the third neural network using the determined errors; 5. The method of claim 1, wherein the subject is trained by performing an action comprising:

6. training the first neural network using the updated spectral representation comprises: jointly training the first neural network and the second neural network; fine-tuning the first neural network using the updated spectral representation produced by the third neural network; 6. The method of any one of claims 1 to 5, comprising:

7. A method for generating audio samples using a first neural network trained using the method of any one of claims 1 to 6.

8. deploying the first neural network and the second neural network without using the third neural network as a speech synthesis inference system.

8. The method of claim 7, further comprising:

9. providing new text input to the speech synthesis inference system; receiving, as output from the speech synthesis inference system, audio samples representing a prediction of the speech corresponding to the new text input; 9. The method of claim 8, further comprising:

10. 10. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 9.

11. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 9.

Images