Generating audio data using an unaligned text input with a generative adversarial network

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for converting conditional text input to audio output using a generative neural network. The generative neural network includes an alignment neural network configured to receive a generative input including the conditional text input and process the generative input to generate an aligned conditional sequence that includes a respective feature representation at each of a plurality of first time steps and is temporally aligned with the audio output.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 035,519, filed June 5, 2020. The disclosure of the prior application is considered part of and is incorporated by reference into the disclosure of this application. BACKGROUND

[0003] This specification relates to generating audio data using a generative adversarial neural network.

[0004] A neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to one or more other layers in the network, i.e., to one or more other hidden layers, to the output layer, or to both. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. SUMMARY

[0005] This specification describes a system implemented as computer programs on one or more computers in one or more locations that generates output audio examples using a feedforward generative neural network.

[0006] In one aspect, this specification describes a computer-implemented method of training a feedforward generative neural network having a plurality of generative parameters. The neural network is configured to generate output audio examples using conditional text inputs. Each conditional text input includes a feature representation at each of a plurality of input time steps, and each output audio example includes a respective audio sample at each of a plurality of output time steps. The feature representation at each input time step is an embedding of a corresponding portion of the original text represented by the conditional text input, e.g., an embedding of a character, a phoneme, or a word. Moreover, the “input time steps” are not aligned in time with the “output time steps.” That is, no information is received that specifies which input time step should be used to generate the output example at each output time step.

[0007] Instead, the method first uses an alignment neural network to predict a duration for each input feature representation, and generates an aligned conditional sequence that is aligned in time with the output audio examples, and generates the output audio examples from the aligned conditional sequence.

[0008] Later, the feature representations are referred to as linguistic feature representations, but the qualifier “linguistic” is optional.

[0009] The feedforward generation neural network is configured to receive a generation input comprising a conditional text input. The generation neural network processes the generation input to generate an audio output.

[0010] The generation neural network is trained. The training process starts by obtaining a training conditional text input. A next step of the training comprises processing a training generation input comprising the training conditional text input using the feedforward generation neural network. The generation neural network is configured with a current set of values which is used as generation parameters to generate a training audio output. The processing step of the training further comprises processing the training generation input using the alignment neural network to generate an aligned conditional sequence, in particular temporally aligned with the audio output. The aligned conditional sequence comprises a feature representation at each of a plurality of first time steps. The feature representation can be a representation of the audio output to be generated for the time step, for example it can be an audio feature representation.

[0011] The processing step of the training further comprises a step of processing the aligned conditional sequence using the generator neural network to generate the training audio output. The training further comprises a step of processing the training audio output using each of one or more discriminators. Each discriminator predicts whether the training audio output is a real audio example or a synthetic audio example. The training step further comprises a step of determining a final prediction using the respective predictions of the one or more discriminators. The training step further comprises a step of determining an update to the current values of the generation parameters to increase a first error of the final prediction.

[0012] In another aspect, the present specification describes a computer-implemented method of generating an output audio example using a conditional text input. Each conditional text input comprises a respective feature representation at each of a plurality of input time steps, which can be referred to as a linguistic feature representation. The method of generating an output audio example comprises a step of obtaining a conditional text input. The method of generating an output audio example comprises a step of processing a generation input using a feedforward generation neural network, the generation input comprising the conditional text input. The generated output audio comprises an audio sample at each of a plurality of output time steps. The processing step of generating an output audio example comprises a first step of processing the generation input using an alignment neural network to generate an aligned conditional sequence, in particular temporally aligned with the audio output. The aligned conditional sequence comprises a respective feature representation at each of a plurality of first time steps, which can be referred to as an audio feature representation. The processing step of generating an output audio example comprises a second step of processing the aligned conditional sequence using a generator neural network to generate the audio output.

[0013] Any of a range of different techniques can be used to generate the alignment condition sequence; some particular techniques will be described later. When an alignment neural network is used to generate the alignment condition sequence, the alignment neural network can be trained in any convenient manner, e.g., but not necessarily as described herein. Generally, the features of the alignment neural network and the method of generating the alignment condition sequence can be the same for training and inference. Thus, for example, generating the alignment condition sequence can involve processing the generation input using a first subnetwork to generate an intermediate sequence having a respective intermediate element at each of a plurality of intermediate time steps, processing the intermediate sequence using a second subnetwork to generate, for each intermediate element, a length prediction that characterizes a predicted temporal length of the intermediate element, and processing the respective length predictions to generate the alignment condition sequence. The respective intermediate elements can have variable lengths (durations). By interpolating (e.g., non-uniform or non-linear interpolation) the intermediate sequence using the respective length predictions of the intermediate elements to generate a feature representation at each of the first time steps, the alignment neural network can generate the alignment condition sequence. For example, the length predictions can be used to determine weights for a weighted combination of the intermediate elements that determines the feature representation at the first time steps.

[0014] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0015] The feedforward generation neural network described in this specification can generate output examples more quickly than prior art that relies on autoregressive generation neural networks, e.g., WaveNet, which is important for applications where fast batch inference is important. Autoregressive neural networks generate output examples across multiple output time steps by performing a forward pass at each output time step. At a given output time step, the autoregressive neural network generates a new output sample to include in the output example conditioned on the output samples that have already been generated. This can consume a large amount of computational resources and take a large amount of time. On the other hand, the feedforward generation neural network can generate the output example in a single forward pass while maintaining a high degree of quality in the generated output example. This greatly reduces the amount of time and computational resources required to generate the output example relative to the time consumed by the autoregressive neural network.

[0016] Some existing training systems for speech synthesis systems require that the conditional text input and the ground truth audio output in the training examples be aligned, which requires a large, manually managed set of training data. Generating this training data can be very expensive and time consuming. Using the techniques described in this specification, the training system can automatically learn the optimal alignment of the conditional text input and the audio output.

[0017] Some existing speech synthesis systems include large pipelines of multiple different subsystems, each of which must be designed and trained separately from the rest of the pipeline. Using the techniques described in this specification, a training system can teach a single end-to-end system to generate audio output from conditional text input, significantly reducing the complexity of the system and the time required for training, thereby saving significant time and computational resources.

[0018] Other existing techniques rely on reversible feedforward neural networks trained by using a probabilistic density distillation autoregressive model, such as Parallel WaveNet. Training in this way allows the reversible feedforward neural network to generate speech signals that sound realistic and correspond to the input text without needing to model all possible variations that occur in the data. The feedforward generative neural networks described in this specification can also generate realistic audio samples that stick to the input text as truth without needing to explicitly model the data distribution of the audio data, but can do so without the distillation and reversibility requirements of the reversible feedforward neural networks.

[0019] Using discriminators that only process audio data samples allows the system to distinguish between low-dimensional distributions. Assigning a particular window size to each discriminator allows the discriminators to operate on different frequencies of the audio samples, improving the realism of the audio samples generated by the feedforward generative neural network. Using discriminators that only process audio data samples also reduces the computational complexity of the discriminators, which can allow the system to train the feedforward generative neural network more quickly.

[0020] Using dilated convolutional layers further widens the receptive field of the feedforward generative neural network and the discriminators, allowing the respective networks to learn dependencies at various frequencies, such as long-term frequencies and short-term frequencies.

[0021] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1A is a diagram of an example text-to-speech inference system.

[0023] Figure 1B is a diagram of an example text-to-speech training system.

[0024] FIG. 2 is a diagram of an example generator block showing layers of a generative adversarial network.

[0025] Figure 3 is a diagram of an example discriminator neural network system.

[0026] Figure 4 is a diagram of an example unconditional discriminator block and an example conditional discriminator block.

[0027] Figure 5 is an example of a dynamic time warping process that illustrates an optimal alignment path.

[0028] Figure 6 is a flowchart of an example process for training a generative neural network.

[0029] Figure 7 is a flowchart of an example process for generating an audio output.

[0030] In the various drawings, like reference numbers and designations indicate like elements. DETAILED DESCRIPTION

[0031] This specification describes a system that trains a generative neural network to generate output audio examples using conditional text inputs. The system can train the generative neural network in a adversarial manner using an alignment neural network, a decoder neural network, and one or more discriminator networks. The trained network can receive a conditional text input and convert it to a generated audio.

[0032] A text-to-speech system includes systems for performing two main operations. The first system is a training system for a feedforward generative neural network that maps conditional text inputs and optionally additional information to output audio examples. The second system is an inference system that uses the trained feedforward generative neural network to perform inference, i.e., to map new conditional text inputs to output audio examples using the feedforward generative neural network.

[0033] Figure 1A is a diagram of an example inference system 110A of the text-to-speech system 100. The inference system 110A is an example of a system implemented as a computer program on one or more computers in one or more locations, where the system, components, and techniques described below can be implemented.

[0034] The inference system 110A receives a conditional text input 102 as input and uses the trained feedforward speech generative neural network to generate an audio output 132, i.e., to generate an output audio example that includes samples of audio waves at each output time step in a sequence of output time steps. The audio samples at a given time step can be amplitude values of audio waves or compressed or companded amplitude values.

[0035] Each conditional text input 102 represents an input text that conditions a corresponding output audio example 132 and includes a sequence of one or more (linguistic) feature representations. Typically, the (linguistic) feature representations include word-level, phoneme-level, or character-level embeddings of the text.

[0036] In this specification, an embedding is an ordered collection of numerical values representing an input in a particular embedding space. For example, an embedding can be a vector of floating point numbers or other numerical values with a fixed dimensionality. It can be provided by an embedding neural network layer.

[0037] The generative neural network of inference system 110A includes an alignment neural network 120 and a decoder neural network 130.

[0038] Before being used for inference, training system 110B trains alignment neural network 120 and decoder neural network 130 on training data 112. Training system 110B operates to optimize the performance of alignment neural network 120 and decoder neural network 130 using an existing set of training data 112 as a ground truth data set. Additional details of the training system will be described below with reference to Figure 1B Description.

[0039] Inference system 110A is configured to receive a conditional text input 102 at an input and to process the conditional text input 102 through alignment neural network 120 and decoder neural network 130 to generate an audio output 132. In some implementations, alignment neural network 120 and decoder neural network 130 are configured as feedforward neural networks, i.e., alignment neural network 120 and decoder neural network 130 generate audio output 132 in a single forward pass.

[0040] In general, the neural networks of alignment neural network 120 or decoder neural network 130 can have any appropriate neural network architecture capable of generating the described results.

[0041] Alignment neural network 120 processes the conditional text input to generate an aligned conditional sequence that includes a respective feature representation at each of a plurality of first time steps and that is temporally aligned with the generated audio output. To be “temporally aligned” means that each feature representation of the conditional sequence corresponds to a different time window in the generated audio output, in particular without overlap between the different time windows. That is, alignment neural network 120 is trained such that the feature representations of the aligned conditional sequence 122 for a given conditional input are temporally aligned with elements of the ground truth audio output for the conditional input. The feature representations of the aligned conditional sequence can be in a learned abstract feature space. The aligned conditional sequence 122 can have a smaller frequency than the audio output, i.e., the frequency of the first time steps can be lower than the frequency of the output time steps. As a particular example, the aligned conditional sequence can have a frequency of 200 Hz, while the audio output can have a frequency of 24 kHz. In some implementations, the aligned conditional sequence 122 can be input into decoder neural network 130.

[0042] The alignment neural network 120 can include a first subnetwork that processes the conditioned text input to generate an intermediate sequence having a respective intermediate element at each of a plurality of intermediate time steps. For example, the first subnetwork can process the conditioned text input using one or more dilated convolutional neural network layers. In some implementations, the first subnetwork also processes a sampling noise embedding, e.g., z ~ N(0, I d ), where N() is a normal distribution and I is an identity matrix of size d. For example, the first subnetwork can use the sampling noise embedding to modulate scalar and shift parameters of one or more batch normalization layers. In some implementations, the first subnetwork can also or instead use a speaker identification embedding to modulate scalar and shift parameters of one or more batch normalization layers.

[0043] Intermediate elements (later referred to as tokens) are characterized by their variable length, in this context variable duration. In some implementations, the intermediate sequence has the same length as the conditioned text input 102, i.e., each intermediate element can be an embedding of a corresponding language feature representation in the conditioned text input 102 that encodes contextual information from surrounding language feature representations.

[0044] The alignment neural network 120 can include a second subnetwork that processes the intermediate sequence to generate, for each intermediate element, a length prediction that characterizes a predicted temporal length of the intermediate element. That is, the length prediction represents a duration of time that speech represented by the intermediate element will be spoken in the audio output. For example, the second subnetwork can process the intermediate sequence using a point-wise multilayer perceptron, i.e., the same multilayer perceptron can process each individual intermediate element of the intermediate sequence. In some implementations, the second subnetwork (e.g., the point-wise multilayer perceptron) can also process the sampling noise embedding and / or the speaker identification embedding.

[0045] The alignment neural network 120 can then process the length predictions to generate the aligned conditioned sequence 122. In some implementations, the alignment neural network 120 uses the respective length predictions of the intermediate elements to determine a cumulative length prediction, e.g., by computing a sum of the length predictions. The alignment neural network 120 can then use i) the cumulative length prediction and ii) a predetermined frequency of the first time step in the aligned conditioned sequence 122 to determine a number of the first time step in the aligned conditioned sequence. That is, the alignment neural network determines how many intermediate elements should be included in the conditioned sequence.

[0046] By interpolating the intermediate sequence using the respective length predictions of the intermediate elements - e.g. non-uniform or non-linear interpolation - to generate a feature representation at each first time step, the alignment neural network 120 can generate the aligned condition sequence 122. That is, for each first time step, the alignment neural network can generate a corresponding (audio) feature representation by using the interpolation to process the intermediate elements - e.g. by determining a weighted combination of the intermediate elements - (such that, in this sense, the interpolation can be considered non-uniform).

[0047] For example, the alignment neural network 120 can determine, for each intermediate element, a predicted time-in-position from the respective length prediction of the intermediate element, e.g. the center point of each length prediction, where l m is the length prediction of the m-th intermediate element.

[0048] The alignment neural network 120 can then determine, for each intermediate element and each first time step, a respective weight value w. For example, for each intermediate element n and each first time step t, the alignment neural network 120 can compute

[0049]

[0050] where N is the number of intermediate elements, c n is the predicted time-in-position of the n-th intermediate element, and σ 2 is a predetermined temperature parameter. Thus, for example, the weight w can depend on a softmax over the squared difference between t and c n scaled by the temperature parameter.

[0051] The alignment neural network 120 can then determine, for each first time step, a corresponding feature representation in the aligned condition sequence 122 by combining the intermediate elements using the respective weight values corresponding to the first time step and each intermediate element. For example, the alignment neural network 120 can determine the feature representation for the first time step t by computing a weighted sum of the intermediate elements, where each intermediate element is weighted by a corresponding computed weight, where h n is the n-th intermediate element.

[0052] The operation of the alignment neural network is described in more detail below with reference to Fig. 2.

[0053] In some implementations, the aligned neural network 120 may also receive as input an identifier of the category 104 to which the audio output 132 should belong. Category 104 may be a member of a set of possible categories. For example, category 104 may correspond to a specific speaker or speaker category that the audio output 132 should sound like. This category may include, for example, a speaker identifier or speaker identifier embedding for a speaker category, such as female or male, young or old, or having a regional accent. That is, the audio output 132 may depict a specific speaker or speaker category speaking the input text.

[0054] The decoder neural network 130 (also known as the generator neural network 130) can process the alignment condition sequence 122 to generate audio output 132. The generator neural network 130 may include a sequence of groups of one or more convolutional neural network layers. Each group of convolutional neural network layers may include one or more dilated convolutional layers.

[0055] like Figure 1A As shown, the generator neural network 130 or decoder neural network 130 includes the ability to embed additional conditions 106, which can include emotion or tone in the generated audio output 132. For example, the conditional text input 102 can include linguistic features characterizing the text input. For example, additional conditions 106 (such as emotion, tone, or even accent) can be added through the input of the generator neural network. In addition to being able to select a speaker ID or category 104, additional conditions 106 also allow the user to include specific linguistic features that may not be present in the selected category 104, which can include adjusting pitch (which can be represented by the logarithmic fundamental frequency log F0 of the input time step) or adding pitch shift. The output of the generator neural network 130 includes an audio output 132 generated from the selected speaker ID or category 104 with user-determined additional conditions 106 such as pitch, emotion, tone, or pitch shift.

[0056] In some implementations, the decoder neural network 130 acquires an alignment condition sequence 122 at its input. The decoder neural network 130 receives the alignment condition sequence 122 and generates a generated speech sequence representing the alignment condition sequence 122. Optionally, the decoder neural network 130 includes a second input for receiving an additional condition 106, which can be applied to the speech sequence as it is generated. Condition 106 may include phase information removed from the category input 104, which can alter the accent, or the pitch or emotion of the generated speech.

[0057] In general, the input time step sequence of the aligner block of inference system 110A and the output time step sequence generated at the output of the decoder block represent the same time period, e.g., 1, 2, 5, or 10 seconds. As a particular example, if the time period is 2 seconds, then the conditioned input 102 can include 400 input time steps (resulting in a time step frequency of 200 Hz), while the audio output 132 can include 48,000 time steps (resulting in an audio sample frequency of 24 kHz). Thus, the neural network of the inference system can generate an audio sample for multiple output time steps (in this case, 120) for each single input time step.

[0058] In some implementations in which the neural network includes a sequence of one or more generator blocks, due to the difference in frequency of the input time steps and the output time steps, one or more of the generator blocks in the neural network can include one or more respective upsampling layers. The dimensionality of the layer output of each upsampling layer is greater than the dimensionality of the layer input of the upsampling layer. The total degree of upsampling across all generator blocks in the neural network can be proportional to the ratio of the frequency of the output time steps and the input time steps.

[0059] Figure 1B FIG. 1 is a diagram of an example training system 110B that is a text-to-speech system. Training system 110B is an example of a system implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0060] System 110B trains a feedforward neural network that includes an alignment neural network 120, a decoder neural network 130 using one or more discriminator neural networks 140 and a parameter update system 150.

[0061] Training system 110B operates by processing training examples from a training data set using a feedforward generative neural network. Each training example can include a training conditioned text input and a ground truth audio output. In implementations, the training conditioned text input and the ground truth audio output are unaligned. That is, the training example does not include a monotonic mapping from the linguistic feature representation of the training conditioned text input to the elements of the ground truth audio output. In some implementations, the feedforward generative neural network also receives as input an identification of a category under which the audio output should be, e.g., an identification of a speaker of the audio output.

[0062] Training system 110B is configured to receive training data 112 as input to the alignment neural network 120. The training data 112 includes a plurality of training examples, each training example including at least (i) a text segment, and (ii) a waveform representing the text being spoken, i.e., a ground truth speech output. The training data 112 is provided as input to the alignment neural network 120, which generates an aligned conditioned sequence 122.

[0063] After the generator neural network generates the audio output 132, the training system 110B can provide the audio output 132 to the discriminator neural network system 140. The training system 110B can train the discriminator neural network system 140 to process the audio output 132 and generate a prediction 142 of whether the audio output 132 is real (i.e., an audio sample captured in the real world) or synthetic (i.e., an audio sample generated by the generator neural network). The generator neural network includes the alignment neural network 120 and the decoder neural network 130.

[0064] As previously specified, the discriminator neural network system 140 can have any suitable neural network architecture suitable for generating audio output predictions. As a particular example, the discriminator neural network system 140 can include one or more discriminators, each of which processes the audio output 132 and predicts whether the audio output 132 is real or synthetic. Each discriminator can include a sequence of convolutional neural network layers, referred to as a “discriminator block.” An example discriminator block is described below with respect to Figure 3 Description.

[0065] In some implementations, one or more discriminators of the discriminator neural network system 140 include one or more conditional discriminators and one or more unconditional discriminators. A conditional discriminator receives, as input, i) the audio output 132 generated by the generator neural network and ii) the conditional text input 102 used by the generator neural network to generate the audio output 132. An unconditional discriminator receives, as input, the output audio 132 generated by the generator neural network, but does not receive, as input, the conditional text input 102. Thus, in addition to measuring the general realism of the audio output 132, a conditional discriminator can measure the extent to which the audio output 132 corresponds to the input text characterized by the conditional text input 102, while an unconditional discriminator only measures the general realism of the audio output 132. In some implementations, an unconditional discriminator can also process the speaker identity embedding. An example discriminator neural network system is described in more detail below with respect to Figure 4 Description.

[0066] The training system 110B can train the feedforward generative neural network in an adversarial manner using one or more discriminators. That is, each of the discriminators predicts whether the output of the feedforward generative neural network is a real example of audio data or a synthetic example. The training system 110B system can combine the respective predictions of all of the discriminators, e.g., by computing an average of the predictions or determining a vote of each prediction, to generate a final prediction. The training system 110B can update the parameters of the feedforward generative neural network to increase an error of the final prediction. In doing so, the parameter update system 150 can obtain the predictions 142 generated by the discriminator neural network system 140 and determine parameter updates 152 from an error of the predictions 142. The parameter update system 150 can update the parameters of each discriminator to reduce the error of the final prediction or the error of the respective predictions of the discriminator system 140. Additionally, the parameter update system 150 can apply the parameter updates 152 to the parameters of the generative neural network comprising the alignment neural network 120 and the decoder neural network 130 and the discriminator neural network system 140. That is, the training system 110B can train the generative neural network and the discriminators in the discriminator neural network system 140 such that the parameters in each of the alignment neural network 120, the decoder neural network 130, and the discriminator neural network 140 can be trained and updated simultaneously.

[0067] In general, the parameter update system 150 determines parameter updates 152 to the parameters of the generative neural network to increase an error of the predictions 142. For example, if the discriminator neural network system 140 correctly predicts that the audio output 132 is synthetic, then the parameter update system 150 generates parameter updates 152 to the parameters of the generative neural network to improve the authenticity of the audio output 132. In doing so, the discriminator neural network system 140 can incorrectly predict that the next audio output 132 is real.

[0068] Conversely, the parameter update system 150 determines parameter updates 152 to the parameters of the discriminator neural network system 140 to reduce an error of the predictions 142. For example, if the discriminator neural network system 140 incorrectly predicts that the audio output 132 is real, then the parameter update system 150 generates parameter updates 152 to the parameters of the discriminator neural network system 140 to improve the predictions 142 of the discriminator neural network system 140.

[0069] In some implementations in which the discriminator neural network system 140 comprises multiple different discriminators, the parameter update system 150 determines parameter updates 152 for each discriminator using the predictions 142 output by the discriminator neural network system 140. That is, the parameter update system 150 determines parameter updates 152 for each particular discriminator using the same combined prediction 142 generated by combining the respective predictions of the multiple discriminators, regardless of the respective prediction generated by the particular discriminator.

[0070] In some other implementations in which the discriminator neural network system 140 includes multiple different discriminators, the parameter update system 150 determines a parameter update 152 for each discriminator using the respective prediction generated by the discriminator. That is, the parameter update system 150 generates a parameter update 152 for a particular discriminator in order to improve the respective prediction generated by the particular discriminator. An additional advantage is that improving the prediction of a particular generator will indirectly improve the combined prediction 142 output by the discriminator neural network system 140 because the combined prediction 142 is generated from the respective predictions generated by the multiple discriminators.

[0071] During training, the training system 110B can also provide the real audio samples 108 to the discriminator neural network system 140. Each discriminator in the discriminator neural network system 140 can process the real audio samples 108 to predict whether the real audio samples 108 are real or synthetic examples of audio data. Again, the discriminator neural network system 140 can combine the respective predictions of each of the discriminators to generate a second prediction 142. The parameter update system 150 can then determine a second parameter update 152 to the parameters of the discriminator neural network system 140 from the error of the second prediction 142. Typically, the training system 110B does not use the second prediction corresponding to the real audio samples 108 to update the parameters of the generator neural network.

[0072] In some implementations in which the generator neural network has been trained to generate synthetic audio outputs 132 belonging to a class 104, the discriminator neural network system 140 does not receive as input an identification of the class 104 to which the received (real or synthetic) audio examples belong. However, the real audio samples 108 received by the discriminator neural network system 140 can include audio samples belonging to each class 104 in the set of classes.

[0073] As a particular example, the parameter update system 150 can determine the parameter update 152 using a Wasserstein loss function, which is:

[0074] D(x) - D(G(z)),

[0075] where D(x) is the likelihood that the real audio sample 108 is real as assigned by the discriminator neural network system 140, G(z) is the synthesized audio output 132 generated by the generative neural network, and D(G(z)) is the likelihood that the synthesized audio output 132 is real as assigned by the discriminator neural network system 140. The goal of the generative neural network is to minimize the Wasserstein loss by maximizing D(G(z)), i.e., by causing the discriminator neural network system 140 to predict that the synthesized audio output 132 is real. The goal of the discriminator neural network system 140 is to maximize the Wasserstein loss, i.e., to correctly predict both real and synthesized audio examples.

[0076] As another particular example, the parameter update system 150 can use the following loss function:

[0077] log(D(x)) + log(l - D(G(z)))

[0078] where again, the goal of the generative neural network is to minimize the loss, and the goal of the discriminator neural network system 140 is to maximize the loss.

[0079] The training system 110B can backpropagate the loss through the generative neural network and the discriminator neural network system 140, thereby training both networks simultaneously.

[0080] In some implementations, the training system 110B can determine an aligner length loss characterizing the length error of the training audio output 132, and determine an update to the parameters of the feedforward generative neural network using the aligner length loss. For example, the aligner length loss can be

[0081]

[0082] where L is the length of the real-valued audio 108 corresponding to the training conditional text input 112, N is the number of intermediate elements, and l n is the length prediction of the nth intermediate element generated by the alignment neural network. That is, the training system 110B can determine an update to the parameter values of the alignment neural network based on the error of the length prediction, i.e., the difference between the length of the real-valued output (in time) and the total predicted length (in time) represented by the features predicted by the alignment neural network.

[0083] In some implementations, one or more discriminators in the discriminator neural network 140 include one or more spectrogram discriminators. The training system 110B can process the training audio outputs 132 to generate spectrograms of the training audio outputs 132 (as used herein, a spectrogram is a shorthand reference to the corresponding data). The spectrograms can be, for example, log-scalar mel-spectrograms, e.g., derived from a STFT (short-time Fourier transform). The training system 110B can process the spectrograms using a spectrogram discriminator to generate the predictions 142 of whether the training audio outputs 132 are real or synthetic audio examples. In some implementations, the spectrogram discriminator also processes the speaker ID or class 104 embeddings. The spectrogram discriminator can include one or more convolutional neural network layers that process the spectrograms. That is, the set of discriminators 140 can include one or more discriminators that operate on spectrograms rather than audio examples. The discriminators that operate on spectrograms can be trained as described above for the remaining discriminators 140 in the set.

[0084] In some implementations, the training system 110B can determine a spectrogram prediction loss and use the spectrogram prediction loss to determine updates to the parameters 152 of the feedforward generation neural network. That is, the training system 110B can generate i) a spectrogram of the training audio outputs 132 and ii) a spectrogram of the ground truth audio 108 outputs and determine a difference between the two spectrograms.

[0085] In some implementations, the spectrogram prediction loss is

[0086]

[0087] where S gt is the ground truth spectrogram, S gen is the predicted spectrogram, T is the number of time points in the spectrograms, and F is the number of frequency bins in the spectrograms.

[0088] In some other implementations, the training system 110B can use an implementation of dynamic time warping to determine the spectrogram prediction loss. That is, because the two spectrograms are not necessarily perfectly aligned, the spectrogram can determine an optimal path that optimizes the alignment of the two spectrograms through the two respective spectrograms. The spectrogram loss and dynamic time warping computation are discussed in more detail below with respect to Figure 5

[0089] As a particular example, the training system 110B can compute a combined loss function,

[0090]

[0091] where L G,adv is the adversarial loss, L pred is the prediction loss, L length ​is a length loss, and λ is a scalar weight. The training system 110B can use the combined loss to determine updates to the parameters 152 of the feedforward generative neural network.

[0092] FIG. 2 is a diagram of a generator neural network 200.

[0093] The generator neural network 200 includes the alignment neural network 120 (including a set of alignment blocks) and the decoder neural network 130 (including a set of generator blocks).

[0094] The generator neural network 200 is configured to process a conditional text input 206 and generate an audio output 208. The generator neural network 200 is configured to receive a text block 210 or, in some cases, a set of phonemes 200 that has been generated from the text block 210 by a phonemizer 218 as input.

[0095] In some implementations, the generator neural network can also be configured to receive a speaker ID 212, a latent vector 214, and a set of random offsets 216 used during training to select the feature window that will be provided to the decoder neural network as input.

[0096] One or more generator blocks of the generator neural network 200 can also receive an identification of the speaker ID 212 as input.

[0097] When used, the phonemizer 218 receives the conditional text input 210 and converts it to phonemes, i.e., to a set of independent sound units that are perceptible in a selected language.

[0098] The alignment neural network 120 can include a first subnetwork that includes a first convolutional layer 221 and a plurality of first blocks 222 and processes the conditional text input to generate an intermediate sequence having a respective intermediate element at each of a plurality of intermediate time steps. For example, the first subnetwork can first process an embedding of the text input 210 or the phonemes 220 using the first convolutional layer 221 and then process the output of the first convolutional layer 221 using the plurality of first blocks 222, each first block 222 processing the input to the block using one or more dilated convolutional neural network layers. In some implementations, the blocks in the first subnetwork also process a sampling noise embedding, i.e., a latent vector 214, e.g., z ~ N(0, I d ), where I is a d-by-d identity matrix; the size of d is arbitrary, e.g., d = 128. For example, the first subnetwork can use the sampling noise embedding to modulate the scalar and shift parameters of one or more batch normalization layers in the blocks. In some implementations, the first subnetwork can also or instead use the speaker identification embedding to modulate the scalar and shift parameters of one or more batch normalization layers in the blocks.

[0099] In some implementations, the intermediate sequence has the same length as the conditional text input 102, i.e., each intermediate element can be an embedding of the corresponding language feature representation in the conditional text input 102 that encodes context information from surrounding language feature representations.

[0100] The first blocks 222 can be configured with any appropriate architecture for generating the desired output. As shown in FIG. 2, the first subnetwork includes a first convolutional block followed by a set of 10 dilated convolutional blocks. The specific number of dilated convolutional blocks can vary based on any of memory requirements, performance, speed, or various other factors.

[0101] In some implementations, each first block 222 can include multiple dilated convolutional neural network layers with increasing dilation. For example, the dilation value of each dilated convolutional neural network layer can double from an initial dilation and then return to the initial dilation in the next aligner block.

[0102] In some implementations, the first blocks 222 are configured in series such that the output of a previous block is provided directly as input to the next block in the set of aligner blocks 222. Each subsequent aligner block in the sequence of aligner blocks receives as input the block output generated by the previous aligner block in the sequence of aligner blocks.

[0103] The alignment neural network also includes a second subnetwork that processes the intermediate sequence generated by the last first block to generate, for each intermediate element, a length prediction that characterizes a predicted temporal length of the intermediate element. That is, the length prediction represents a duration of time that the speech represented by the intermediate element will be spoken in the audio output. For example, the second subnetwork can process the intermediate sequence using a point-wise multi-layer perceptron (MLP), or can process the intermediate sequence using two second blocks 224 and 226, each of which includes a batch normalization layer, an activation layer (shown as a ReLU layer), and a convolutional neural network layer to generate the length prediction or to generate an input to the point-wise MLP.

[0104] In some implementations, the second subnetwork is also conditioned on the speaker ID 212, the noise input (i.e., the latent vector 214), or both. For example, the batch normalization layers in blocks 224 and 226 can be conditional batch normalization layers that are conditioned on the output of respective linear layers 215. The linear layers 215 can combine, e.g., concatenate, a linear embedding of the speaker ID 212 that the output audio should belong to (e.g., a particular speaker that the output audio should sound like) with the noise input 214 to generate a combined representation, and provide the combined representation to the conditional batch normalization layers. The conditional batch normalization layers can then be conditioned on the combined representation.

[0105] Length prediction is a length N vector 228 of predicted token lengths of the N elements in the output of the block 222, i.e., comprising a respective predicted token length for each intermediate element in the intermediate sequence. Each predicted token length in the length N vector has a predicted length l m where m is the m-th predicted token length in the N-length vector of predicted token lengths, and corresponds to the m-th intermediate element in the output of the block 222, i.e., in the intermediate sequence. The total length l n of the tokenized conditional text input 210 is equal to the sum of all l m of the tokens. The associated length l m of each phoneme indicates the predicted duration of the pronunciation associated with the predicted phoneme (in time). The length of a token can vary between each token, and the total length is the sum of all tokens in the predicted token vector. The (output) tokens are arranged in a vector, each token (also referred to as “intermediate element”) having an associated length l m .

[0106] In some implementations, the alignment neural network receives the predicted lengths 228 and determines a center position c n 230 of each token. The center is calculated by the equation The token centers 232 computed for all N intermediate elements (or “tokens”) are provided as output and used as input to a window aligner block 234 in the generator neural network 200 for optimizing the alignment of the intermediate sequence to the time steps t.

[0107] In some implementations, during training, the window aligner block 234, represented in Fig. 2 as an alignment grid, aligns the predicted tokenized sequence 228 with a random set of time offsets. For training, the decoder neural network can need to receive a fixed length input and generate a fixed length output. Therefore, in this case, the alignment step can be performed for a shortened snapshot or a shortened window of 210 in order to provide a fixed length input to the decoder neural network 130. For inference, the alignment step has to be performed on the entire input sequence.

[0108] In some implementations, the operation of the window aligner block 234 can be interpreted as a grid. Along the horizontal axis of the grid is the number n, which corresponds to the n-th token in the predicted tokenized sequence 228. The vertical axis corresponds to the time t, and each step can be aligned to an offset h from the beginning of the sequence. The offsets can be randomly sampled from a distribution and provided as input to the neural network. In some implementations, the alignment step only needs to consider offsets h for which a token of interest overlaps with the offset. As shown in Fig. 2, the alignment of interest during training is represented by the non-grayed-out blocks, e.g., the white block area and the darkened blocks within the white block area representing the overlap region.

[0109] The window aligner block 234 generates a weight for each token n and a given time t by the following equation:

[0110]

[0111] The block 234 then generates a feature representation for time t in the aligned condition sequence by generating a weighted combination of the N feature representations in the intermediate sequence using the determined weights for time t where w t n is the weight for the n-th token for time t.

[0112] During training, the neural network only needs to generate aligned condition sequence feature representations for times in a fixed size window defined for the offset. After training, during inference, the neural network generates an aligned condition sequence which represents the entire text input 210.

[0113] In some implementations, the last block 240 of the alignment neural network 202 is a batch normalization and activation layer. The last block 240 has an input for receiving the tokenized output sequence of optimal alignments from the transpose layer 238 and an input for receiving a linear condition 241. The linear condition is combined with the batch normalization and activation layer to embed the language condition features into the output text, such as the speaker ID 212. The output of the last block is the aligned condition sequence which can be decoded by the decoder block 204 where an audio output is generated.

[0114] Layer Temporal dimension Frequency Number of channels Alignment condition sequence 400 200 Hz 256 Input convolutional layer 400 200 Hz 768 G block 400 200 Hz 768 G block 400 200 Hz 768 G block, upsample x2 800 400 Hz 384 G block, upsample x2 1600 800 Hz 384 G block, upsample x2 3200 1600 Hz 384 G block, upsample x3 9600 4800 Hz 192 G block, upsample x5 48000 24 kHz 96 Output convolutional layer 48000 24 kHz 1

[0115] Table 1: Example generator neural network architecture

[0116] Table 1 describes an example architecture of the generator neural network 200. The example shown in Table 1 is for illustrative purposes only and many different configurations of the generator neural network are possible. The input to the generator neural network 200 is a language feature vector for each input time step corresponding to 400 input time steps of two seconds of audio (200 Hz frequency). Each language feature vector corresponding to a respective input time step includes 567 channels.

[0117] As described above, the decoder neural network can have any appropriate architecture that allows the decoder to map the final aligned sequence to an audio example. As one example, the decoder can have Bi' nkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. In ICLR, 2020 the architecture described in Bi' nkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. In ICLR, 2020, which is incorporated by reference.

[0118] Figure 3 is a diagram of an example discriminator neural network system 300.

[0119] The discriminator neural network system 300 can be a component of a training system, such as the training system 110B depicted in FIG. 1, that is configured to train the generator neural network 200. The discriminator neural network system 300 has been trained to receive an audio example 302 and generate a prediction 306 of whether the audio example is a real audio example or a synthetic audio example generated by the generator neural network. The audio example can include a respective magnitude value at each output time step in a sequence of output time steps (referred to as “output” time steps because the audio example can have been an output of the generator neural network). The audio example corresponds to a conditional text input 304 that includes a respective vector of linguistic features at each input time step in a sequence of input time steps. Note that even if the audio example 302 is a real audio example, the conditional text input 304 still corresponds to the audio example 302.

[0120] The discriminator neural network system 300 can include one or more unconditional discriminators, one or more conditional discriminators, or both. In the depicted example, the discriminator neural network system 300 includes five unconditional discriminators (including the unconditional discriminator 320) and five conditional discriminators (including the conditional discriminator 340). Figure 3 In the depicted example, the discriminator neural network system 300 includes five unconditional discriminators (including the unconditional discriminator 320) and five conditional discriminators (including the conditional discriminator 340).

[0121] In some implementations, each of the discriminators in the discriminator neural network system 300 does not process the entire audio example 302, but rather processes a different subset of the audio example (specifically, a so-called proper subset according to mathematical definitions). For example, each of the discriminators can randomly sample a subset of the audio example 302. That is, each discriminator only processes the magnitudes of a subsequence of consecutive output time steps of the audio example 302, where the subsequence of consecutive output time steps is randomly sampled from the entire sequence of output time steps. In some implementations, the size of the random sample, i.e., the number of output time steps that are sampled, is the same for each discriminator. In some other implementations, the size of the random sample is different for each discriminator and is referred to as the “window size” of the discriminator.

[0122] In some implementations, one or more of the discriminators in the discriminator neural network system 300 can have the same network architecture and the same parameter values. For example, the training system can update one or more of the discriminators in the same way during training of the discriminator neural network system.

[0123] In some implementations, different discriminators have different window sizes. For example, for each window size in a set of multiple window sizes, the system can include a conditional discriminator with the window size and an unconditional discriminator with the window size. As a specific example, the discriminator neural network 300 includes one conditional discriminator and one unconditional discriminator for each window size in a set of five window sizes (240, 480, 960, 1920, and 3600 output time steps).

[0124] Each conditional discriminator also obtains a sample of the conditional text input 304 that corresponds to the random sample of the audio example 302 obtained by the conditional discriminator. Because the sample of the audio example 302 and the sample of the conditional text input 304 must align, the conditional discriminator can be constrained to sample a subsequence of the audio output 302 that starts at the same point as the input time step at which the conditional text input 304 starts. That is, because the number of input time steps in the conditional text input 304 is less than the number of output time steps in the audio example 302, each conditional discriminator can be constrained to sample the audio example 302 at points that align with the input time steps of the conditional text input 304. Because the unconditional discriminators do not process the conditional text input 304, the unconditional discriminators do not have this constraint.

[0125] In some implementations, before processing a random sample of audio example 302, each discriminator first down-samples the random sample by a factor proportional to the window size of the discriminator using a "resample" layer. The down-sampling effectively allows discriminators having different window sizes to process audio example 302 at different frequencies, with the frequency at which a particular discriminator operates being proportional to the window size of the particular discriminator. Down-sampling by a factor proportional to the window size also allows each down-sampled representation to have a common dimension, which allows each of the discriminators to have a similar architecture and similar computational complexity despite having different window sizes. In Figure 3 In the depicted example, the common dimension is 240 time steps; each discriminator is labeled with a factor k by which the discriminator down-samples its random sample (1, 2, 4, 8, and 15, respectively).

[0126] In some implementations, the discriminators down-sample the respective random samples of audio example 302 using a strided convolutional neural network layer (i.e., a neural network layer configured to perform a strided convolution).

[0127] Unconditional discriminator 320 randomly samples a sample 312 including 480 output time steps from audio example 302. Unconditional discriminator 320 includes a resample layer 322 that down-samples random sample 312 by a factor of 2 to generate a network input including 240 time steps.

[0128] Unconditional discriminator 320 includes a sequence of discriminator blocks ("DBlocks") 324. Although unconditional discriminator 320 is depicted as including five discriminator blocks, in general an unconditional discriminator can have any number of discriminator blocks. Each discriminator block in unconditional discriminator 320 is an unconditional discriminator block. An example unconditional discriminator block is described below with respect to Figure 4 Description.

[0129] The first discriminator block in sequence of discriminator blocks 324 is configured to receive a network input from resample layer 322 and to process the network input to generate a block output. Each subsequent discriminator block in sequence of discriminator blocks 324 is configured to receive as input a block output generated by a preceding discriminator block in sequence 324 and to generate a subsequent block output.

[0130] In some implementations, one or more discriminator blocks in unconditional discriminator 320 perform further down-sampling. For example, as Figure 3As depicted, the second discriminator block in the sequence 324 of discriminator blocks down-samples by a factor of 5, and the third discriminator block in the sequence 324 down-samples by a factor of 3. In general, down-sampling helps the unconditional discriminator 320 limit the dimensionality of the internal representation, improve efficiency, and allow the unconditional discriminator 320 to learn relationships between more distant elements of the samples 312 of the audio example 302. In some implementations, the number of down-sampling layers in a respective unconditional discriminator and the degree of down-sampling performed by a respective unconditional discriminator depends on the factor k of the respective unconditional discriminator.

[0131] The unconditional discriminator 320 includes an output layer 326 that receives the block output of the final discriminator block in the sequence 324 of discriminator blocks as input and generates a prediction 328. The prediction 328 predicts whether the audio example is a real or a synthetic audio example. For example, the output layer 326 can be an average pooling layer that generates a scalar value that characterizes the likelihood that the audio example 302 is a real audio example, where a greater scalar value indicates a higher confidence that the audio example 302 is real.

[0132] The unconditional discriminator 340 randomly samples the samples 332 that include 3600 output time steps from the audio example 302. The unconditional discriminator 340 includes a reshaping layer 342 that down-samples the random samples 332 by a factor of 15 to generate a network input that includes 240 time steps.

[0133] The unconditional discriminator 340 includes a sequence 344 of discriminator blocks. Although the conditional discriminator 340 is depicted as including five unconditional discriminator blocks and one conditional discriminator block, in general a conditional discriminator can have any number of conditional discriminator blocks and unconditional discriminator blocks.

[0134] The first discriminator block in the sequence 344 of discriminator blocks is configured to receive the network input from the reshaping layer 342 and process the network input to generate a block output. Each subsequent discriminator block in the sequence 344 of discriminator blocks is configured to receive as input the block output generated by the preceding discriminator block in the sequence 344 and generate a subsequent block output.

[0135] The conditional discriminator block is configured to receive as input i) the block output of the preceding discriminator block in the sequence 344 and ii) a sample of the conditional text input 304 that corresponds to the sample 332 of the audio example. An example conditional discriminator block is described below with respect to Figure 4

[0136] ​Due to the difference in frequency of the output time steps and the input time steps, one or more unconditional discriminator blocks preceding a conditional discriminator block in the sequence of discriminator blocks 344 and / or the conditional discriminator block itself can perform downsampling. For example, the total degree of downsampling across all discriminator blocks in the conditional discriminator 340 can be proportional to the ratio of the frequency of the input time steps and the output time steps. In Figure 3 In the depicted example, the conditional discriminator 340 downsamples the network input by a factor of 8 to reach a frequency of 200 Hz, which is the same frequency as the conditional text input.

[0137] The conditional discriminator 340 includes an output layer 346 that receives the block output of the final discriminator block in the sequence of discriminator blocks 344 as input and generates a prediction 348 that the audio example is a real or a synthetic audio example. For example, the output layer 346 can be an average pooling layer that generates a scalar that characterizes the likelihood that the audio example 302 is a real audio example.

[0138] After each discriminator in the discriminator neural network system 300 generates a prediction, the discriminator neural network system 300 can combine the respective predictions to generate a final prediction 306 that the audio example is a real or a synthetic audio example. For example, the discriminator neural network system 300 can determine a sum or average of the respective predictions of the discriminators. As another example, the discriminator neural network system 300 can generate the final prediction 306 according to a voting algorithm, e.g., predicting that the audio example 302 is a real audio example if and only if a majority of the discriminators predict that the audio example 302 is a real audio example.

[0139] Note that the discriminator neural network system 300 does not receive as input a class associated with the audio input. However, in some implementations, the discriminator neural network system 300 can do so, e.g., as a further conditional input to the conditional discriminator blocks of each conditional discriminator.

[0140] Figure 4 are diagrams of an example unconditional discriminator block 400 and an example conditional discriminator block 450. The discriminator blocks 400 and 450 can be components of a discriminator neural network system, e.g., the discriminator neural network system 120 depicted in FIG. 1. The discriminator neural network system can include one or more discriminators, and each discriminator can include a sequence of one or more discriminator blocks. In some implementations, each discriminator block in the sequence of discriminator blocks in a discriminator has the same architecture. In some other implementations, one or more discriminator blocks have a different architecture than other discriminator blocks in the sequence of discriminator blocks in a discriminator.

[0141] The unconditional discriminator block 400 can be a component of one or more unconditional discriminators, one or more conditional discriminators, or both. The conditional discriminator block 450 can be a component of one or more conditional discriminators only. That is, a conditional discriminator can include an unconditional discriminator block, but an unconditional discriminator cannot include a conditional discriminator block.

[0142] The unconditional discriminator block 400 is configured to receive a block input 402 to generate a block output 404. In some implementations, if the unconditional discriminator block 400 is the first discriminator block in a sequence of discriminator blocks of a discriminator, the block input 402 is an audio example. In some other implementations, if the unconditional discriminator block 400 is the first discriminator block in a sequence of discriminator blocks of a discriminator, the block input 402 is an embedding of an audio example, such as an embedding generated by an input convolutional neural network layer of the discriminator. If the unconditional discriminator block 400 is not the first discriminator block in the sequence, the block input 402 can be the block output of a previous discriminator block in the sequence.

[0143] The unconditional discriminator block 400 includes a first stack of neural network layers 412 that includes a down-sampling layer 412a, an activation layer 412b, and a convolutional neural network layer 412c.

[0144] For unconditional discriminator blocks that down-sample the block input 402, such as the second discriminator block and the third discriminator block in the sequence of discriminator blocks 324 depicted in FIG. 3, the down-sampling layer 412a down-samples the block input 402 by a corresponding down-sampling factor. Although Figure 3 The unconditional discriminator block 400 depicted in FIG. 4 has a single down-sampling layer 412a, but in general an unconditional discriminator block can have any number of down-sampling layers throughout the unconditional discriminator block. Figure 4 The unconditional discriminator block 400 depicted in FIG. 4 has a single down-sampling layer 412a, but in general an unconditional discriminator block can have any number of down-sampling layers throughout the unconditional discriminator block.

[0145] In some implementations, the activation layer 412b can be a ReLU activation layer.

[0146] The convolutional neural network layer 412c processes a layer input that includes N channels per element and generates a layer output that includes m-N channels per element. N corresponds to the number of channels in the block input 202, and m corresponds to a multiplier. In some cases, m = 1. In general, any one or more convolutional layers of an unconditional discriminator block can change the number of channels of a layer input.

[0147] The unconditional discriminator block 400 includes a second stack of neural network layers 414 that processes the output of the first stack of neural network layers 412. The second stack of neural network layers 414 includes an activation layer 414a and a convolutional neural network layer 414b.

[0148] The unconditional discriminator block 400 includes a skip connection 426 that combines the input of the first stack of neural network layers 412 with the output of the second stack of neural network layers 414. For example, the skip connection 426 can add or concatenate the input of the first stack of neural network layers 412 and the output of the second stack of neural network layers 414.

[0149] In cases where the first stack of neural network layers 412 or the second stack of neural network layers 414 includes a convolutional neural network layer that changes the number of channels of the layer input (e.g., the convolutional neural network layer 412c), the unconditional discriminator block can include another convolutional neural network layer 422 before the skip connection 426, such that the two inputs of the skip connection 426 have the same number of channels.

[0150] In cases where the first stack of neural network layers 412 or the second stack of neural network layers 414 includes a down-sampling layer (e.g., the down-sampling layer 412a), the unconditional discriminator block can include another down-sampling layer 424 before the skip connection 426, such that the two inputs of the skip connection 426 have the same dimensions.

[0151] In some implementations, one or more of the convolutional neural network layers in the unconditional discriminator block 400 are dilated convolution layers.

[0152] The conditional discriminator block 450 is configured to receive a block input 452 and a conditional text input 454 corresponding to an audio example and generate a block output 456.

[0153] The conditional discriminator block 450 includes a first stack of neural network layers 462 that includes a down-sampling layer 462a, an activation layer 462b, and a convolutional neural network layer 462c.

[0154] For a conditional discriminator block that down-samples the block input 452, the down-sampling layer 462a down-samples the block input 452 by a down-sampling factor of the stack. Although Figure 4 The conditional discriminator block 450 depicted in FIG. 4 has a single down-sampling layer 462a, but in general a conditional discriminator block can have any number of down-sampling layers throughout the conditional discriminator block. In particular, the conditional discriminator block 450 down-samples the layer input 452 so that it has the same dimensionality in the time dimension as the conditional text input 454.

[0155] In some implementations, the activation layer 462b can be a ReLU activation layer.

[0156] The convolutional neural network layer 462c processes the layer input that includes N channels per element and generates a layer output that includes m-N channels per element; in this example m = 2. In general, any of the one or more convolutional layers of a conditional discriminator block can change the number of channels of the layer input.

[0157] The conditional discriminator block 450 includes a convolutional neural network layer 472 that processes the conditional text input 454 and generates a layer output that includes m-N channels per element. That is, the convolutional neural network layer 472 changes the number of channels of the conditional text input (567 in this case) to match the output of the convolutional neural network layer 462c in the first stack of neural network layers 462.

[0158] The conditional discriminator block 450 includes a combination layer 474 that combines the outputs of the convolutional neural network layers 462c and 472, e.g., by adding or by concatenating.

[0159] The conditional discriminator block 450 includes a second stack of neural network layers 482 that processes the output of the combination layer 474. The second stack of neural network layers 482 includes an activation layer 482a and a convolutional neural network layer 482b.

[0160] The conditional discriminator block 450 includes a skip connection 496 that combines the input of the first stack of neural network layers 462 with the output of the second stack of neural network layers 482. For example, the skip connection 496 can add or concatenate the input of the first stack of neural network layers 462 and the output of the second stack of neural network layers 482.

[0161] In cases where the first or second stack of neural network layers 462, 482 includes a convolutional neural network layer that changes the number of channels of the layer input. For example, the convolutional neural network layer 462c, the conditional discriminator block 450 can include another convolutional neural network layer 492 before the skip connection 496, such that the two inputs of the skip connection 496 have the same number of channels.

[0162] In cases where the first or second stack of neural network layers 462, 482 includes a down-sampling layer (e.g., the down-sampling layer 462a), the conditional discriminator block 450 can include another down-sampling layer 494 before the skip connection 496, such that the two inputs of the skip connection 496 have the same dimensions.

[0163] In some implementations, one or more of the convolutional neural network layers in the conditional discriminator block 450 are dilated convolutional layers.

[0164] In addition to or instead of Figure 3 and 4 The discriminator system can also include one or more spectrogram discriminators, e.g., also a convolutional neural network as described above, that receives a spectrogram as input and predicts whether the spectrogram is a real or a synthetic audio example.

[0165] Figure 5An implementation of the dynamic time warping method is shown that assists in aligning the generated spectrogram with the ground truth spectrogram to compute a spectrogram prediction loss.

[0166] The generated output tokens do not have a fixed length, their length can vary from one token to the next. The prediction loss is computed by an iterative process that finds a minimum cost alignment path 508 between the spectrogram of the generated output tokens and the ground truth audio.

[0167] Each step k in the path 508 includes a pair of time points, including a first time point representing a time point in the ground truth spectrogram and a second time point representing a time point in the predicted spectrogram. In Figure 5 The example shown has 10 points distributed along the horizontal axis for the generated audio and 10 points distributed along the vertical axis for the ground truth audio. The 12 pairs of time points in the path are represented by the shaded dots in Figure 5 The spectrogram alignment starts with the first step on each sequence, e.g., {A gt，1 , A ga，1}. At each step k, the system can determine to either advance both spectrograms by one time point or advance one of the spectrograms without advancing the other spectrogram. That is, at each step k e [1, K p - 1], the system determines the next element of the alignment path according to one of the following actions:

[0168] Item 502, p gen，k+1 = p gen，k + 1, p gt，k+1 = p gt，k + 1,

[0169] Item 504, p gen，k+1 = p gen，k , p gt，k+1 = p gt，k + 1, or

[0170] Item 506, p gen，k+1 = p gen，k + 1, p gt，k+1 = p gt，k .

[0171] This results in a path p = <(p p , p gen，1 ),..., (p gt，1 , K gen , p p , K gt ) of K p elements, where each element (·, ·) includes a pair of matching time points, p gen indexes the element of the sub-path in the time dimension of the predicted spectrogram, and pgt The elements of the sub-paths are indexed in the time dimension of the true-value spectrogram. One or more such paths can be used to determine the spectrogram prediction loss.

[0172] In some implementations, the system can greedily determine path 508, that is, at each step k∈[1, K] p At [-1], select the action that minimizes the step size cost. The step size cost can be S. gen [p gen，k ] and S gt [p gt，k The distance between ] and if S gt and S gen The combination of unadvanced twisted punishment w.

[0173] In some other implementations, the system can globally determine path 508, that is, for each possible path, determine the total path cost by combining the step cost of each step of the possible paths, and select the possible path that minimizes the total path cost. For example, the total path cost of the possible alignment paths could be...

[0174]

[0175] Where δ k It is aimed at S gt and S gen An indicator with a step size of 1 that has not been advanced; F is the number of frequency regions in the spectrum; and K p It is the number of elements in a possible aligned path, T≤K p ≤2T-1, where T is the number of time points in the spectrum.

[0176] In some of these implementations, the spectrogram prediction loss is

[0177]

[0178] Where P is the set of possible alignment paths. In some other implementations of this kind, the spectrogram prediction loss is:

[0179]

[0180] Where P is the set of possible alignment paths, and τ is the temperature parameter.

[0181] Figure 6 This is a flowchart of an example process 600 for training a generative neural network. For convenience, process 600 will be described as being executed by a system of one or more computers located in one or more locations. For example, a training system appropriately programmed according to this specification (e.g., Figure 1B The training system 110B described in the text can execute process 600.

[0182] The system obtains a training conditioned text input (step 602).

[0183] The system processes a training generation input that includes the training conditioned text input using the feed-forward generation neural network in accordance with the current values of the generation parameters to generate a training audio output (step 604). Specifically, the system processes the training generation input using the alignment neural network to generate an aligned conditioned sequence and using the generator neural network to generate the training audio output, the aligned conditioned sequence including a respective feature representation at each of a plurality of first time steps.

[0184] The system processes the training audio output using each of the one or more discriminators, where each discriminator predicts whether the training audio output is a real audio example or a synthetic audio example (step 606).

[0185] The system determines a final prediction using respective predictions of the one or more discriminators (step 608).

[0186] The system determines an update to the current values of the generation parameters to increase a first error of the final prediction (step 610). The system can also determine other updates based on other objectives, such as a spectrogram loss, a length loss, or both, as described above.

[0187] Figure 7 is a flowchart of an example process 700 for generating an audio output. For convenience, the process 700 will be described as being performed by a system of one or more computers located in one or more locations. For example, an inference system (e.g., the inference system 110A depicted in FIG. 1), appropriately programmed, can perform the process 700. Figure 1A An inference system 110A depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform process 700.

[0188] The system obtains a conditioned text input (step 702).

[0189] The system processes a generation input that includes the conditioned input using the alignment neural network to generate an aligned conditioned sequence that includes a respective feature representation at each of a plurality of first time steps (step 704). Specifically, the conditioned sequence is referred to as an “aligned” sequence because each time step in the aligned sequence corresponds to a fixed length of time in the audio output that will be generated by the generator neural network.

[0190] The system processes the aligned conditioned sequence using the generator neural network to generate an audio output (step 706).

[0191] The specification uses the term“configured” in connection with systems and computer program components. A system of one or more computers is configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation, causes or cause the system to perform the operations or actions. One or more computer programs are configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0192] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structural equivalents of such software, firmware, or hardware, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0193] The term“data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0194] A computer program, which can also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and

[0195] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0196] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0197] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0198] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone, that is running a messaging application, and receiving responsive messages from the user as a result.

[0199] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common, compute-intensive parts of machine learning training or production (i.e., inference, workloads).

[0200] Machine learning models can be implemented and deployed using a machine learning framework, such as a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.

[0201] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0202] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving data from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received from the device at the server.

[0203] While this specification contains many specifics, these should not be construed as limitations on the scope of any invention or on the scope of what can be claimed, but rather as descriptions of features that can be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also combine to provide certain embodiments. Conversely, various features that are described in the context of a single embodiment can also be implemented separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0204] Similarly, while operations are depicted in the drawings and described as being performed in a particular order, this should not be understood as requiring such order nor that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated in a single software product or packaged into multiple software products.

[0205] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the acts recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order or sequential arrangement of steps presented therein, nor that all illustrated steps be performed, to achieve desirable results. In some instances, multitasking and parallel processing can be advantageous.

Claims

1. A computer-implemented method for training a feedforward generative neural network, the feedforward generative neural network having multiple generation parameters and configured to generate audio output using conditional text input. in, Each conditional text input includes a corresponding linguistic feature representation at each of multiple input time steps. The feedforward generative neural network is configured to receive a generative input including the conditional text input and process the generative input to generate an audio output, the audio output including a corresponding audio sample at each of a plurality of output time steps, and The training includes, for each of the training condition text inputs: Obtain the training condition text input; Based on the current value of the generation parameters, the feedforward generative neural network processes the training generation input, including the training conditional text input, to generate a training audio output. The processing includes: The training-generated input is processed using an aligned neural network to generate an alignment-conditional sequence comprising corresponding feature representations at each of a plurality of first time steps, the aligned neural network including a first subnetwork and a second subnetwork; and The alignment condition sequence is processed using a generator neural network to generate the training audio output; The training audio output is processed using each of one or more discriminators, wherein each discriminator predicts whether the training audio output is a real audio example or a synthesized audio example. The final prediction is determined using the corresponding predictions of the one or more discriminators; and Determine an update to the current value of the generated parameters to increase the first error of the final prediction. The process of processing the training-generated input using the aligned neural network includes: The training-generated input is processed using the first subnet to generate an intermediate sequence with a corresponding intermediate element at each of the multiple intermediate time steps. The intermediate sequence is processed using the second subnet to generate a length prediction for each intermediate element, representing the predicted time length of that intermediate element; and The corresponding length prediction is processed to generate the alignment condition sequence.

2. The method according to claim 1, wherein: Each discriminator has multiple corresponding discrimination parameters. Each discriminator processes the training audio output based on the current value of the corresponding discrimination parameter, and The method further includes determining an update to the current value of the discrimination parameter to reduce the first error in the prediction.

3. The method according to claim 1, wherein: Each discriminator has multiple corresponding discrimination parameters; and The training also includes: Get real audio examples; The real audio example is processed using each of the one or more discriminators, wherein each discriminator predicts whether the real audio example is a real audio example or a synthesized audio example; A second final prediction is determined using the corresponding predictions of the one or more discriminators; and An update to the current value of the discrimination parameter is determined to reduce the second error of the second final prediction.

4. The method according to claim 1, wherein, The generator neural network comprises a sequence of groups of one or more convolutional neural network layers, wherein each group comprises one or more dilated convolutional layers.

5. The method according to claim 1, wherein, The one or more discriminators include one or more unconditional discriminators, wherein each unconditional discriminator processes a corresponding subset of the training audio outputs instead of the training conditional text inputs to predict whether the training audio outputs are real audio examples or synthetic audio examples.

6. The method according to claim 5, wherein, Each unconditional discriminator includes a discriminator neural network, which comprises a sequence of groups of one or more convolutional neural network layers, wherein each group comprises one or more dilated convolutional layers.

7. The method according to claim 5, wherein: Each corresponding subset of the training audio output is a proper subset of the training audio output, and At least two of the unconditional discriminators process different true subsets of the training audio output.

8. The method according to claim 7, wherein, Processing the corresponding proper subset of the training audio output includes, for each unconditional discriminator: Obtain random samples of the training audio output, wherein the random samples comprise multiple consecutive audio samples, and the size of the random samples given the discriminator is predetermined; and The random samples of the training audio output are processed.

9. The method according to claim 8, wherein, Processing the random samples of the training audio output includes: for each unconditional discriminator, downsampling the random samples of the training audio output to generate a downsampled representation, wherein each unconditional discriminator downsamples the random samples by a predetermined downsampling factor.

10. The method according to claim 9, wherein: Each unconditional discriminator's corresponding predetermined downsampling factor corresponds to the size of the random sample of that unconditional discriminator; and Each downsampling indicates that all unconditional discriminators in the unconditional discriminator have a common dimension.

11. The method according to claim 9, wherein, The random samples of the training audio output are downsampled by processing the random samples of the training audio output using strided convolutional neural network layers.

12. The method according to claim 5, wherein, Each unconditional discriminator also processes the embedding of the speaker identity in the training audio output.

13. The method according to claim 1, wherein, The generated input also includes an identifier of the category to which the audio output should belong.

14. The method according to claim 1, wherein, The alignment condition sequence has a frequency that is lower than the frequency of the training audio output.

15. The method according to claim 1, wherein, The first subnetwork includes multiple dilated convolutional neural network layers.

16. The method according to claim 1, wherein, The second subnet includes point-by-point multilayer perceptrons.

17. The method according to claim 1, wherein, Processing the corresponding length prediction to generate the alignment condition sequence includes: The cumulative length prediction is determined using the corresponding length predictions of the plurality of intermediate elements; and The cumulative length prediction is used to determine the number of first time steps, wherein the first time step has a predetermined frequency.

18. The method according to claim 1, wherein, Processing the corresponding length prediction to generate the alignment condition sequence includes: Non-uniform interpolation is performed on the intermediate sequence using the corresponding length prediction of the intermediate elements to generate the feature representation at each of the plurality of first time steps in the alignment condition sequence.

19. The method according to claim 18, wherein, Non-uniform interpolation of the intermediate sequence includes: Determine the position of each intermediate element in the prediction time; For each intermediate element and for each first time step, determine the corresponding weight value; and For each first time step, the feature representation in the alignment condition sequence is determined by combining the intermediate elements using the corresponding weight values ​​for that first time step and each intermediate element.

20. The method according to claim 19, wherein, Determining the position of each intermediate element in the prediction time includes: For each intermediate element, calculate the cumulative sum of the length predictions. ,as well as For each intermediate element, calculate the position in the predicted time. .

21. The method according to claim 19, wherein, Determine the intermediate element n and first time step t The corresponding weight values ​​include calculations: ， in, N It is the number of the intermediate elements. c n It is the first n The position of the intermediate element in the predicted time, and σ 2 This is the preset temperature parameter.

22. The method according to claim 1, wherein, Determining an update to the current value of the generated parameters includes determining the update based on the alignment length loss, which characterizes the error in the length of the training audio output.

23. The method according to claim 22, wherein, The alignment length loss is ， in, L It is the length of the audio output corresponding to the true value of the text input under the training conditions. N It is the number of intermediate elements, and l n It is the first n The predicted length of each intermediate element.

24. The method according to claim 1, wherein, Processing the training audio output using each of one or more discriminators includes processing the training audio output using a spectrogram discriminator, including: The training audio output is processed to generate a spectrogram of the training audio output; The spectrogram discriminator is used to process the spectrogram of the training audio output to predict whether the training audio output is a real audio example or a synthesized audio example.

25. The method according to claim 24, wherein, The spectrogram discriminator also processes the embedding of the speaker identity in the training audio output.

26. The method according to claim 24, wherein, The spectrogram discriminator includes one or more convolutional neural network layers.

27. The method according to claim 1, wherein, Determining an update to the current value of the generated parameters includes determining the update based on the spectrogram prediction loss, including: The training audio output is processed to generate a predicted spectrogram of the training audio output; Process the ground truth audio output corresponding to the training condition text input to generate a ground truth spectrogram of the ground truth audio output; and Determine the spectrogram prediction loss, which characterizes the difference between the predicted spectrogram and the true spectrogram.

28. The method according to claim 27, wherein, The spectrogram prediction loss is ， in, S gt This is the spectrum of the true value. S gen This is the predicted spectrum. T It is the number of time points in the spectrum, and F It is the number of frequency regions in the spectrum.

29. The method according to claim 27, wherein, The spectrogram prediction loss is determined using dynamic time warp, including determining the alignment path between the ground truth spectrogram and the predicted spectrogram. p ， in, K p The alignment path p The number of elements; p gen It is a sub-path in the time dimension of the predicted spectrogram; p gt It is a sub-path in the time dimension of the true value spectrum; and p gen,1 = p gt,1 =1, and Among them, the alignment path is determined. p Including at each step The next element of the alignment path is determined based on one of the following actions: Advance S gt and S gen Both of these make , Only advance S gt , making ,or Only advance S gen , making .

30. The method according to claim 29, wherein, Determine the alignment path p Including for each step size : For each action, determine the step cost, including: Sure and The distance between them; and if S gt and S gen If neither is advanced, then the distortion penalty is determined. w .

31. The method according to claim 30, wherein, Determine the alignment path p include: For each of a plurality of possible alignment paths, the total path cost is determined by combining the step cost of each step size in that possible alignment path; and Select the alignment path that minimizes the total path cost. p .

32. The method according to claim 31, wherein, The total path cost of the possible alignment path is ， in, It is aimed at S gt and S gen An indicator that neither of them is advanced by a step size of 1; F It is the number of frequency regions in the spectrum; and K p It is the number of elements in the possible alignment path. ,in, T It is the number of time points in the spectrum.

33. The method according to claim 32, wherein, The spectrogram prediction loss is ， in, P It is a set of possible alignment paths.

34. The method according to claim 32, wherein, The spectrogram prediction loss is ， in, P It is the set of possible alignment paths, and τ is the temperature parameter.

35. A computer-implemented method for generating audio output using a feedforward generative neural network trained using any one of claims 1 to 34.

36. A computer-implemented method for generating audio output using conditional text input, wherein, Each conditional text input includes a corresponding linguistic feature representation at each of multiple input time steps, and the method includes: Obtain conditional text input; and The feedforward generative neural network is used to process the generative input, including the conditional text input, to generate an audio output comprising corresponding audio samples at each of a plurality of output time steps, the processing comprising: The generated input is processed using an aligned neural network to generate an alignment condition sequence comprising corresponding feature representations at each of a plurality of first time steps, the aligned neural network including a first subnetwork and a second subnetwork; and The alignment condition sequence is processed using a generator neural network to generate the audio output. The process of generating the input using the aligned neural network includes: The first subnet is used to process the generated input to generate an intermediate sequence with a corresponding intermediate element at each of the multiple intermediate time steps. The intermediate sequence is processed using the second subnet to generate a length prediction for each intermediate element, representing the predicted time length of that intermediate element; and The corresponding length prediction is processed to generate the alignment condition sequence.

37. The method of claim 36, wherein, The generated input also includes an identifier of the category to which the audio output should belong.

38. The method according to claim 36, wherein, The alignment condition sequence has a frequency lower than the frequency of the audio output.

39. The method according to claim 36, wherein, The first subnetwork includes multiple dilated convolutional neural network layers.

40. The method of claim 36, wherein, The second subnet includes point-by-point multilayer perceptrons.

41. The method according to claim 36, wherein, Processing the corresponding length prediction to generate the alignment condition sequence includes: The cumulative length prediction is determined using the corresponding length predictions of the plurality of intermediate elements; and The cumulative length prediction is used to determine the number of first time steps, wherein the first time step has a predetermined frequency.

42. The method according to claim 36, wherein, Processing the corresponding length prediction to generate the alignment condition sequence includes: Non-uniform interpolation is performed on the intermediate sequence using the corresponding length prediction of the intermediate elements to generate the feature representation at each of the plurality of first time steps in the alignment condition sequence.

43. The method according to claim 42, wherein, Non-uniform interpolation of the intermediate sequence includes: Determine the position of each intermediate element in the prediction time; For each intermediate element and for each first time step, determine the corresponding weight value; and For each first time step, the feature representation in the alignment condition sequence is determined by combining the intermediate elements using the corresponding weight values ​​for that first time step and each intermediate element.

44. The method according to claim 43, wherein, Determining the position of each intermediate element in the prediction time includes: For each intermediate element, calculate the cumulative sum of the length predictions. ,as well as For each intermediate element, calculate the position in the predicted time. .

45. The method according to claim 43, wherein, Determine the intermediate element n and first time step t The corresponding weight values ​​include calculations: ， in, N It is the number of the intermediate elements. c n It is the first n The position of the intermediate element in the predicted time, and σ 2 This is the preset temperature parameter.

46. ​​A system comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to perform the operation of the method according to any one of claims 1 to 45.

47. One or more computer storage media storing instructions, which, when executed by one or more computers, cause the one or more computers to perform the operation of the method according to any one of claims 1 to 45.

Images