Conditional output generation by data density gradient estimation

Abstract

The present invention relates to conditional output generation by data density gradient estimation. In one aspect, a method includes obtaining a network input, initializing a current network output, and generating a final network output by updating the current network output at each of a plurality of iterations, where each iteration corresponds to a respective noise level, and where updating includes, at each iteration: processing an iteration's model input using a noise estimation neural network configured to process a model input to generate a noise output, the model input comprising (i) the current network output and (ii) the network input, where the noise output comprises a respective noise estimate for each value in the current network output; and updating the current network output using the noise estimates and the iteration's noise level.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese Invention Patent Application No. 202180045795.6, filed on September 2, 2021.

[0003] Cross-reference to related applications

[0004] This application claims priority to U.S. Patent Application No. 63 / 073,867, filed September 2, 2020, the disclosure of which is incorporated herein by reference. Technical Field

[0005] This specification relates to using machine learning models to generate outputs conditioned on network inputs. Background Technology

[0006] Machine learning models receive input and generate outputs based on those inputs, such as predicted outputs. Some machine learning models are parametric models, generating outputs based on the received inputs and the values ​​of the model's parameters.

[0007] Some machine learning models are deep models that use multiple layers to generate outputs from received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each applying a nonlinear transformation to the received input to generate an output. Summary of the Invention

[0008] This specification describes a system of computer programs implemented on one or more computers at one or more locations that generates network output in response to network input.

[0009] According to a first aspect, a method is provided for generating a final network output including multiple outputs conditioned on a network input, the method comprising: obtaining a network input; initializing a current network output; and generating the final network output by updating the current network output at each of multiple iterations, wherein each iteration corresponds to a corresponding noise level, and wherein the update includes, at each iteration: processing the iterated model input using a noise estimation neural network configured to process model input to generate a noisy output, the model input including (i) the current network output and (ii) the network input, wherein the noisy output includes a corresponding noise estimate for each value in the current network output; and updating the current network output using the noise estimate and the iterated noise level.

[0010] In some implementations, the network input is a spectrogram of an audio segment, and the final network output is a waveform of the audio segment.

[0011] In some implementations, the audio segment is a speech segment.

[0012] In some implementations, spectrograms have been generated from text segments or the linguistic features of text segments using a text-to-speech model.

[0013] In some implementations, the spectrogram is a Mel spectrogram or a log-Mel spectrogram.

[0014] In some implementations, updating the current network output using noise estimation and iterative noise levels includes: generating an iterative update based at least on the noise estimation and the noise level corresponding to the iteration; and subtracting the update from the current network output to generate an initially updated network output.

[0015] In some implementations, updating the current network output further includes modifying the initially updated network output based on the iterative noise level to generate a modified initially updated network output.

[0016] In some implementations, for the final iteration, the modified initial updated network output is the updated network output after the final iteration, and for each iteration prior to the final iteration, the updated network output after the final iteration is generated by adding noise to the modified initial updated network output.

[0017] In some implementations, initializing the current network output includes sampling each of a plurality of initial values ​​of the current network output from a corresponding noise distribution.

[0018] In some implementations, the model input for each iteration includes iteration-specific data that differs for each iteration.

[0019] In some implementations, the model input for each iteration includes the noise level corresponding to that iteration.

[0020] In some implementations, the model input for each iteration includes the total noise level of the iteration, generated based on the noise level corresponding to the iteration and any subsequent iterations within a plurality of iterations.

[0021] In some embodiments, the noise estimation neural network includes: a noise generation neural network including a plurality of noise generation neural network layers and configured to process network inputs to map network inputs to noise outputs; and a network output processing neural network including a plurality of network output processing neural network layers and configured to process the current network output to generate an alternative representation of the current network output, wherein: at least one of the noise generation neural network layers receives an input derived from: (i) the output of another noise generation neural network layer, (ii) the output of a corresponding network output processing neural network layer, and (iii) iterative-specific data.

[0022] In some implementations, the final network output has a higher dimension than the network input, and the alternative representation has the same dimension as the network input.

[0023] In some implementations, the noise estimation neural network includes a corresponding Feature Linear Modulation (FiLM) module corresponding to each of at least one noise generation neural network layer, wherein the FiLM module corresponding to a given noise generation neural network layer is configured to process (i) the output of another noise generation neural network layer, (ii) the output of the corresponding network output processing neural network layer, and (iii) iteratively specific data for generating inputs to the noise generation neural network layer.

[0024] In some implementations, the FiLM module corresponding to a given noise-generating neural network layer is configured to: process the output of the neural network layer according to (ii) the corresponding network output and generate scaling and bias vectors based on iterative-specific data of (iii) the iteration; and generate input to the given noise-generating neural network layer by applying an affine transformation to the output of another noise-generating neural network layer in (i) the noise-generating neural network layer.

[0025] In some implementations, at least one of the noise-generating neural network layers includes an activation function layer that applies a non-linear activation function to the input of the activation function layer.

[0026] In some implementations, another noise-generating neural network layer in the noise-generating neural network layer corresponding to the activation function layer is a residual connection layer or a convolutional layer.

[0027] In some implementations, a method for training a noise estimation neural network includes repeatedly performing the following operations: obtaining training network inputs and corresponding training network outputs; selecting iteration-specific data from a set of all iteration-specific data including multiple iterations; sampling a noise output including a corresponding noise value for each value in the training network output; generating a modified training network output based on the noise output and the corresponding training network output; processing a model input using the noise estimation neural network to generate a training noise output, the model input including (i) the modified training network output, (ii) the training network input, and (iii) the iteration-specific data; and determining an update to the network parameters of the noise estimation neural network based on the gradient of an objective function that measures the error between the sampled noise output and the training noise output.

[0028] In some implementations, the objective function measures the distance between the sampled noise output and the training noise output.

[0029] In some implementations, the distance is the L1 distance.

[0030] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages.

[0031] The described technique generates network outputs conditioned on network inputs in a non-autoregressive manner. In general, autoregressive models have been shown to generate high-quality network outputs, but require a large number of iterations, leading to high latency and resource consumption (e.g., memory and processing power). This is because autoregressive models generate each given output in the network output sequentially, where each output is conditioned on all previous outputs within the network output.

[0032] On the other hand, the described technique starts with an initial network output (e.g., a noisy output including values ​​sampled from a noise distribution) and iteratively refines the network output via a gradient-based sampler conditioned on the network input; that is, an iterative denoising process can be used. Therefore, the method is non-autoregressive and requires only a constant number of generation steps during inference. For example, for spectrogram-conditioned audio synthesis, the described technique can generate high-fidelity audio samples in very few iterations (e.g., six or fewer), comparable to or even exceeding those generated by existing autoregressive models, with significantly reduced latency and while using far fewer computational resources. Furthermore, the described technique can generate higher quality (e.g., higher fidelity) samples compared to those generated by existing non-autoregressive models.

[0033] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0034] Figure 1 This is a block diagram of an example conditional output generation system.

[0035] Figure 2 This is a flowchart of an example process for generating output conditioned on network input.

[0036] Figure 3 This is a block diagram of an example noise estimation neural network.

[0037] Figure 4 This is a block diagram of the example network output processing neural network block.

[0038] Figure 5 This is a block diagram of an example Feature Linear Modulation (FiLM) module.

[0039] Figure 6 This is a block diagram of an example noise generation neural network block.

[0040] Figure 7 This is a flowchart of an example process for training a neural network for noise estimation.

[0041] The same reference numerals and names in the various figures denote the same elements. Detailed Implementation

[0042] Figure 1 An example conditional output generation system 100 is shown. The conditional output generation system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components and techniques described below are implemented.

[0043] The conditional output generation system 100 generates a final network output 104 based on the network input 102.

[0044] The conditional output generation system 100 described in this paper is widely applicable and not limited to a single implementation. However, for illustrative purposes, a small number of example implementations are described below.

[0045] For example, the system can be configured to generate a waveform of audio conditioned on a spectrogram of the audio (e.g., a Mel spectrogram or a spectrogram in which frequencies are on different scales). As a specific example of this, the spectrogram can be a spectrogram of a speech segment, and the waveform can be the waveform of that speech segment. For example, the spectrogram can be the output of a text-to-speech machine learning model that converts the linguistic features of text into a spectrogram of the spoken text.

[0046] As another example, the system can be configured to perform image processing tasks on network input to generate network output. For example, the network input can be a class of an object that specifies the class of the image object to be generated (e.g., represented by a one-hot vector), and the network output can be the generated image of that object's class (e.g., represented by a set of luminance values ​​or RGB values ​​for each pixel in the image).

[0047] As another specific example, the task could be conditional image generation, and the network input could be a sequence of text, with the network output being an image that reflects the text. For example, the text sequence could include sentences or a sequence of adjectives describing a scene in the image.

[0048] In another specific example, the task could be image embedding generation, and the network input could be an image, and the network output could be a digital embedding of the input image that represents the image.

[0049] As another specific example, the task could be object detection, and the network input could be an image and the network output could identify the location of a specific type of object in the input image, for example, specifying the bounding box containing the object depiction in the input image.

[0050] As another specific example, the task could be image segmentation, and the network input could be an image and the network output could be a segmentation output that assigns each of the multiple pixels of the input image to a category from a set of categories, for example, assigning each pixel a corresponding score representing the probability that the pixel belongs to that category.

[0051] More generally, a task can be any task that outputs continuous data conditioned on network input.

[0052] To generate a final network output 104 conditioned on network input 102, the conditional output generation system 100 obtains network input 102 and initializes the current network output 114. For example, the system 100 can obtain the output from a corresponding noise distribution (e.g., Gaussian distribution, such as...). ,in I The current network output 114 is initialized by sampling each value in the current network output (which is the identity matrix) from the current network output (i.e., the first instance of the current network output 114 can be generated). That is, the initial current network output 114 includes the same number of values ​​as the final network output 104, but each value is sampled from the corresponding noise distribution.

[0053] System 100 then generates the final network output 104 by updating the current network output 114 at each of the multiple iterations. In other words, the final network output 104 is the current network output 114 after the last iteration of the multiple iterations.

[0054] In some cases, the number of iterations is fixed.

[0055] In other cases, system 100 or another system can adjust the number of iterations based on the latency requirement for generating the final network output. That is, system 100 can select the number of iterations such that the final network output 104 will be generated to meet the latency requirement.

[0056] In other cases, system 100 or another system can adjust the number of iterations based on the computational resource consumption requirements for generating the final network output 104; that is, the number of iterations can be selected such that the final network output will be generated to meet the requirements. For example, the requirement could be the maximum number of floating-point operations (FLOPS) to be performed as part of generating the final network output.

[0057] In each iteration, the system uses a noise estimation neural network 300 to process the model input for the iteration, which includes (i) the current network output 114, (ii) the network input 102, and optionally (iii) iteration-specific data for the iteration. The iteration-specific data is typically derived from noise levels 106 (e.g., where each noise level corresponds to a specific iteration). The system can use noise levels 106 as a scale for updating the current network output for each iteration. That is, each noise level in noise levels 106 can correspond to a specific iteration, and the corresponding noise level for the iteration can guide the scale for updating the current network output 114 under the iteration.

[0058] The noise estimation neural network 300 is a neural network with parameters (“network parameters”) and configured to process model inputs based on the current values ​​of the network parameters to generate a noise output 110, which includes a corresponding noise estimate for each value in the current network output 114. Details of the noise estimation neural network will be referenced below. Figure 3 Let's discuss this in more detail.

[0059] Generally, the noise estimate of a given value in the current network output is an estimate of the noise corresponding to the actual value added to the actual network output to generate the given value. In other words, the noise estimate defines how the actual value (if known) needs to be modified to generate the given value in the current network output, given the noise level corresponding to the current iteration. In other words, a given value can be generated by applying the noise estimate to the actual value based on the noise level of the current iteration.

[0060] This noise estimation can be interpreted as an estimation of the gradient of the data density, and therefore the generation process can be viewed as an iterative process of generating the network output through the data density estimation.

[0061] System 100 then uses update engine 112 to update the current network output 114 in the direction of noise estimation.

[0062] Specifically, update engine 112 uses the noise estimate and the corresponding noise level under iteration to update the current network output 114. That is, update engine 112 uses the corresponding noise estimate of the noisy output 110 and the corresponding noise level under iteration to update each value of the current network output 114, as shown in the reference... Figure 2 As discussed in more detail.

[0063] After the final iteration, the conditional output generation system 100 outputs an updated network output 114 as the final network output 104. For example, in an implementation where the final network output 104 represents an audio waveform, the system can play back the audio using a speaker or transmit the audio for playback, etc. In another example, in an implementation where the final network output 104 represents an image, the system can display the image on a user's display or transmit the image for display, etc. In some embodiments, the system 100 can save the final network output 104 to a data storage device or transmit the final network output 104 for storage.

[0064] Before system 100 uses the noise estimation neural network 300 to generate the final network output, system 100 or another system trains the noise estimation neural network 300 with training data. See below. Figure 7 Describe the training.

[0065] Figure 2 This is a flowchart of an example process 200 for generating output conditioned on network input. For convenience, process 200 will be described as being executed by a system of one or more computers located in one or more locations. For example, a conditional output generation system appropriately programmed according to this specification (e.g., Figure 1 The conditional output generation system 100) can execute process 200.

[0066] The system obtains network inputs (202) that condition the final network output. For example, for a network output that is an audio waveform, the network input can be a spectrogram, a Mel spectrogram, or the linguistic features of the subject of the text reflected by the audio waveform.

[0067] The system initializes the current network output (204). For a final network output that includes multiple values, the system can sample each value from the initial current network output, which has the same number of values ​​as the final network output from the noise distribution. For example, the system can use a... The noise distribution (e.g., Gaussian noise distribution) is used to initialize the current network output, where I It is the identity matrix and In N This indicates the expected number of iterations. The system can start from iterations... N To Iteration 1 N In each iteration, the initial current network output is updated in descending order.

[0068] The system then updates the current network output at each of multiple iterations. Generally, the current network output at each iteration can be interpreted as the final network output with added noise. That is, the current network output is a noisy version of the final network output. For example, for the initial current network output... ,in N Indicates the number of iterations. The system can improve its iterations by removing the estimate corresponding to the noise in the iterations. N The system updates the current network output at each step of step 1. In other words, the system refines the current network output at each iteration by determining an estimate of the noise and updating the current network output based on that estimate. The system can use descending order for the iterations until the final network output is reached. .

[0069] In each of the multiple iterations, the system generates an iterative noise output by processing the model input using a noise estimation neural network, the model input including (1) the current network output, (2) the network input, and optionally (3) iteration-specific data (206). The iteration-specific data is typically derived from the noise levels of the iterations, where each noise level corresponds to a particular iteration. The noise output may include a noise estimate for each value in the current network output. For example, the corresponding noise estimate for a particular value in the current network output may represent an estimate of the noise of the corresponding actual value that has been added to the actual network output to generate the particular value. That is, the noise estimate for the particular value would represent how the actual value (if known) would need to be modified to generate the particular value given the corresponding noise level.

[0070] In each of the multiple iterations, the system updates the current network output of the current iteration using the noise output of the current iteration and the noise level corresponding to the current iteration (208). The system can update each value in the current network output using the corresponding noise estimate in the noise output and the noise level of the current iteration. The system can generate an iterative update based on the noise estimate and the noise level of the iteration, and then subtract the update from the current network output to generate an initial updated network output. The system can then modify the initial updated network output based on the noise level of the iteration to generate a modified initial updated network output. , (1) in n Indicates the index of the iteration. Iteration n The current network output is below. This represents the modified initial update network output. x Indicates network input. Iteration n noise level, Iteration n The total noise level (e.g., generated based on the noise level under the current iteration and any iterations after the current iteration), and This represents a parameterized representation generated by a noise estimation neural network. Noise output. Noise level. and total noise level From the noise table The noise level is determined from (for example, a linear noise table that varies linearly from minimum to maximum, a Fibonacci-based table, or a custom table generated based on a data-driven or heuristic approach). and total noise level The following sampling can be performed from a uniform distribution: , (2) in n Index representing iteration As in equation (2) Sampling enables the system to generate updates based on different noise scales. Each iteration... n noise level and total noise level It can be predetermined by the system and obtained as part of the model input.

[0071] For the final iteration, the modified initial updated network output is the updated network output after the final iteration, and for each iteration before the last iteration, the updated network output after the last iteration is generated by adding noise to the modified initial updated network output. That is, if the iteration is not the final iteration (i.e., if...), The system will also update the modified initial update network output to, , (3) in n Indicates the index of the iteration. According to the noise table Or another method to determine (e.g., as a function of the noise table, or by using hyperparameter tuning through empirical experiments), and .include This is to enable the modeling of multimodal distributions.

[0072] The system determines whether a termination criterion (210) is met. For example, a termination criterion may include that a specific number of iterations have been performed (e.g., being determined to meet a minimum performance metric, maximum latency requirement, or maximum computational resource requirement, such as the maximum number of FLOPS). If the specific number of iterations has not been performed, the system can restart from step (206) and perform another update to the current network output.

[0073] If the system determines that the termination criteria have been met, the system outputs the final network output (212), which is the updated network output after the final iteration.

[0074] Process 200 can be used to generate network outputs conditioned on network inputs in a non-autoregressive manner. In general, autoregressive models have been shown to generate high-quality network outputs, but require a large number of iterations, leading to high latency and resource consumption (e.g., memory and processing power). This is because autoregressive models generate each given output in the network outputs one by one, where each output is conditioned on all outputs preceding the given output within the network outputs. Process 200, on the other hand, starts with an initial network output (e.g., a noisy output including values ​​sampled from a noise distribution) and iteratively refines the network output via a gradient-based sampler conditioned on the network inputs. Therefore, this method is non-autoregressive and requires only a constant number of generation steps during inference. For example, for spectrogram-conditioned audio synthesis, the described technique can generate high-fidelity audio samples in very few iterations (e.g., six or fewer) that are comparable to or even exceed those generated by prior art autoregressive models, with significantly reduced latency and while using far fewer computational resources.

[0075] Figure 3 An example architecture of the noise estimation network 300 is shown.

[0076] The example noise estimation network 300 includes various types of neural network layers and neural network blocks (e.g., each neural network block includes multiple neural network layers), including convolutional neural network layers, noise generation neural network blocks, feature linear modulation (FiLM) module neural network blocks, and network output processing neural network blocks.

[0077] The noise estimation network 300 processes the model input to generate a noise output 110, which includes (1) the current network output 114, (2) the network input 102, and (3) iteration-specific data including a total noise level 306 corresponding to the current iteration. The network output 114 has a higher dimension than the network input 102, and the noise output 110 has the same dimension as the current network output 114. For example, for the current network output representing an audio waveform at 24 kHz, the network input may include an 80 Hz Mel spectrogram signal corresponding to the audio waveform (e.g., predicted by another system during inference).

[0078] The noise estimation network 300 includes multiple network output processing blocks to process the current network output 114 to generate a corresponding alternative representation of the current network output 114.

[0079] The noise estimation network 300 also includes a network output processing block 400 to process the current network output 114 to generate an alternative representation of the current network output, wherein the alternative representation has a smaller dimension than the current network output.

[0080] The noise estimation network 300 also includes additional network output processing blocks (e.g., network output processing blocks 318, 316, 314, and 312) to process alternative representations generated by previous network output processing blocks to generate another alternative representation with a smaller dimension than the previous alternative representation (e.g., network 318 processes the alternative representation from block 400 to generate an alternative representation with a smaller dimension than the output of block 400, block 316 processes the alternative representation from block 318 to generate an alternative representation with a smaller dimension than the output of block 318, etc.). The alternative representation of the current network output generated from the final network output processing block (e.g., 312) has the same dimension as the network input 102.

[0081] For example, given a current network output including an audio waveform of 24 kHz and a network input including a Mel spectrogram of 80 Hz, the network output processing blocks can "downsample" (i.e., reduce the dimension) the dimensionality by factors of 2, 2, 3, 5, and 5 (e.g., via network output processing blocks 400, 318, 316, 314, and 312, respectively), until the alternative representation produced by the final layer 312 is 80 Hz (i.e., reduced by factors of up to 300 to match the Mel spectrogram). Reference Figure 4 The architecture of the example network output processing block is discussed in more detail.

[0082] Noise estimation block 300 includes multiple FiLM module neural network blocks to process iteration-specific data corresponding to the current iteration (e.g., total noise level 306) and alternative representations from the network output processing neural network block, thereby generating inputs for the noise generation neural network block. Each FiLM module processes the total noise level 306 and alternative representations from the corresponding network output processing block to generate inputs for the corresponding noise generation block (e.g., FiLM module 500 processes alternative representations from network output processing block 400 to generate inputs for noise generation block 600, FiLM module 328 processes alternative representations from network output processing block 318 to generate inputs for noise generation block 338, etc.). Specifically, each FiLM module generates scaling and bias vectors as inputs to the corresponding noise generation block (e.g., as inputs to the affine transformation neural network layer within the corresponding noise generation block), as referenced. Figure 5 As discussed in more detail.

[0083] The noise estimation network 300 includes multiple noise generation neural network blocks to process network input 102 and output from the FiLM module to generate a noise output 110. The noise estimation network 300 may include convolutional layers 302 to process network input 102 to generate input to the first noise generation block 332; and convolutional layers 304 to process output from the final noise generation block 600 to generate the noise output 110. Each noise generation block generates an output with a higher dimension than network input 102. Specifically, each noise generation block after the first noise generation block generates an output with a higher dimension than the output from the previous noise generation block. The final noise generation block generates an output with the same dimension as the current network output 114.

[0084] The noise estimation network 300 includes a noise generation block 332 to process the output from convolutional layer 302 (i.e., the convolutional layer that processes network input 102) and the output from FiLM module 332 to generate input to noise generation block 334. The noise estimation network 300 also includes noise generation blocks 336, 338, and 600. Noise generation blocks 334, 336, 338, and 600 each process the output from their respective previous noise generation blocks (e.g., block 334 processes the output from block 332, block 336 processes the output from block 334, etc.) and the output from their respective FiLM modules (e.g., noise generation block 334 processes the output from FiLM module 324, noise generation block 336 processes the output from FiLM module 326, etc.) to generate input for the next neural network block. Noise generation block 600 generates input for convolutional layer 304, which processes the input to generate noise output 110. (Reference) Figure 6 The architecture of the example noise generation block (e.g., noise generation block 600) is discussed in more detail.

[0085] Each noise generation block before the last noise generation block can generate an output with the same dimensions as the alternative representation corresponding to the current network output (e.g., noise generation block 332 generates an output with the same dimensions as the alternative representation generated by network output processing block 314, noise generation block 334 generates an output with the same dimensions as the output from network output processing block 316, etc.).

[0086] For example, given a current network output including an audio waveform of 24 kHz and a network input including a Mel spectrogram of 80 Hz, the noise generation block can "upsample" (i.e., increase the dimension) by factors of 5, 5, 3, 2, and 2 (e.g., through noise generation blocks 332, 334, 336, 338, and 600, respectively) until the output of the final noise generation block (e.g., noise generation block 600) is 24 kHz (i.e., increased by factors of up to 300 to match the current network output 114).

[0087] Figure 4 An example architecture of the network output processing block 400 is shown.

[0088] Network output processing block 400 processes the current network output 114 to generate an alternative representation 402 of the current network output 114. The alternative representation has a smaller dimension than the current network output. Network output processing block 400 includes one or more neural network layers. The one or more neural network layers may include various types of neural network layers, including downsampling layers (e.g., for “downsampling” or reducing the dimension of the input), activation layers with non-linear activation functions (e.g., fully connected layers with a leaky ReLU activation function), convolutional layers, and residual connection layers.

[0089] For example, a downsampling layer can be a convolutional layer with the necessary stride to reduce (“downsample”) the dimension of the input. In a specific example, a stride X can be used to reduce the dimension of the input by a factor of X (e.g., a stride 2 can be used to reduce the dimension of the input by a factor of 2; a stride 5 can be used to reduce the dimension of the input by a factor of 5, etc.).

[0090] The left branch of the residual connection layer 420 includes a convolutional layer 402 and a downsampling layer 404. The convolutional layer 402 processes the current network output 114 to generate the input to the downsampling layer 404. The downsampling layer 404 processes the output from the convolutional layer 402 to generate the input to the residual connection layer 420. The output of the downsampling layer 404 has a reduced dimensionality compared to the current network output 114. For example, the convolutional layer 402 may include a filter with a stride of size 1×1 (i.e., to preserve dimensionality), and the downsampling layer 404 may include a filter with a stride of size 2×1 to downsample the dimension of the input by a factor of 2.

[0091] The right branch of the residual connection layer 420 includes a downsampling layer 406 and three subsequent blocks of activation layers followed by convolutional layers (e.g., activation layer 408, convolutional layer 410, activation layer 412, convolutional layer 414, activation layer 416, and convolutional layer 418). The downsampling layer 406 processes the current network output 114 to generate inputs for the subsequent three blocks of activation and convolutional layers. The output of the downsampling layer 406 has a smaller dimension compared to the current network input 114. The subsequent three blocks process the output from the downsampling layer 406 to generate inputs to the residual connection layer 420. For example, the downsampling layer 406 may include a 2×1 filter with a stride of 2 to reduce the dimension of the input by a factor of 2 (e.g., to properly match the downsampling layer 404). The activation layers (e.g., 408, 412, and 416) may be fully connected layers with a leaky ReLU activation function. Convolutional layers (e.g., 410, 414, and 418) may include filters of size 3×1 with a stride of 1 (i.e., for preserving dimensions).

[0092] The residual connection layer 420 combines the outputs from the left branch and the outputs from the right branch to generate an alternative representation 402. For example, the residual connection layer 420 can add the outputs from the left branch and the outputs from the right branch (e.g., element-wise) to generate the alternative representation 402.

[0093] Figure 5 An example of a Feature Linear Modulation (FiLM) module 500 is shown.

[0094] The FiLM module 500 processes the alternative representation 402 of the current network output and the total noise level 306 corresponding to the current iteration to generate a scaling vector 512 and a bias vector 516. The scaling vector 512 and bias vector 516 can be processed into corresponding noise generation blocks (e.g., ...). Figure 3 The noise estimation network 300 takes the input of a specific layer (e.g., an affine transformation layer) in the noise generation block 600. The FiLM module 500 includes a position encoding function and one or more neural network layers. The one or more neural network layers may include various types of neural network layers, including residual connection layers, convolutional layers, and activation layers with nonlinear activation functions (e.g., fully connected layers with a leaky ReLU activation function).

[0095] The left branch of the residual connection layer 508 includes a position encoding function 502. Position encoding function 502 processes the total noise level 306 to generate a positional code for the noise level. For example, the total noise level 306 can be multiplied by position encoding function 502, which is a combination of a sine function for even-numbered dimension indices and a cosine function for odd-numbered dimension indices, as in the preprocessing of the transformer model.

[0096] The right branch of the residual connection layer 508 includes a convolutional layer 504 and an activation layer 506. The convolutional layer 504 processes the alternative representation 402 to generate the input to the activation layer 506. The activation layer 506 processes the output from the convolutional layer 504 to generate the input to the residual connection layer 508. For example, the convolutional layer 504 may include a 3×1 filter with a stride of 1 (for dimensionality preservation), and the activation layer 506 may be a fully connected layer with a leaky ReLU activation function.

[0097] The residual connection layer 508 can combine the outputs from the left branch (e.g., the output from the positional encoding function 502) and the outputs from the right branch (e.g., the output from the activation layer 506) to generate inputs to both convolutional layers 510 and 514. For example, the residual connection layer 508 can add the outputs from the left branch and the outputs from the right branch (e.g., element-wise addition) to generate inputs to both convolutional layers (e.g., 510 and 514).

[0098] Convolutional layer 510 processes the output from residual connection layer 508 to generate scaled vector 512. For example, convolutional layer 510 may include a filter of size 3×1 with stride 1 (for preserving dimension).

[0099] Convolutional layer 514 processes the output from residual connection layer 508 to generate bias vector 516. For example, convolutional layer 514 may include a filter of size 3×1 with stride 1 (for preserving dimension).

[0100] Figure 6 An example noise generation network 600 is shown. The noise generation network 600 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components and techniques described below are implemented.

[0101] Noise generation block 600 is in the noise estimation neural network (e.g., Figure 3 The example neural network architecture used in the noise generation block of the noise estimation network (300).

[0102] The noise generation block 600 processes the input 602 and the output from the FiLM module 500 to generate the output 310. The input 602 can be a network input processed by one or more previous neural network layers (e.g., from...). Figure 3 The noise generation blocks 338, 336, 334, 332 and convolutional layer 302). Output 310 can be the input to a subsequent convolutional layer, which processes output 310 to generate noise output 110 (e.g., Figure 3 (Convolutional layer 304). Noise generation block 600 includes one or more neural network layers. The one or more neural network layers may include various types of neural network layers, including activation layers with non-linear activation functions (e.g., fully connected layers with leaky ReLU activation functions), upsampling layers (e.g., those that "upsample" or increase the dimension of the input), convolutional layers, affine transformation layers, and residual connection layers.

[0103] For example, an upsampling layer can be a neural network layer that "upsamples" (i.e., increases) the dimension of the input. That is, an upsampling layer generates an output with a higher dimension than the input to that layer. In specific examples, an upsampling layer can generate an output with X copies of each value in the input, thereby increasing the dimension of the output by a factor of X compared to the input (e.g., for an input (2, 7, -4), generating an output with two copies of each value would be (2, 2, 7, 7, -4, -4), or generating an output with five copies of each value would be (2, 2, 2, 2, 2, 7, 7, 7, 7, -4, -4, -4, -4, -4, -4), etc.). In general, an upsampling layer can fill each additional point in the output with the closest value from the input.

[0104] The left branch of the residual connection layer 618 includes an upsampling layer 602 and a convolutional layer 604. The upsampling layer 602 processes the input 602 to generate the input to the convolutional layer 604. The input to the convolutional layer has a higher dimension than the input 602. The convolutional layer 604 processes the output from the upsampling layer 602 to generate the input to the residual connection layer 618. For example, the upsampling layer can increase the dimension of the input by a factor of up to 2 by generating an output with two copies of each value in the input 602. The convolutional layer 604 may include a filter with a dimension of 3×1 and a stride of 1 (e.g., for preserving dimension).

[0105] The right branch of the residual connection layer 618 includes, in sequence, an activation layer 606 (e.g., a fully connected layer with a leaky ReLU activation function), an upsampling layer 608, a convolutional layer 610 (e.g., with a 3×1 filter size and a stride of 1), an affine transformation layer 612, an activation layer 614 (e.g., a fully connected layer with a leaky ReLU activation function), and a convolutional layer 616 (e.g., with a 3×1 filter size and a stride of 1).

[0106] Activation layer 606 processes input 602 to generate input to upsampling layer 608. The upsampling layer increases the dimension of the output from activation layer 606 to generate an input to convolutional layer 610 with a higher dimension than input 602 (e.g., by a factor of 2 to match upsampling layer 602). Convolutional layer 610 processes the output from upsampling layer 608 to generate input to affine transformation layer 612 (e.g., using a 3×1 dimension filter with a stride of 1 to preserve dimension). Activation layer 614 and convolutional layer 616 also process the output from affine transformation layer 612 to generate input to residual connection layer 618 (e.g., using a leaky ReLU function for network 614 and a 3×1 dimension filter with a stride of 1 for network 616).

[0107] For example, an affine transformation function can process the output from a preceding neural network layer (e.g., convolutional layer 610 in noise generation block 600) and the output from the FiLM module to generate an output. For example, the FiLM module can generate a scaling vector and a bias vector. The affine transformation layer can add the bias vector to the result of scaling the output from the previous neural network layer using the scaling vector from the FiLM module (e.g., using Hadamard product or element-wise multiplication).

[0108] The affine transformation layer 612 can process the output from the convolutional layer 610 and the output from the FiLM module 500 to generate the input to the activation layer 614. For example, this can be achieved by adding a bias vector from the FiLM module 500 to the result of scaling the output from the convolutional layer 610 using a scaling vector from the FiLM module 500.

[0109] The residual connection layer 618 combines the output from the left branch (e.g., the output from convolutional layer 604) and the output from the right branch (e.g., the output from convolutional layer 616) to generate an output. For example, the residual connection layer 618 can sum the output from the left branch and the output from the right branch to generate an output.

[0110] The left branch of residual connection layer 632 includes the output from residual connection layer 618. The left branch can be interpreted as the identity function of the output from residual connection layer 618.

[0111] The right branch of residual connection layer 632 comprises two consecutive blocks of an affine transformation layer, an activation layer, and a convolutional layer, respectively, to process the output from residual connection layer 618 and generate the input to residual connection layer 632. Specifically, the first block contains an affine transformation layer 620, an activation layer 622, and a convolutional layer 624. The second block contains an affine transformation layer 626, an activation layer 628, and a convolutional layer 630.

[0112] For example, for each block, the corresponding affine transformation layer can process the output from FiLM module 500 and the output from the corresponding previous neural network layer (e.g., affine transformation layer 620 can process the output from residual connection layer 618, and affine transformation layer 626 can process the output from convolutional layer 624) to generate the corresponding output. Each affine transformation layer can generate the corresponding output by scaling the output from the previous neural network layer using a scaling vector from FiLM module 500 and summing the scaled result with a bias vector from FiLM module 500. Each activation layer (e.g., 620 and 628) can be a corresponding fully connected layer with a leaky ReLU activation function. Each convolutional layer can include a corresponding 3×1 dimension filter with a stride of 1 (e.g., for dimensionality preservation).

[0113] Residual connection layer 632 combines the output from the left branch (e.g., the identity function from the output of residual connection layer 618) and the output from the right branch (e.g., the output from convolutional layer 630) to generate output 310. For example, residual connection layer 632 can sum the output from the left branch and the output from the right branch to generate output 310. Output 310 can be a convolutional layer (e.g., ... Figure 3 The input to the convolutional layer 304 will generate a noise output 110.

[0114] The noise generation block 600 may include multiple channels. Figure 3 Each noise generation block (e.g., 600, 338, 336, 334, and 332) can include a corresponding number of channels. For example, noise generation blocks 600, 338, 336, 334, and 332 can include 128, 128, 256, 512, and 512 channels, respectively.

[0115] Figure 7 This is a flowchart of an example process for training a noise estimation neural network. For convenience, process 700 will be described as being performed by a system of one or more computers located in one or more locations.

[0116] The system can execute process 700 in each of multiple training iterations to repeatedly update the values ​​of the parameters of the noise estimation neural network.

[0117] The system obtains a batch of training network input-training network output pairs (702). For example, the system can randomly sample training pairs from a data storage device. For example, each training network output can be an audio waveform, and each network input can be a true Mel-spectrum calculated based on the corresponding audio waveform.

[0118] For each training pair in the batch, the system selects iteration-specific data from a set of iteration-specific data that includes all iterations (704). For example, the system may sample specific iterations from a discrete uniform distribution that includes integers 1 to the final iteration, and then select iteration-specific data based on the specific iterations sampled from that distribution. The iteration-specific data may include the noise level, the total noise level (e.g., as determined in equation (2)), or the number of iterations itself. Thus, the system may condition the noise estimation neural network with a discrete index, or it may condition the noise estimation neural network with a continuous scalar indicating the noise level. Conditioning with a continuous scalar indicating the noise level may be advantageous because once the noise estimation neural network is trained, different numbers of refinement steps (i.e., iterations) can be used when generating the final network output at inference time.

[0119] For each training pair in the batch, the system samples the noise output, which includes the corresponding noise value (706) for each value in the training network output. For example, the system can sample the noise output from a noise distribution. In a specific example, the noise distribution could be a Gaussian noise distribution (such as N(0, ...). I ),in I It is an identity matrix with dimension n×n, where n is the number of values ​​in the output of the trained network.

[0120] For each training pair in the batch, the system generates a modified training network output (708) based on the noise output and the corresponding training network output. The system can combine the noise output and the corresponding training network output to generate the modified training network output. For example, the system can generate the modified training network output as... , (4) in This represents the modified output of the training network. This represents the corresponding output of the trained network. Indicates noise output, and This indicates iterating over specific data (e.g., totaling noise levels).

[0121] For each training pair in the batch, the system generates a training noise output (710) by processing the model input using a noise estimation neural network based on the current values ​​of the network parameters. This model input includes (1) the modified training network output, (2) the training network input, and (3) iterative specific data. The noise estimation neural network can process the model input to generate the training noise output, such as... Figure 2 As described in the process. For example, iterating over a specific criterion may include the total noise level. .

[0122] The system determines the update of the network parameters of the noise estimation network based on the gradient of the objective function used for the training batch (712). The system can determine the gradient of the objective function with respect to the neural network parameters of the noise estimation network for each training pair, and then update the current values ​​of the neural network parameters using the gradient (e.g., a linear combination of gradients, such as the average of gradients) using any of a variety of appropriate optimization methods (such as stochastic gradient descent with momentum or ADAM).

[0123] The objective function can measure the error between the noisy output and the training noisy output generated by the noise estimation network for each training pair. For example, for a particular training pair, the objective function can include a loss term that measures the L1 distance between the noisy output and the training noisy output. , (5) in Represents the loss function. Indicates noise output. This indicates that the neural network estimates parameters using noise. The generated training noise output, This represents the modified output of the training network. Indicates the input to the training network, and This indicates iterating over specific data (e.g., totaling noise levels).

[0124] The system can repeat steps (702) to (712) for multiple batches (e.g., multiple batches of training network input-training network output pairs).

[0125] This specification uses the term "configured to" in conjunction with system and computer program components. A system of one or more computers being configured to perform a specific operation or action means that software, firmware, hardware, or a combination thereof are installed on the system that, when operated, causes the system to perform those operations or actions. A computer program being configured to perform a specific operation or action means that one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0126] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-volatile storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, program instructions may be encoded on artificially generated propagated signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0127] The term "data processing device" refers to data processing hardware and encompasses various means, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or include special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0128] Computer programs (also referred to or described as programs, software, software applications, applets, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but does not need to, correspond to a file in a file system. A program may be stored as a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple harmonizing files (e.g., a file storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites interconnected by a data communication network.

[0129] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0130] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by a dedicated logic circuit system (such as an FPGA or ASIC) or by a combination of a dedicated logic circuit system and one or more programmable computers.

[0131] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented or incorporated into a dedicated logic circuit system. Generally, a computer will also include one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or be operatively coupled to receive data from or transfer data to one or more mass storage devices, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0132] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices, magnetic disks (such as internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0133] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Furthermore, the computer can interact with the user by transmitting and receiving documents from a device used by the user, for example, by transmitting a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and subsequently receiving response messages from the user.

[0134] The data processing apparatus for implementing machine learning models may also include, for example, dedicated hardware accelerator units for processing general and computationally intensive portions (i.e., inference, workloads) of machine learning training or production.

[0135] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0136] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes, for example, a back-end component acting as a data server, or includes middleware components (e.g., an application server), or includes front-end components (e.g., a client computer having a graphical user interface, web browser, or applet that a user can interact with through an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), such as the Internet.

[0137] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship arises from computer programs running on individual computers and having a client-server relationship with each other. In some embodiments, the server transmits data, such as HTML pages, to a user device, for example, to display data to a user interacting with a device acting as a client and to receive user input from that user. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.

[0138] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Moreover, although features may be described above as functioning in certain combinations and even thereby initially claimed, in some cases one or more features from a claimed combination may be removed from that combination, and the claimed combination may involve sub-combinations or variations thereof.

[0139] Similarly, although the operations are depicted in a specific order in the accompanying drawings and are set forth in a specific order in the claims, this should not be construed as requiring that these operations be performed in the specific order shown or in sequential order, or that all the operations shown be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0140] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions listed in the claims may be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require the specific order or sequential sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method executed by one or more computers, the method comprising: Obtain a phoneme sequence comprising the corresponding phonemes acquired at each of the multiple input time steps; The phoneme sequence is processed using an encoder neural network to generate a hidden representation of the phoneme sequence; Generate conditional inputs based on the hidden representation; The current waveform output is initialized by sampling each of a plurality of initial values ​​from the corresponding noise distribution of the current waveform output; as well as A final waveform output defining the speaker's phoneme sequence is generated by updating the current waveform output in each of a plurality of iterations, wherein each iteration corresponds to a corresponding noise level, and the final waveform output is an updated current output waveform following the last iteration of the plurality of iterations, wherein the update includes, in each iteration: The model input for the iteration is processed using a noise estimation neural network, the model input including (i) the current waveform output and (ii) the conditional input, the noise estimation neural network being configured to process the model input to generate a noise output, wherein the noise output includes a corresponding noise estimate for each value in the current waveform output; as well as The current waveform output is updated using the noise estimate and the noise level of the iteration.

2. The method according to claim 1, wherein, The noise estimation neural network and the encoder neural network have been trained end-to-end on training data including multiple training phoneme sequences and corresponding real waveforms for each training phoneme sequence.

3. The method according to claim 1, wherein, Updating the current waveform output using the noise estimate and the noise level of the iteration includes: An update to the iteration is generated based on at least the noise estimate and the noise level corresponding to the iteration; and The update is subtracted from the current waveform output to generate the initial updated output waveform.

4. The method according to claim 3, wherein, Updating the current waveform output further includes: The output waveform of the initial update is modified based on the noise level of the iteration to generate the modified output waveform of the initial update.

5. The method according to claim 4, wherein, For the final iteration, the modified initial update output waveform is the updated output waveform after the final iteration, and for each iteration before the final iteration, the updated output waveform after the final iteration is generated by adding noise to the modified initial update output waveform.

6. The method according to claim 1, wherein, The model input at each iteration includes iteration-specific data that is different for each iteration.

7. The method according to claim 6, wherein, The model input for each iteration includes the noise level corresponding to that iteration.

8. The method according to claim 6, wherein, The model input for each iteration includes the aggregated noise level of the iteration, generated based on the noise level corresponding to the iteration and any iteration following the iteration in the plurality of iterations.

9. The method according to claim 6, wherein, The noise estimation neural network includes: A noise generation neural network, comprising multiple noise generation neural network layers and configured to process the conditional input to map the conditional input to the noise output; and An output waveform processing neural network, comprising multiple output waveform processing neural network layers, is configured to process the current waveform output to generate an alternative representation of the current waveform output, wherein: At least one of the noise generation neural network layers receives an input derived from (i) the output of another noise generation neural network layer in the noise generation neural network layer, (ii) the output of a corresponding output waveform processing neural network layer, and (iii) iterative-specific data of the iteration.

10. The method according to claim 9, wherein, The final output waveform has a higher dimension than the conditional input, and the alternative representation has the same dimension as the conditional input.

11. The method according to claim 9, wherein, The noise estimation neural network includes a corresponding Feature Linear Modulation (FiLM) module corresponding to each of the at least one noise generation neural network layer, wherein the FiLM module corresponding to a given noise generation neural network layer is configured to process (i) the output of another noise generation neural network layer, (ii) the output of the corresponding output waveform processing neural network layer, and (iii) iterative-specific data of the iteration to generate the input to the noise generation neural network layer.

12. The method according to claim 11, wherein, The FiLM module corresponding to the given noise-generating neural network layer is configured as follows: Based on the corresponding output waveform of the neural network layer as described in (ii) and the iteration-specific data of the iteration described in (iii), a scaling vector and a bias vector are generated; and The input to the given noise generation neural network layer is generated by applying an affine transformation to the output of another noise generation neural network layer in (i).

13. The method according to claim 9, wherein, At least one of the noise-generating neural network layers includes an activation function layer that applies a non-linear activation function to the input to the activation function layer.

14. The method according to claim 13, wherein, The other noise generation neural network layer in the noise generation neural network layer corresponding to the activation function layer is a residual connection layer or a convolutional layer.

15. The method according to claim 1, wherein, The hidden representation includes a corresponding hidden vector for each of a plurality of hidden time steps, and wherein generating conditional inputs based on the hidden representation includes: The hidden representation is processed using a persistence predictor neural network to generate, for each hidden time step, a predicted persistence in the utterance represented by the hidden vector at said hidden time step; and The conditional input is generated by continuously upsampling the hidden representation based on the prediction to match the time scaling of the final waveform output.

16. The method according to claim 15, wherein, The conditional input includes a corresponding conditional vector for each of the multiple quantization time segments within the final waveform output.

17. The method of claim 15, further comprising: For each hidden time step, a predicted influence range in the utterance is generated by the hidden vector at the hidden time step, and wherein upsampling the hidden representation includes applying Gaussian upsampling to the hidden representation using the predicted duration and the predicted influence range of the hidden time step.

18. A system comprising one or more computers and one or more storage devices storing instructions, the instructions being operable, when executed by the one or more computers, to cause the one or more computers to perform the operation of a corresponding method according to any one of claims 1 to 17.

19. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operation of a corresponding method according to any one of claims 1 to 17.

Images