Training a speech synthesis neural network by using energy scores

Abstract

The present disclosure relates to training a speech synthesis neural network by using energy scores. Methods, systems and apparatus, including a computer program encoded on a computer storage medium, are used to train a generative neural network to convert conditional text input into audio output by using energy scores.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 035,553, filed on June 5, 2020. The disclosure of that prior application is considered part of the disclosure of the present application and is incorporated by reference into the disclosure of the present application. Technical Field

[0003] This application is related to training neural networks. Background Art

[0004] A neural network is a machine learning model in which one or more layers of nonlinear units are used to predict outputs for received inputs. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as input to one or more other layers in the network (i.e., one or more other hidden layers, the output layer, or both). Each layer of the network generates an output from the received input according to the current values ​​of the corresponding parameter sets. Summary of the invention

[0005] This specification describes a system implemented as a computer program on one or more computers in one or more locations that trains a neural network configured to generate output audio data.

[0006] In some embodiments, the output audio data is an output audio example comprising an audio wave sample at each of the sequence of output time steps. The audio sample at a given time step may be an amplitude value of the audio wave, or a compressed or companded amplitude value.

[0007] In some other embodiments, the output audio data is a spectrogram representing an audio example. In these embodiments, when inferring, the system can process the generated spectrogram to generate a corresponding audio example. For example, the system can linearly project the generated spectrogram into waveform space by applying an inverse spectrogram transform to the generated spectrogram.

[0008] The subject matter described in this specification can be implemented in specific embodiments to realize one or more of the following advantages.

[0009] Some existing speech synthesis systems rely on adversarial training, i.e., the training system trains the speech synthesis neural network and one or more discriminators in parallel. These generative adversarial networks can be difficult to train and require many training epochs to converge. Using energy scores to determine parameter updates, as described in this specification, allows the system to achieve stable training and fast convergence.

[0010] The feedforward generative neural network described in this specification can generate output examples faster than the prior art that relies on autoregressive generative neural networks (e.g., WaveNet), which is important for applications where fast batch inference is important. Autoregressive neural networks generate output examples across multiple output time steps by performing a forward pass at each output time step. At a given output time step, the autoregressive neural network generates new output samples to be included in the output examples conditioned on the output samples that have been generated. This consumes a lot of computing resources and takes a lot of time. On the other hand, the feedforward generative neural network can generate output examples in a single forward pass while maintaining the high quality of the generated output examples. This greatly reduces the time and amount of computing resources required to generate the output examples relative to how much time is consumed by the autoregressive neural network.

[0011] Other prior art techniques rely on reversible feedforward neural networks that are trained by distilling autoregressive models using probability densities (e.g., Parallel WaveNet). Training in this manner allows reversible feedforward neural networks to generate speech signals that sound realistic and correspond to input text without having to model every possible change that occurs in the data. Feedforward generative neural networks as described in this specification can also generate realistic audio samples that are faithful to the input text without explicitly modeling the data distribution of the audio data, but can do so more efficiently in terms of both model size and training time, and do not require the distillation and reversibility requirements of reversible feedforward neural networks.

[0012] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter of this specification will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a diagram of an exemplary training system for training a generative neural network.

[0014] Figure 2 is a diagram showing the training process.

[0015] Figure 3 is a flow chart of an exemplary process for training a generative neural network.

[0016] Like reference numbers and designations throughout the various drawings represent like elements. DETAILED DESCRIPTION

[0017] This specification describes a system that trains a generative neural network to generate output audio data using conditional text input. The system can train the generative neural network using energy scores, as described in more detail below.

[0018] Figure 1 is a diagram of an exemplary training system 100 for training a generative neural network 110. Training system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.

[0019] The training system 100 includes a generative neural network 110 , an energy score generation system 120 , and a parameter update system 130 .

[0020] The training system 100 is configured to train a generative neural network 110 to receive a conditional text input 102 and process the conditional text input 102 to generate an audio output 112. In some implementations, the generative neural network 110 is a feed-forward neural network, i.e., the generative neural network 110 generates the audio output 112 in a single forward pass.

[0021] The conditional text input 102 represents input text, and the audio output 112 depicts speech corresponding to the input text.

[0022] In some embodiments, the conditional text input 102 includes the input text itself, such as a character-level or word-level embedding of the input text. Alternatively or additionally, the conditional text input may include language features for characterizing the text input. For example, the conditional text input may include a corresponding language feature vector for each input time step in the sequence of input time steps. As a specific example, the language features for each input time step may include i) a phoneme and ii) a duration of the text at the input time step. The language features may also include pitch information; for example, the pitch may be determined by the logarithmic fundamental frequency logF of the input time step. 0 To express.

[0023] Generative neural network 110 also receives noise input 104 as input.

[0024] For example, the noise input 104 may be randomly sampled from a predetermined distribution (eg, a normal distribution). The noise input 104 may ensure variability in the audio output 112.

[0025] In some implementations, the generative neural network 110 may also receive as input an identification of a class 106 to which the audio output 112 should belong. The class 106 may be a member of a set of possible classes. For example, the class 106 may correspond to a particular speaker that the audio output 112 should sound like. That is, the audio output 112 may depict a particular speaker speaking the input text.

[0026] Generally, audio output 112 (also referred to as “output audio data”) represents the audio, ie, speech, in which the input text represented by conditional text input 102 is being spoken by a speaker.

[0027] In some embodiments, the output audio data 112 is an output audio example that includes an audio sample of an audio wave at each of a series of output time steps. The audio sample at a given output time step can be an amplitude value of the audio wave or a compressed or companded amplitude value.

[0028] In some other implementations, the output audio data 112 is data defining a spectrogram representing the audio example, ie, data specifying amplitude components of a short-time Fourier transform (STFT) applied to the audio example.

[0029] In these embodiments, when inferring, the system 100 can process the generated spectrogram to generate a corresponding audio example. For example, the system can generate an audio example by applying an inverse spectrogram transform to the generated spectrogram and linearly projecting the generated spectrogram into a waveform space.

[0030] Generative neural network 110 may have any suitable neural network architecture.

[0031] As a specific example, the generative neural network 110 may include a sequence of convolutional neural network layer groups (referred to as "generator blocks"). The first generator block in the sequence of generator blocks may receive as input a conditional text input (or an embedding of a conditional text input) and generate a block output. Each subsequent generator block in the sequence of generator blocks may receive as input a block output generated by a previous generator block in the sequence of generator blocks and generate a subsequent block output.

[0032] In the following, reference will be made to Figure 2 An exemplary architecture of generative neural network 110 is described in more detail.

[0033] To train the generative neural network 110, the system 100 repeatedly updates the values ​​of the parameters of the neural network 110 by using mini-batches of training examples sampled from a set of training data.

[0034] Each mini-batch includes one or more training examples, each of which includes a training condition text input 102 and a ground-truth audio output 108 .

[0035] As will be apparent from the description below, the true audio output 108 may be (i) an audio example comprising a sample sequence of audio waves and representing input text for characterizing the training conditional text input 102 being spoken by a speaker, or (ii) a spectrogram of the audio example representing input text for characterizing the training conditional text input 102 being spoken by a speaker.

[0036] During training, the system 100 processes the conditional text input 102 in each training example multiple times using the neural network 110 , each time generating a corresponding predicted audio output 112 .

[0037] In particular, the system 100 samples multiple noise inputs 104 for each training example, for example, z~N(0,I d ), where I is an identity matrix of size d. In some embodiments, the system 100 samples the noise input by using latent variable truncation. That is, the system 100 samples the initial noise input, and for each element in the noise input whose absolute value exceeds a certain threshold, resamples the element to ensure that the magnitude of each element is below the certain threshold, thereby effectively truncating the distribution to have a maximum absolute value equal to the certain threshold.

[0038] For each noise input 104, the system then processes the conditional text input 102 and the noise input 104 using a neural network 110 to generate a corresponding audio output 112. Because the noise inputs 104 will typically be different because they are different samples from the same distribution, the corresponding audio outputs 112 will typically be different even if the same conditional text input 102 is used to generate the audio output.

[0039] The energy score generation system 120 then generates an estimated energy score 122 for each training example in the mini-batch by using the plurality of audio outputs 112 for the training examples and the true value outputs 108 in the training examples, and provides the estimated energy score 122 to the parameter update system 130 .

[0040] In the following, reference will be made to Figure 2 and 3 Generating an estimated energy score is described in more detail.

[0041] The parameter update system 130 can obtain the estimated energy scores 122 generated by the energy score generation system 120, and determine the parameter update 132 according to a loss function that characterizes the combined energy scores across the training examples in the mini-batch (e.g., as the sum or average of the estimated energy scores 122). In particular, the system 130 can calculate the gradient of the parameters of the neural network 110 with respect to the loss function, and then apply an appropriate optimizer to the gradient, such as an Adam optimizer, an rmsProp optimizer, an Adafactor optimizer, an SGD optimizer, etc., to determine the parameter update 132.

[0042] In some implementations, the loss function also includes one or more other terms generated by using other components of the system 110. For example, the loss function may also include one or more adversarial loss terms in which one or more discriminator neural networks are used that attempt to predict whether the audio output 112 is a true audio output from training data or an output that has been generated by the neural network 110.

[0043] The training system 100 may apply parameter updates 132 to the parameters of the generative neural network 110 by, for example, adding or subtracting the updates 132 from the current values ​​of the parameters.

[0044] By repeatedly updating the parameters in this manner, the training system 100 trains the neural network 110, i.e., determines training values ​​for the parameters of the neural network 110, so that the neural network 110 generates output examples (or generates a histogram that can be used to generate output examples) that sound realistic and accurately synthesize speech from the input text.

[0045] Figure 2 is a diagram illustrating the training of the neural network 110 .

[0046] In particular, Figure 2 The neural network 110 (“model f”) is shown as being trained on training examples including training condition text input c 102 and true value audio output x 108. θ ").exist Figure 2 In the example of , the true-valued audio output x directly defines the waveform 210, ie, is a sequence of samples of the audio wave.

[0047] In some embodiments, the system samples training examples uniformly and randomly from the training data. In some other embodiments, the system samples training examples from the training data set with a probability proportional to the length of the training examples. In some embodiments, the system processes only a portion of the sampled complete training examples; that is, the system determines the training examples as fixed-size subsequences of a larger conditional text input and corresponding subsequences of the ground-truth audio output for the larger conditional text input.

[0048] exist Figure 2 In the example of FIG. 2 , the system samples two noise inputs, namely: a first noise input z 220 and a second noise input z′ 230 .

[0049] The system then processes the first trained generative input including the training conditional text input c and the first noise input z by using the neural network 110 according to the current value of the parameter to generate a first predicted audio output y 240, and processes the second trained generative input including the training conditional text input c and the second noise input z' by using the neural network 110 according to the current value of the parameter to generate a second predicted audio output y'250.

[0050] In some embodiments, for example, when the audio output is an audio example, the neural network 110 includes multiple one-dimensional convolutional neural network layers. For example, the neural network can include one or more "G blocks", each of which includes multiple convolutional neural network layers with increasing dilation rates, as described in "High Fidelity Speech Synthesis with Adversarial Networks", Bińkowski et al., (arXiv:1909.11646). In some embodiments, each G block includes one or more nearest neighbor upsampling layers and one or more conditional batch normalization layers, for example, conditional batch normalization layers conditioned on sampled noise embeddings.

[0051] In some embodiments, the neural network includes one or more filter response normalization (FRN) layers, each of which converts a sequence having a time dimension of t and c channels As input, and by calculating the following formula, the output sequence

[0052]

[0053] in, and is the learned vector, ∈ is a positive scalar, and is the threshold value.

[0054] In some embodiments, one or more specific FRN layers are conditioned on sampled noise inputs. For example, for each specific FRN layer, the parameters γ and β of the layer can be learned linear functions of the corresponding noise embeddings, i.e., functions jointly learned during neural network training. As a specific example, for each specific FRN layer, the noise input z~N(0,I d ) and calculate the following to determine the parameters γ and β:

[0055] γ=W γ z+(b γ +1)

[0056] β=W β z+b β

[0057] Among them, W γ , and b γ , is specific to that particular FRN layer.

[0058] In some other embodiments, for example, when the audio output specifies a spectrogram, the neural network 110 includes a one-dimensional convolution with a kernel size of 1, which is used to project the conditional text input to a corresponding projection vector for each small block, where each small block corresponds to a predetermined number of input time steps. The projection vector is then fed into a stack of bottleneck ResNet blocks. Each ResNet block consists of a convolution with a kernel size of 1 for 512 channels, 2 convolutions with a kernel size of 5 at 512 channels, and then a projection to a fixed number of channels. Between convolutions, the neural network 110 uses conditional batch normalization to condition on the noise vector and optional class input.

[0059] After the ResNet block, the neural network 110 projects the output down to 240 dimensions per small block. Of these dimensions, one dimension is used to scale the remaining 239 features exponentially. The remaining 239 features (after scaling) are then interpreted as non-redundant elements of an STFT with a window size of 240 and a frame step of 120, i.e., as the audio output for specifying the spectrogram. These 239 features can then be projected into waveform space to generate a waveform by using a linear inverse STFT transform.

[0060] In some implementations, the neural network generates each of the plurality of predicted audio outputs in parallel.

[0061] After the neural network generates a plurality of different predicted audio outputs, the system can determine an estimated energy score characterizing the distance between the true audio output x and the plurality of predicted audio outputs y and y'.

[0062] That is, the energy score characterizes the similarity between the target distribution p represented by the true audio output and the estimated distribution q of the generative model of the neural network. Energy scores and examples are discussed in more detail in "Strictly Proper Scoring Rules, Prediction, and Estimation", Gneiting et al. (DOI: 10.1198 / 016214506000001437).

[0063] To determine the estimated energy score, the system may, for a first predicted audio output among multiple predicted audio outputs, i) calculate a distance between the first predicted audio output and a true audio output based on a distance metric; and ii) calculate a distance between the first predicted audio output and a second predicted audio output among the multiple predicted audio outputs based on the distance metric.

[0064] An example of estimating the energy score is:

[0065] 2d(x i ,y i )-d(y i ,y′ i )

[0066] Where d is the distance metric, x i is the true value audio output, y i is the first predicted audio output, and y i ' is the second predicted audio output.

[0067] Because the estimated energy score is larger when the distance between the true value output and the first predicted audio output is larger, the term used to measure this distance can be referred to as an "attractiveness" term 260 because minimizing the loss function based on the estimated energy score encourages a smaller distance between the true value and the first predicted audio output.

[0068] Because the estimated energy score is smaller when the distance between the second predicted audio output and the first predicted audio output is larger, the term used to measure this distance can be referred to as a "repulsive" term 270 because minimizing the loss function based on the estimated energy score encourages a larger distance between the first and second predicted outputs.

[0069] Including this exclusionary term in the loss encourages diverse samples and provides statistical consistency guarantees. In other words, the presence of the exclusionary term is necessary for the loss to be an appropriate scoring rule for learning the conditional distribution of the given linguistic features of the audio, and prevents the generated samples from collapsing to a single point without capturing the full distribution of possible samples that could correspond to the given input text.

[0070] In some embodiments, the system determines updates to the current values ​​of the parameters of the neural network by using a mini-batch of training examples by combining the corresponding energy scores of each training example. For example, the system can calculate the loss function:

[0071]

[0072] Where M is the number of training examples in a mini-batch.

[0073] In some embodiments, the loss function also includes one or more other terms generated by using other components of the system. For example, the loss function may also include one or more adversarial loss terms that use one or more discriminator neural networks that attempt to predict whether the audio output is a true audio output from training data or an output generated by the neural network 110.

[0074] Typically, the distance metric is negative definite, which makes the estimated energy score a "proper scoring rule", that is, when q = p, the estimated energy score will be minimized. For example, the distance metric can be taken as The form of . If α∈(0,2] and β∈(0,α], then the estimated energy score will be the appropriate scoring rule. Losses with appropriate scoring rules have been shown to lead to statistically consistent learning methods and can be computed from external mini-batches, resulting in improved training stability and higher quality resulting models.

[0075] In some embodiments, the distance function d is a spectrogram distance function that operates on spectrograms, ie, it measures the distance between two spectrograms.

[0076] In particular, in these embodiments, the distance function measures the distance between two audio outputs as the distance between spectrograms corresponding to the two audio outputs.

[0077] When an audio output specifies a spectrogram, the spectrogram corresponding to the audio output is the spectrogram specified by the audio output.

[0078] When the audio output is an audio example, the spectrogram corresponding to the audio output is a spectrogram generated by the system from the audio output (i.e., from the audio example). That is, the system may first generate a spectrogram corresponding to each predicted audio output and the true audio output, and calculate the distance function on the generated spectrogram.

[0079] That is, although Figure 2The example shows that the true output x and the predicted outputs y and y' are waveforms, which are then converted to corresponding spectrograms, but in some other examples, the true output, the predicted output, or both are spectrograms and therefore do need to be converted.

[0080] The system can determine the spectrogram of the audio example using any suitable technique. However, as a specific example, the system can determine the spectrogram using an overcomplete Fourier basis with a Hann window.

[0081] In particular, in some embodiments, the system may perform the above operation by, for each spectrogram window length k, i), clipping the audio output to a length to generate a cropped audio output; and ii) determining a spectrogram of the cropped audio output to determine a spectrogram of the audio output x of length N.

[0082] The system may use any of a variety of spectrogram distance metrics to calculate distances between audio outputs.

[0083] For example, a spectrogram distance metric can be a linear combination of one or more distance functions, each of which is of the form or Among them, x i is the first audio output, x j is the second audio output, and s(x) is the spectrogram of x. As described above, if α∈(0,2] and β∈(0,α], then the estimated energy score is an appropriate scoring rule.

[0084] As a specific example, the spectrogram distance metric may be:

[0085]

[0086] Where k is the specific window length of the spectrogram, K is the set of window lengths of the spectrogram, t is the time point of the spectrogram, is the value of the spectrogram of the audio output x with window length k at time point t, and α k are the learned or predetermined weight values.

[0087] As another specific example, the spectrogram distance metric may be:

[0088]

[0089] Where k is the specific window length of the spectrogram, K is the set of window lengths of the spectrogram, t is the time point of the spectrogram, is the value of the spectrogram of the audio output x with window length k at time point t, and α kare the learned or predetermined weight values.

[0090] The set of window lengths for the spectrogram K may be a sequence of geometrically spaced window lengths, such as a power-of-2 sequence. As a specific example, K=[2 6 ,…,2 11 ].

[0091] Figure 3 3 is a flow chart of an exemplary process 300 for training a generative neural network. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system (e.g., Figure 1 The training system 100 shown may perform process 300 .

[0092] The system obtains a training example, which includes a conditional text input and a true value audio output for the training example (step 302). The training conditional text input may include a corresponding language feature representation at each of a plurality of input time steps. For example, the language feature representation at each input time step may include a phoneme, duration, and logarithmic fundamental frequency at that time step.

[0093] The system processes a plurality of training generative inputs in a training example, each training generative input comprising a training conditional text input, using a generative neural network to generate a plurality of training audio outputs for the training example (step 304).

[0094] The system generates training audio outputs based on the current values ​​of the parameters of the generative neural network. Each training audio output can include a corresponding audio sample at each of a plurality of output time steps, or a spectrogram of an audio example can be specified.

[0095] In particular, each training generated input also includes a noise input. The system can sample a plurality of different noise inputs, and for each noise input, process the training generated input including the noise input and the training condition text input to generate a corresponding training output.

[0096] In some embodiments, generating the input further comprises an identification of the class to which the output wave should belong. In some such embodiments, one or more layers in the neural network are further conditioned on the identification of the class.

[0097] The system determines an estimated energy score for the training example using a true audio output in the training example and a plurality of training audio outputs for the training example (step 306). As described above, the estimated energy score characterizes a distance between the true audio output and the plurality of predicted audio outputs and is based at least on (i) a distance between a first predicted audio output of the plurality of outputs and the true output according to a distance metric, and (ii) a distance between the first predicted audio output and the second predicted audio output according to the distance metric.

[0098] The system determines an update to the current value of the parameter based on the estimated energy score (step 308). For example, the system can determine an estimated energy score for each training example in the mini-batch and then determine the update by calculating the gradient of a loss function that measures the combined energy score for the training examples (e.g., the sum or average of the energy scores for the training examples). The system can then apply an optimizer to the gradient to determine an update to the current value of the parameter.

[0099] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination of software, firmware, and hardware that, in operation, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operations or actions.

[0100] The embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, tangibly implemented computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-temporary storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0101] The term "data processing apparatus" refers to data processing hardware and includes all kinds of apparatus, devices and machines for processing data, including, by way of example, a programmable processor, a computer, or a plurality of processors or computers. The apparatus may also be or further include a dedicated logic circuit, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus may optionally include, in addition to the hardware, code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0102] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script or code, may be written in any form of programming language including compiled or interpreted languages ​​or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file for holding other programs or data, such as one or more scripts stored in a markup language document; in a single file dedicated to the program in question or in multiple coordinated files, such as files for storing one or more modules, subroutines or portions of code. A computer program may be deployed to execute on one computer, or on multiple computers located at one site or distributed on multiple sites and interconnected by a data communication network.

[0103] In this specification, the term "database" is used broadly to refer to any data set: the data need not be structured in any particular way, or at all, and it may be stored on a storage device in one or more locations. Thus, for example, an index database may include multiple data sets, each of which may be organized and accessed differently.

[0104] Similarly, the term "engine" is used broadly in this specification to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same computer or multiple computers.

[0105] The processes and logic flows described in this specification can be performed by one or more programmable computers for executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit such as an FPGA or ASIC, or by a combination of a dedicated logic circuit and one or more programmed computers.

[0106] The computer suitable for executing computer programs can be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or a central processing unit of any other kind. Usually, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential element of a computer is a central processing unit for executing or implementing instructions and one or more storage devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit or incorporated in the special logic circuit. Usually, the computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk or an optical disk, or be coupled in operation to receive data from the one or more large-capacity storage devices, or data is transferred to the one or more large-capacity storage devices, or both. However, the computer is not necessarily provided with such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game controller, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, etc.

[0107] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including by way of example: semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.

[0108] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to receiving a request from the web browser. In addition, a computer may interact with a user by sending a text message or other form of message to a personal device, such as a smart phone running a messaging application, and then receiving a response message from the user.

[0109] The data processing apparatus for implementing the machine learning model may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive parts of machine learning training or production (i.e., inference, workloads).

[0110] The machine learning model can be implemented and deployed by using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0111] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server; or includes a middleware component, such as an application server; or includes a front-end component, such as a client computer with a graphical user interface, a web browser, or an app that a user can use to interact with an implementation of the subject matter described in this specification; or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by digital data communication in any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0112] The computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data such as HTML pages to the user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input from the user. Data generated at the user device, for example, the result of the user interaction, can be received at the server from the user device.

[0113] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination. In addition, although features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may be removed from the combination in some cases, and a claimed combination may be directed to a sub-combination or a variation of a sub-combination.

[0114] Similarly, although operations are depicted in the drawings and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0115] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims may be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for training a neural network having a plurality of parameters and configured to generate an audio output for defining an audio example by using a conditional text input, in, The neural network is configured to receive a generative network input comprising a conditional text input and a noise input, and to process the generative network input to generate an audio output comprising a corresponding audio sample at each of a plurality of output time steps, and The training includes: Obtaining a training example, wherein the training example includes a training condition text input and a corresponding ground-truth audio output; generating a plurality of predicted audio outputs for the training examples by, for each of a plurality of noise inputs, processing a training generative input including the training conditional text input and the noise input using the neural network according to current values ​​of the plurality of parameters to generate a corresponding predicted audio output; Determining an estimated energy score characterizing a distance between the true audio output and the plurality of predicted audio outputs comprises: For a first predicted audio output of the plurality of predicted audio outputs, calculating a distance between the first predicted audio output and the true audio output according to a distance metric; and For the first predicted audio output and a second predicted audio output of the plurality of predicted audio outputs, calculating a distance between the first predicted audio output and the second predicted audio output according to the distance metric; and determining an update to current values ​​of the plurality of parameters based on the estimated energy scores, Wherein, calculating the distance between the first audio output and the second audio output comprises: calculating the distance between the spectrogram corresponding to the first audio output and the spectrogram corresponding to the second audio output according to a spectrogram distance metric.

2. The method of claim 1, wherein: The estimated energy score is equal to: 2d(x i ,y i )-d(y i ,y i ′), where d is the distance metric, x i is the true value audio output, y i is the first predicted audio output, and y i ' is the second predicted audio output.

3. The method of claim 2, wherein: Determining updates to current values ​​of the plurality of parameters based on the estimated energy scores includes determining the updates based on a loss function for characterizing combined energy scores across a plurality of training examples, wherein the loss function is: Where M is the number of training examples.

4. The method of claim 1, wherein: The distance metric is negative definite.

5. The method of claim 1, wherein: The spectrogram distance metric is a linear combination of one or more distance functions, each of which is of the form or Among them, x i is the first audio output, x j is the second audio output, and s(x) is the spectrogram of x.

6. The method of claim 5, wherein: α∈(0,2] and β∈(0,α].

7. The method of claim 6, wherein: The spectrum graph distance metric is: Where k is the specific window length of the spectrogram, K is the set of window lengths of the spectrogram, t is the time point of the spectrogram, is the value of the spectrogram of the audio output x with window length k at time point t, and α k are the learned or predetermined weight values.

8. The method of claim 6, wherein: The spectrum graph distance metric is: Where k is the specific window length of the spectrogram, K is the set of window lengths of the spectrogram, t is the time point of the spectrogram, is the value of the spectrogram of the audio output x with window length k at time point t, and α k are the learned or predetermined weight values.

9. The method of claim 7, wherein: K is a sequence of geometrically spaced window lengths.

10. The method of claim 1, wherein: Each of the first audio output and the second audio output comprises a respective plurality of samples of an audio wave, wherein calculating a distance between the first audio output and the second audio output comprises: A corresponding spectrogram is determined for each of the first and second audio outputs.

11. The method of claim 10, wherein: Determining a corresponding spectrogram for each of the first and second audio outputs includes determining the corresponding spectrogram using an overcomplete Fourier basis and a Hann window.

12. The method of claim 10, wherein: Determining the spectrogram of an audio output x of length N includes, for each spectrogram window length k: Crop the audio output x to length to generate a cropped audio output; and A spectrogram of the cropped audio output is determined.

13. The method of claim 1, wherein: Examples of training include: sampling a complete training example from a training dataset with a probability proportional to the length of the complete training example; and The training examples are determined as fixed-size portions of the sampled complete training examples.

14. The method of claim 1, wherein: The neural network includes multiple one-dimensional convolutional neural network layers.

15. The method of claim 1, wherein: The neural network includes one or more filter response normalization layers, wherein the one or more filter response normalization layers convert the sequence receives as input, and outputs the sequence in, and is the learned vector, ∈ is a positive scalar, is the threshold value.

16. The method of claim 15, wherein: For each noise input, one or more specific filter response normalization layers are conditioned on the noise input during processing of the training generative input including the noise input.

17. The method of claim 16, wherein: For each of the one or more specific filter response normalization layers, the parameters γ and β of the layer are learned linear functions of the noise input.

18. The method of claim 17, wherein: For each of the one or more specific filter response normalization layers, the parameters γ and β of the layer are determined by: For noise input z~N(0,I d ) is sampled, where I is an identity matrix of size d; and calculate: γ=W γ with+(b γ +1) β=W β with+with β in, and is specific to the layer in question.

19. The method according to any one of claims 1 to 18, wherein: Generating the plurality of predicted audio outputs includes generating each of the plurality of predicted audio outputs in parallel.

20. The method of any one of claims 1 to 18, wherein: Each conditional text input includes a corresponding linguistic feature representation at each of a plurality of input time steps.

21. The method according to any one of claims 1 to 18, wherein: Generating the plurality of predicted audio outputs includes sampling each of the plurality of noise inputs from a noise distribution.

22. The method of claim 21, wherein: Sampling each of the plurality of noise inputs comprises: Sampling the initial noise input; determining that an absolute value of one or more elements of the initial noise input is above a predetermined threshold; and In response, the determined one or more elements are resampled.

23. A method of generating output audio examples by using a neural network, the neural network having been trained by using the method of any one of claims 1-22.

24. A system comprising one or more computers and one or more storage devices storing instructions, wherein when the instructions are executed by the one or more computers, the one or more computers are caused to perform corresponding operations of the method of any one of claims 1-23.

25. One or more computer-readable storage media storing instructions, which, when executed by one or more computers, cause the one or more computers to perform corresponding operations of the method of any one of claims 1-23.

Images