Text-to-speech using duration prediction

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for synthesizing audio data from text data using duration prediction. One of the methods includes processing, using a first neural network, an input text sequence comprising respective text elements at each of a plurality of input time steps to generate a modified input sequence comprising, for each input time step, a representation of a respective text element in the input text sequence; processing, using a second neural network, the modified input sequence to generate, for each input time step, a predicted duration for a corresponding text element in an output audio sequence; upsampling the modified input sequence according to the predicted durations to generate an intermediate sequence comprising, for each of a plurality of intermediate time steps, a respective intermediate element; and generating the output audio sequence using the intermediate sequence.

Description

Background Technology

[0001] This manual relates to using neural networks to perform text-to-speech.

[0002] Neural networks are machine learning models that use one or more non-linear units to predict outputs from received inputs. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer serves as input to the next layer in the network—the next hidden layer or output layer. Each layer of the network generates its output from the received inputs based on the current values ​​of its corresponding parameter set. Summary of the Invention

[0003] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which executes a neural network system configured to process an input text sequence representing a text sample and generate an output audio sequence representing audio data characterizing a speaker who uttered the text sample. This specification also describes a system implemented as a computer program on one or more computers at one or more locations that trains the neural network system. The neural network system can use a duration prediction neural network to generate the output audio sequence, the duration prediction neural network predicting the corresponding duration of a text element in the output audio sequence for each text element in the input text sequence.

[0004] The subject matter described in this specification can be implemented in particular embodiments to achieve one or more of the following advantages.

[0005] Some existing techniques use autoregressive neural networks to generate audio output from text input, where the neural network iteratively generates new output elements until the network determines to stop, for example, by generating a "stop" marker. Such autoregressive neural networks may be less robust than traditional non-deep methods because abruptly determining when to stop generating output markers can lead to problems such as premature interruption or failure to stop altogether. Using the techniques described in this specification, a neural network configured to perform text-to-speech can use a duration prediction neural network to predict the duration of each input marker in the output sequence, for example, the number of output markers corresponding to the input marker or the length of time represented by the input marker. Therefore, the neural network can determine the length of the audio output before generating it, thereby eliminating or significantly reducing the risk of premature interruption or failure to stop.

[0006] Furthermore, existing autoregressive neural networks impose little or no constraints to prevent the repetition or skipping of output tokens. As described in this specification, by directly modeling the duration of each input token, a deep neural network configured to perform text-to-speech can determine the precise correspondence between the input and output sequences, thereby minimizing the risk of repetition or skipping of output tokens.

[0007] Conventionally, non-autoregressive systems upsample the input sequence to generate intermediate sequences by simply repeating each input marker in the input sequence N times. Using the techniques described in this specification, the system can fully utilize the predicted duration of each input marker, and optionally the range parameter of the predicted importance of the input markers, to upsample the sequence in a more sophisticated way, thereby generating improved priors for the intermediate sequences. Specifically, in some embodiments described in this specification, the system can determine a distribution on the intermediate sequence for each input marker in the input sequence, such as a Gaussian distribution, which models the range of influence of the input markers. The system can then determine the value of each intermediate marker in the intermediate sequence based on a combination of the corresponding influences of the input markers, for example, a weighted sum of the intermediate marker values ​​in the corresponding distributions of the input markers.

[0008] Using the techniques described in this specification, a training system can train a duration prediction neural network as a component of a text-to-speech neural network in an unsupervised (or semi-supervised) manner. That is, the training system can train the duration prediction neural network concurrently with one or more other subnetworks of the text-to-speech neural network using training input sequences, for example, without having access to the true-value durations of each input token in the training input sequence. The true-value durations of each text element in a text corpus may be few; for example, in some cases, a user must manually tune a computationally expensive separate model to generate the true-value durations. By unsupervising the duration prediction neural network, the training system avoids the need to perform this labor-intensive process.

[0009] In some embodiments described in this specification, the system can control the pace of synthesized audio at each word or phoneme level by modifying the predicted duration of each word or phoneme determined by the duration prediction neural network, while still maintaining the naturalness of the synthesized speech.

[0010] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0011] Figure 1 This is a diagram of an example neural network system.

[0012] Figure 2 This is a graph of an example duration prediction neural network.

[0013] Figure 3 This is a diagram of an example training system.

[0014] Figure 4 This is a flowchart illustrating an example process for using a neural network system to process an input text sequence to generate an output audio sequence.

[0015] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0016] Figure 1 This is a diagram of an example neural network system 100. Neural network system 100 is an example of a system in which the systems, components, and techniques described below can be implemented as a computer program on one or more computers at one or more locations.

[0017] The neural network system 100 is configured to process an input text sequence 102 representing a text sample and generate an output audio sequence 142 representing audio data of the speaker who uttered the text sample. In other words, the output audio sequence 142 is "conditioned" on the input text sequence 102.

[0018] The input text sequence 102 may include corresponding text elements at each of multiple input time steps. For example, each text element may represent a character, word, or phoneme. As another example, each text element may include linguistic features derived from the input text and corresponding to the respective input time step. As a specific example, for each word or subword in the input text, the linguistic features may include: one or more morphological lemmas of the word or subword; one or more other morphological features of the word or subword, such as capitalization, digits, gender, person, and / or tense; part-of-speech tagging of the word or subword; dependency tags identifying different words that the word or subword depends on; or, an identifier of whether the subword occupies the beginning, interior, or end of the corresponding word. In some implementations, the input text sequence 102 includes corresponding text elements for word boundaries, punctuation, and / or sequence end markers.

[0019] The output audio sequence 142 may include samples of time-domain audio waveforms. That is, the output audio sequence 142 may include corresponding samples of the audio wave at each point in the output time-step sequence. For example, the audio sample at a given output time step may be the amplitude value, compressed amplitude value, or compressed-spread amplitude value of the audio wave. The output audio sequence can therefore represent synthesized speech corresponding to the input text sequence.

[0020] The neural network system 100 includes an encoder neural network 110, a duration prediction neural network 120, a decoder neural network 130, and a vocoder system 140.

[0021] The encoder neural network 110 is configured to process the input text sequence 102 and generate a modified input sequence 112, which includes representations of corresponding text elements in the input text sequence 102 at each of a plurality of input time steps. Each representation in the modified input sequence 112 may be an embedding of the corresponding text element in the input text sequence 102. In this specification, an embedding is an ordered set of numerical values ​​representing the input in a particular embedding space. For example, an embedding may be a floating-point vector or other numerical values ​​with a fixed dimension.

[0022] For each text element in the input text sequence 102, the encoder neural network 110 can obtain a predetermined embedding for that text element. For example, if the text element is an identifier of a phoneme, the encoder neural network 110 can obtain an embedding for each phoneme identified in the input text sequence 102. The predetermined embedding of the text element can be machine-generated. For example, the embedding can be trained concurrently with one or more neural networks in the neural network system 100. As another example, the embedding can be pre-trained using different neural networks to perform text processing machine learning tasks, such as language modeling machine learning tasks.

[0023] In some implementations, the encoder neural network 110 can directly compose a modified input sequence 112 from a predetermined embedding, wherein the representation in the modified input sequence 112 corresponding to each text element is equal to the embedding of the text element. That is, the encoder neural network 110 can be an encoder system that does not include any neural network layers.

[0024] In some other implementations, the encoder neural network 110 may use one or more neural network layers to process the embeddings of text elements in the input text sequence 102 to generate a modified representation of the text elements in the input sequence 112. For example, the encoder neural network 110 may use one or more convolutional neural network layers and / or one or more long short-term memory (LSTM) neural network layers—e.g., bidirectional LSTM neural network layers—to process the embedded sequence. As another example, the encoder neural network 110 may use one or more self-attention neural network layers to process the embedded sequence.

[0025] In some implementations, the encoder neural network 110 may use one or more neural network layers to process the embeddings of text elements to generate a corresponding initial representation for each text element. Then, for each initial representation corresponding to a given text element, the encoder neural network 110 may combine i) the initial representation and ii) an identifier of the class to which the output audio sequence 142 should belong to, to generate a representation of the text element in the modified input sequence 112. For example, the class may identify a specific speaker that the audio data represented by the output audio sequence 142 should sound like. The identified class may be one of a predetermined number of possible classes to which the output audio sequence 142 may belong.

[0026] The duration prediction neural network 120 is configured to process the modified input sequence 112 and generate a predicted duration for each representation in the modified input sequence, corresponding to a text element in the output audio sequence 142. That is, for each representation in the modified input sequence 112, the predicted duration of the representation represents the length of time the corresponding text element will take to be spoken in the audio data represented by the output audio sequence.

[0027] After the duration prediction neural network 120 generates a corresponding predicted duration for each representation in the modified input sequence 112, the duration prediction neural network 120 can upsample the modified input sequence 112 based on the predicted duration to generate an intermediate sequence 122, which includes corresponding intermediate elements at each of a plurality of intermediate time steps. Typically, the intermediate time steps present in the intermediate sequence 122 are more than the input time steps in the input text sequence 102.

[0028] For example, each prediction duration can be a floating-point value, representing the number of seconds or milliseconds the corresponding text element will take. As another example, each prediction duration can be an integer, representing the number of output time steps the corresponding text element will take.

[0029] In some implementations, the duration prediction neural network is configured to process the modified input sequence 112 and generate a corresponding single value representing the predicted duration for each representation in the modified input sequence 112. As a particular example, the duration prediction neural network 120 may use one or more bidirectional LSTM neural network layers to process the modified input sequence 112. The duration prediction neural network 112 may also include an output projection neural network layer, such as a feedforward neural network layer, configured to receive the corresponding layer input for each representation in the modified input sequence 112 and generate a predicted duration for each representation in the modified input sequence 112.

[0030] In some other embodiments, the duration prediction neural network 120 is configured to generate a corresponding distribution over the intermediate elements of the intermediate sequence 122 for each representation in the modified input sequence 112, the distribution modeling the impact on the representation of each intermediate element in the intermediate sequence 122. That is, for each representation in the modified input sequence 112, the corresponding distribution may include the value of each intermediate element in the intermediate sequence 122, representing the impact on the representation of the intermediate element. (Refer to below...) Figure 2 Describe the process in more detail.

[0031] In some implementations, an initial prediction duration is generated for each representation in the modified input sequence 112 (e.g., using one or more neural network layers, as referenced below). Figure 2 After that, the duration prediction neural network 120 can tentatively update one or more initial predicted durations to generate the final predicted duration.

[0032] For example, the duration prediction neural network 120 can scale each initial predicted duration using the same scaling factor (e.g., a scaling factor between 0.8 and 1.25). This is very useful when synthesizing audio data for speakers with different speech patterns—e.g., speakers who speak slower or faster than average.

[0033] As a specific example, the neural network system 100 may determine the scaling factor by comparing (i) the distribution of word / phoneme durations in audio data corresponding to a specific speaker to be represented in the output audio sequence 142 and (ii) the distribution of word / phoneme durations in audio data corresponding to a broader population. For example, the scaling factor may be determined to be proportional to the ratio of the means of the two distributions.

[0034] As another specific example, the duration prediction neural network 120 can generate (i) a predicted duration corresponding to a specific speaker (e.g., using an identifier of a class corresponding to the specific speaker, as described above) and (ii) a corresponding predicted duration corresponding to one or more other speakers (e.g., professional speakers) for each representation in the modified input sequence 112. The duration prediction neural network 120 can then combine the predicted durations corresponding to the respective speakers for each representation to generate a final predicted duration for the representation, for example, by determining an average of the predicted durations (e.g., a geometric mean). Performing this technique can improve the intelligibility of the output audio sequence 142. For example, a specific speaker may have a unique speech pattern that is difficult to understand. Therefore, the duration prediction neural network 120 can determine the final predicted duration based on both the specific speaker (such that the output audio sequence 142 represents the unique speech pattern of the specific speaker) and one or more professional speakers (such that the output audio sequence 142 is easier for the listener to understand).

[0035] As another example, the duration prediction neural network 120 can identify one or more specific representations in the modified input sequence 112 whose initial predicted durations should be modified. As a specific example, the duration prediction neural network 120 can identify one or more representations whose corresponding text elements in the input text sequence 102 represent words or phonemes that should be emphasized. For example, the duration prediction neural network 120 can obtain the identifiers of text elements from an external natural language machine learning model configured to process the input text sequence 102 (or a model input representing the same input text sequence 102) and identify one or more text elements that should be emphasized in the corresponding synthesized audio data.

[0036] After determining the predicted duration for each representation in the modified input sequence 112, the duration prediction neural network 120 can upsample the modified input sequence 112 based on the predicted duration to generate an intermediate sequence 122.

[0037] For example, in an implementation where the duration prediction neural network 120 generates a corresponding single value for the predicted duration of each representation in the modified input sequence 112, the duration prediction neural network 120 may repeat the representation in the intermediate sequence 122 a number of times identified by the predicted duration for each representation in the modified input sequence 112.

[0038] As another example, in an implementation where the duration prediction neural network 120 generates distributions on intermediate sequences 122, the duration prediction neural network 120 can determine the value of each intermediate element in the intermediate sequence 122 by considering the corresponding influence of the representation in the input sequence 112 modified according to its corresponding distribution combination on the intermediate element. (See below for reference.) Figure 2 Describe the process in more detail.

[0039] In some implementations, the intermediate time step is the same as the output time step; that is, each intermediate element in the intermediate sequence 122 may correspond to a corresponding audio sample in the output audio sequence 142. In some other implementations, there is a smaller intermediate time step than the output time step, and the intermediate time step may be further upsampled to generate the output audio sequence 142, for example, by the vocoder system 140, as described in more detail below.

[0040] In some implementations, after upsampling the modified input sequence 112 to generate an upsampled sequence including a corresponding upsampled representation at each intermediate time step, the duration prediction neural network 120 combines i) the upsampled representation and ii) the positional embedding of the upsampled representation for each intermediate time step to generate an intermediate element corresponding to an intermediate time step in the intermediate sequence 122.

[0041] The upsampled representation's position embedding represents the position of the upsampled representation in the upsampled sequence. Each position embedding can be machine-machined concurrently with one or more of, for example, encoder neural network 110, duration prediction neural network 120, or decoder neural network 130. As a specific example, the position embedding can be a sinusoidal position embedding.

[0042] In some implementations, the position embedding of the upsampling representation represents the global position of the upsampling representation in the upsampling sequence, i.e., the position of the upsampling representation within an intermediate time step.

[0043] In some other implementations, the positional embedding of the upsampled representation represents a local position of the upsampled representation within the upsampled sequence. For example, the positional embedding may represent the position of the upsampled representation within a subsequence of upsampled representations that correspond to the same representation in the modified input sequence (i.e., the same text element in the input text sequence 102). That is, each upsampled representation may correspond to a specific representation in the modified input sequence 112—that is, the upsampled representation is located in the modified input sequence 112 during its prediction duration. A first upsampled representation in each subsequence may have the same positional embedding, a second upsampled representation in each subsequence may have the same positional embedding, and so on.

[0044] In some other implementations, before upsampling the modified input sequence 112, the duration prediction neural network 120 may add positional embeddings to the modified input sequence 112 (e.g., where the positional embedding of each representation in the modified input sequence 112 represents the position of a representation within the modified input sequence 112) to generate an intermediate sequence 122. Reference will be made below. Figure 2 Describe the process in more detail.

[0045] In some implementations, during the training of one or more downstream neural networks to the duration prediction neural network 120 (e.g., the vocoder neural network of the decoder neural network 130 and / or the vocoder system 140), the training system may perform “teacher coercion” when upsampling the modified input sequence 112 using the predicted duration. That is, instead of upsampling the modified input sequence 112 based on the predicted duration actually generated by the duration prediction neural network 120 to generate the intermediate sequence 122, the training system can upsample the modified input sequence 112 based on the “true value” duration representing the output that the duration prediction neural network 120 should generate in response to processing the modified input sequence 112. Example techniques for training the duration prediction neural network 120 will be discussed in more detail below.

[0046] The decoder neural network 130 is configured to process the intermediate sequence 122 and generate an audio feature sequence comprising a set of audio features for each intermediate time step in the intermediate sequence 122. The set of audio features for a specific intermediate time step represents the output audio sequence 142 at one or more output time steps corresponding to that specific intermediate time step.

[0047] For example, the audio feature sequence can represent the spectral representation of the output audio sequence 142. As a specific example, the decoder neural network 130 can be configured to generate a corresponding Mel-frequency cepstral coefficient (MFCC) feature representation for each intermediate time step.

[0048] As another specific example, the decoder neural network 130 can be configured to generate a Mel spectrogram 132 corresponding to the output audio sequence 142, that is, a Mel spectrogram 132 representing the same audio data as the output audio sequence 142. The Mel spectrogram 132 may include corresponding spectrogram frames at each intermediate time step of the intermediate sequence 122. The spectrogram frame corresponding to each intermediate time step represents the predicted distribution of audio frequencies of the output audio sequence 142 at one or more output time steps corresponding to the intermediate time step.

[0049] Although the following description relates to the implementation of the decoder neural network 130 generating the Mel spectrogram 132, it should be understood that the techniques described below can be used to implement the decoder neural network 130, which generates any appropriate set of audio features corresponding to the output audio sequence 142.

[0050] In some implementations, the decoder neural network 130 autoregressively generates spectrogram frames of the Mel spectrogram 132. For example, at a first processing time step in the processing time step sequence, the decoder neural network 130 may process a first intermediate element in the intermediate sequence 122 to generate a first frame of the Mel spectrogram 132. Then, at each subsequent processing time step in the processing time step sequence, the decoder neural network 130 may process: i) subsequent intermediate elements in the intermediate sequence 122, and ii) previous frames of the Mel spectrogram 132 generated in the previous processing time step, to generate subsequent frames of the Mel spectrogram 132. That is, each processing time step may correspond to a corresponding intermediate time step in the intermediate sequence 122, and at each processing time step, the decoder neural network 130 generates a spectrogram frame of the Mel spectrogram 132 corresponding to the corresponding intermediate time step.

[0051] In some implementations, during the training of the decoder neural network 130, the training system can perform teacher coercion while autoregressively processing previous spectrogram frames to generate new spectrogram frames. That is, instead of processing the spectrogram frames actually generated by the decoder neural network 130 at the previous processing time step, the training system can process the corresponding spectrogram frames representing the “true value” Mel spectrograms of the output that the decoder neural network 130 should generate in response to processing the intermediate sequence 122. Example techniques for training the decoder neural network 130 are discussed in more detail below.

[0052] In some implementations, the decoder neural network 130 includes a "front-end" subnetwork that processes previous frames of the Mel spectrogram 132 at each processing time step to generate embeddings of those previous frames. The decoder neural network 130 can then process: i) intermediate elements of the intermediate sequence 122 corresponding to the current processing time step and ii) the embeddings of the previous spectrogram frames to generate subsequent spectrogram frames of the Mel spectrogram 132. As a specific example, the front-end subnetwork may include one or more feedforward neural network layers.

[0053] In some such implementations, the decoder neural network 130 connects i) intermediate elements of the intermediate sequence 122 corresponding to the current processing time step and ii) embeddings of previous spectrogram frames to generate a first connected representation, and then processes the first connected representation using a first subnet of the decoder neural network 130 to generate an embedding of the first connected representation. For example, the first subnet may include one or more unidirectional LSTM neural network layers.

[0054] The decoder neural network 130 can then concatenate i) the intermediate elements of the intermediate sequence 122 corresponding to the current processing time step and ii) the embedding of the first concatenation representation to generate a second concatenation representation, and then process the second concatenation representation using a second subnet of the decoder neural network 130 to generate subsequent frames of the Mel spectrogram 132. For example, the second subnet may include an output projection neural network layer, such as a feedforward neural network layer, configured to generate subsequent frames of the Mel spectrogram.

[0055] In some implementations, after generating a series of spectrogram frames as described above, the decoder neural network 130 is configured to further process the Mel spectrogram frames to generate a final Mel spectrogram 132. That is, for example, the Mel spectrogram generated by the second subnet can be an "initial" Mel spectrogram, and the decoder neural network 130 can process the initial Mel spectrogram to generate the final Mel spectrogram 132. For example, the decoder neural network 130 can use a "post-network" subnet to process the initial Mel spectrogram to generate the final Mel spectrogram 132. As a particular example, the post-network subnet may include one or more convolutional neural network layers, each configured to apply convolutional filters to the spectrogram frames (or a processed version of the spectrogram frames) of the initial Mel spectrogram.

[0056] The vocoder system 140 is configured to process the Mel spectrogram 132 to generate an output audio sequence 142. The vocoder system 140 can use any suitable technique to generate the output audio sequence 142 from the Mel spectrogram 132. For example, the vocoder system 140 may include a vocoder neural network configured to process the Mel spectrogram 132 to generate an output audio sequence.

[0057] In some implementations, the vocoder system 140 further upsamples the Mel spectrogram such that the output time steps present in the output audio sequence 142 are more than the intermediate time steps in the Mel spectrogram 132. For example, the vocoder system 140 may be configured to generate an initial output audio sequence from the Mel spectrogram 132, which includes corresponding initial audio samples for each intermediate time step, and then generate the output audio sequence 142 by processing the initial output audio sequence using a neural network—such as a convolutional neural network—configured to refine the initial output audio sequence.

[0058] In some implementations, the neural network system 100 does not include a vocoder system 140, but instead outputs a Mel spectrogram 132. That is, the neural network system 100 can be configured to generate a Mel spectrogram representing the output audio sequence instead of being configured to generate the output audio sequence itself.

[0059] In some implementations, multiple neural networks of neural network system 100 can be trained concurrently. For example, one or more of the following can be trained concurrently: encoder neural network 110; duration prediction neural network 120; decoder neural network 130; or, optionally, vocoder neural network of vocoder system 140. For example, the training system can identify errors in the output audio sequence 142 and backpropagate the errors through neural network system 100 to determine parameter updates for one or more neural networks, for example, using stochastic gradient descent.

[0060] In some such implementations, supervised learning can be used to train the encoder neural network 110, the duration prediction neural network 120, and the decoder neural network 130 concurrently. That is, the training system can use a set of training examples to determine the parameter updates of the neural networks, each training example including i) a training input text sequence, ii) a ground truth Mel spectrogram, and iii) a ground truth duration corresponding to each text input in the training input text sequence.

[0061] The training system can use the neural network system 100 to process each training input text sequence to generate a predicted Mel spectrogram 132, and then compute a loss function representing the error in the predicted Mel spectrogram 132.

[0062] For example, the loss function may include a first term that characterizes the error in the predicted duration represented by the modified input sequence 112 and generated by the duration prediction neural network 120. As a specific example, the first term may be (or proportional to):

[0063]

[0064] Where N is the number of representations in the modified input sequence 112, d represents the prediction duration, d* represents the true duration, and ||·||2 is the L2 loss.

[0065] As another example, the loss function may include a second term characterizing the errors in the generated Mel spectrogram 132. As a specific example, the second term may be (or proportional to):

[0066]

[0067] Where T is the number of intermediate time steps in the Mel spectrogram, K is the number of frequencies represented in the Mel spectrogram, and y t The value of the predicted Mel spectrum at the intermediate time step t, y t * represents the true value of the Mel-ray spectrogram at the intermediate time step t, ||·||1 is the L1 loss, and ||·||2 is the L2 loss.

[0068] In some implementations, as described above, the decoder neural network 130 first generates an initial Mel spectrogram and then processes the initial Mel spectrogram to generate a final predicted Mel spectrogram 132. In some such implementations, the second term of the loss function may include a corresponding term corresponding to the initial Mel spectrogram and the final Mel spectrogram 132. That is, the training system can use the true Mel spectrogram to determine the error in both the initial Mel spectrogram and the final Mel spectrogram 132. As a specific example, the second term may be (or proportional to):

[0069]

[0070] Among them, y t ' is the value of the initial Mel-ray spectrogram at the intermediate time step t, y t It is the value of the final Mel-frequency spectrum at the intermediate time step t, and y t * represents the true value of the Mel spectrum at the intermediate time step t.

[0071] In some other implementations, semi-supervised or unsupervised learning can be used to concurrently train the encoder neural network 110, the duration prediction neural network 120, and the decoder neural network 130. That is, the training system can determine parameter updates for the neural networks when training examples including true durations and / or true Mel-spectrum graphs are scarce or unavailable. See below for reference. Figure 3 This describes example techniques for training a neural network system 100 using unsupervised or semi-supervised learning.

[0072] The neural network system 100 can be deployed in any suitable setup. For example, the neural network system 100 can be deployed on edge devices such as mobile phones, tablets, and smart speakers, or on devices embedded in vehicles. For example, the neural network system 100 can be configured to generate an output audio sequence 142 representing audio data to be played for a user. As a particular example, audio data can be played by the speaker of a mobile phone or tablet, for example, in response to a query provided by a user. As another particular example, audio data can be played by the speakers of a vehicle, for example, to provide an alarm or instruction to the driver or another user in the vehicle.

[0073] As another example, the neural network system 100 can be deployed in the cloud, for example, in a data center that communicates with one or more edge devices. The edge devices can provide the neural network system 100 with text data and requests for synthesized audio representing the text data. The neural network system 100 can process the input text sequence 102 corresponding to the text data and generate an output audio sequence 142, and then, in response to a request, provide the output audio sequence 142 (or other data representing the output audio sequence 142) back to the edge device.

[0074] Figure 2 This is a diagram of an example duration prediction neural network 200. Duration prediction neural network 200 is an example of a system in which the systems, components, and techniques described below can be implemented as a computer program on one or more computers at one or more locations.

[0075] The duration prediction neural network 200 is configured to obtain an embedding 202 of an input text sequence representing text data. The input text sequence may include corresponding text elements at each of a plurality of input time steps, and the embedded input sequence 202 may include a corresponding representation of each text element corresponding to the respective input time step.

[0076] The duration prediction neural network 200 is configured to process the input text sequence 102 and generate an upsampled sequence 232 representing the same text data as the input text sequence 102. The upsampled sequence 232 includes a corresponding intermediate element at each of a plurality of intermediate time steps, wherein the number of intermediate time steps is greater than the number of input time steps in the embedded input sequence 202.

[0077] The duration prediction neural network 200 is configured to generate an upsampled sequence 232 by predicting the corresponding duration of the text element if the text data were spoken, for each text element represented by the embedded input sequence 202. That is, if audio data is generated from the embedded input sequence 202, the duration representation of the text element will be used to indicate the amount of time it would take to speak the text represented by the text element.

[0078] For example, duration prediction neural network 200 can be a component of a neural network system configured to process an input text sequence and generate an output audio sequence representing the speaker who uttered the input text sequence. As a specific example, duration prediction neural network 200 can be the one mentioned above. Figure 1 The neural network system 100 described is a duration prediction neural network 120.

[0079] The duration prediction neural network 200 includes a duration prediction subnetwork 210, a range prediction subnetwork 220, and an upsampling system 230.

[0080] The duration prediction neural network 200 is configured to determine a corresponding distribution over the intermediate elements of the upsampled sequence 232 for each representation in the embedded input sequence 202, the distribution modeling the influence of the representation on each intermediate element in the upsampled sequence 232. For each representation in the embedded input sequence 202, the corresponding distribution can define a value for each intermediate element in the upsampled sequence 232, representing the influence of that representation on the intermediate element.

[0081] For each representation in the embedded input sequence 202, the distribution can be parameterized by: (i) the predicted duration 212 of the representation determined by the duration prediction subnet and (ii) the range parameter 222 of the representation determined by the range prediction subnet 220. As described in more detail below, the center of the distribution can be defined by the corresponding predicted duration 212, while the variance of the distribution can be defined by the corresponding range parameter 222. The range parameter 222 of a particular representation can represent the importance of that representation, for example, its relative influence on the output audio sequence.

[0082] Specifically, the duration prediction subnet 210 is configured to process the embedded input sequence 202 and generate a corresponding predicted duration 212 for each representation in the embedded input sequence 202. For example, each predicted duration 212 may be an integer representing the number of intermediate time steps in the upsampled sequence 232.

[0083] As a specific example, the duration prediction subnetwork 210 may use one or more bidirectional LSTM neural network layers to process the embedded input sequence 202. The duration prediction subnetwork 210 may also include an output projection neural network layer, such as a feedforward neural network layer, which is configured to receive the corresponding layer input for each representation in the embedded input sequence 202 and generate a predicted duration 212 for each representation in the embedded input sequence 202.

[0084] Range prediction subnetwork 220 is configured to process the embedded input sequence 202 and optionally the predicted duration 212 generated by duration prediction subnetwork 210, and generate a corresponding range parameter 222 for each representation in the embedded input sequence 202. For example, for each representation in the embedded input sequence 202, range prediction subnetwork 220 may, for example, use a connection combination: i) the representation and ii) the predicted duration 212 of the representation. Range prediction subnetwork 220 can then process the combined representations to generate the corresponding range parameter 222. As a particular example, range prediction subnetwork 220 may include one or more bidirectional LSTM neural network layers. Range prediction subnetwork 220 may also include an output projection neural network layer, such as a feedforward neural network layer, configured to receive a corresponding layer input for each representation in the embedded input sequence 202 and generate the range parameter 222 for each representation in the embedded input sequence 202.

[0085] Upsampling system 230 is configured to generate upsampled sequence 232 from embedded input sequence 202 based on the corresponding prediction duration 212 and range parameter 222 of the representation of the embedded input sequence 202. For each intermediate element in upsampled sequence 232, upsampling system 230 can then determine the value of the intermediate element, as defined by prediction duration 212 and range parameter 222, by combining the corresponding effects of each representation in embedded input sequence 202 on the intermediate element.

[0086] For example, for each intermediate element of the upsampled sequence 232, the upsampling system 230 can determine a weighted sum of the values ​​of the intermediate elements in the corresponding distribution of each representation in the embedded input sequence 202. In some implementations, the sum of the corresponding values ​​of the intermediate elements in each distribution can be used to normalize the weights corresponding to each representation in the upsampled input sequence 232.

[0087] As a specific example, the distribution over each representation's upsampled sequence 232 can be a Gaussian distribution. In this example, the value u of the intermediate element t in the upsampled sequence 232... t It can be determined by calculating the following formula:

[0088]

[0089] Among them, h i The value representing i, w, is in the embedded input sequence 202. ti It is calculating the value u of the intermediate element t. t The weight of representation i is given by N, where N is the number of representations in the embedded input sequence 202, and It is the value of the intermediate element t in the distribution corresponding to the intermediate element i.

[0090] The distribution of each representation i over the intermediate elements can have a center c determined by the prediction duration 212 using that representation. i Center C i This can correspond to the center of the predicted duration 212 in the upsampled sequence 232. For example, the upsampling system 230 can determine the center c of the distribution corresponding to representation i by calculating the following formula. i :

[0091]

[0092] Where, d i It represents the predicted duration of i, and each d j It is the predicted duration of the corresponding representation preceding i in the embedded input sequence 202.

[0093] The distribution of each representation i over the intermediate elements can have a variance determined using the range parameter 222 of the representation. In some implementations, the upsampling system 230 is determined to represent that the range parameter 222 generated by i is equal to the variance of the corresponding distribution. In some other implementations, the upsampling system 230 determines that the generated range parameter 222 is equal to the standard deviation σ of the corresponding distribution. i .

[0094] In some implementations, upsampling based on the corresponding distribution on the upsampled sequence 232 is fully differentiable, thereby allowing the duration prediction neural network 200 to work with one or more downstream neural networks (e.g., decoder neural networks, as referenced above). Figure 1 The decoder neural network 130 described is trained end-to-end. This discriminability may be particularly important when the true duration is unavailable and therefore cannot be used to train the duration prediction neural network 200 in a supervised manner.

[0095] In some implementations, the duration prediction neural network 200 does not include a range prediction subnetwork 220, and the variance of the distribution for each representation i is... It is fixed. In some such implementations, the variance... It can depend on the speaker that the output audio sequence is meant to represent.

[0096] As referenced above Figure 1 In some embodiments, the duration prediction neural network 200 combines each intermediate element in the upsampled sequence 232 with its corresponding position embedding.

[0097] In some other embodiments, the duration prediction neural network 200 incorporates positional embeddings into the representations in the embedded input sequence 202 (e.g., by appending positional embeddings to the representations) before upsampling the embedded input sequence 202 to generate the upsampled sequence 232. Because each intermediate element in the upsampled sequence 232 can be a linear combination of the representations in the embedded input sequence 202 (wherein the linear combination is determined according to the distribution described above), the positional embeddings appended to the representations can also be linearly combined to generate a correspondingly different positional embedding for each intermediate element in the upsampled sequence 232. In other words, the duration prediction neural network 200 can determine the corresponding positional embedding p for each intermediate element in the upsampled sequence 232 by calculating the following formula. t :

[0098]

[0099] Where, q i It is the position embedding of the i-th representation in the embedded input sequence 202, and w ti Defined as above.

[0100] In some implementations, the duration prediction subnet 210 and the range prediction subnet 220 are trained concurrently with one or more other neural networks, such as one or more of the following: encoder neural networks (e.g., referenced above). Figure 1 The encoder neural network 110 described is configured to generate an embedded input sequence 202 from an input sequence; the decoder neural network (e.g., referenced above) Figure 1 The decoder neural network 130 described is configured to generate an audio feature set from the upsampled sequence 232; or, a vocoder neural network (e.g., as referenced above) Figure 1 The vocoder system 140 described herein is a vocoder neural network, which is configured to generate an output audio sequence from an audio feature set generated from an upsampled sequence 232.

[0101] For example, as referenced above Figure 1 The duration prediction subnet 210 and the range prediction subnet 220 can be trained using the true value duration of each representation in the embedded input sequence 202.

[0102] As another example, the duration prediction subnet 210 and the range prediction subnet 220 can be trained without access to any training examples that include the true duration, or with very little access to training examples that include the true duration. See below for reference. Figure 3 A more detailed description of example techniques for training a duration prediction neural network 200 using unsupervised or semi-supervised learning is provided.

[0103] Figure 3 This is a diagram of an example training system 300. Training system 300 is an example of a system in which the systems, components, and techniques described below can be implemented as computer programs on one or more computers at one or more locations.

[0104] Training system 300 is configured to generate machine learning parameters for a neural network, which is configured to process input text sequence 302 and generate a Mel spectrogram 352 representing audio data of the spoken input text sequence 302. For example, the neural network can be a neural network system (e.g., see above reference). Figure 1 The components of the neural network system 100 described are configured to use a vocoder system (e.g., the one mentioned above). Figure 1 The described vocoder system 140 generates an output audio sequence from a Mel spectrogram 352.

[0105] The neural network includes an encoder neural network 310, a direction prediction neural network 340, and a decoder neural network 350.

[0106] The training system 300 is configured to train a neural network using a training dataset stored in a training data repository 301. The training dataset includes multiple training examples, each including (i) a training input text sequence 302 and (ii) a set of ground truth audio features 304 representing audio data characterizing the speaker uttering the training input text sequence 302. However, these training examples do not include the ground truth durations of the text elements of the training input text sequence 302.

[0107] In order to train a neural network without access to the duration of the true value, the training system 300 may modify the architecture of the neural network to add (i) an audio feature embedding neural network 320 and (ii) an attention-based neural network 330.

[0108] The audio feature embedding neural network 320 and the attention-based neural network 330 are configured to process the ground truth audio features 304 and extract latent features 332 from the ground truth audio features 304 to guide the training of neural networks that do not have ground truth durations. For example, the audio feature embedding neural network 320 and the attention-based neural network 330 can make full use of the duration information implicitly encoded into the ground truth audio features 304.

[0109] The training input text sequence 302 includes corresponding text elements for each of the multiple input time steps.

[0110] The ground truth audio features 304 include a corresponding set of audio features for each of a plurality of intermediate time steps, wherein the number of intermediate time steps is greater than the number of input time steps. The ground truth audio features 304 may include any suitable features characterizing the output audio sequence. For example, the ground truth audio features 304 may include, for instance, a Mel spectrogram of the output audio sequence generated from a speaker uttering the ground truth output audio sequence of the training input text sequence 302. Alternatively or additionally, the ground truth audio features may include a logarithmic spectrogram generated from the ground truth output audio sequence, waveform features of the ground truth output audio sequence, or pitch contour features of the ground truth output audio sequence. Alternatively or additionally, the ground truth audio features may include vocoder parameters configured to synthesize the output audio sequence.

[0111] For each training example, the training system 100 can use an encoder neural network 310 to process the training input text sequence 302 to generate a modified input sequence 312, which includes a corresponding representation of each text element of the training text sequence 302. For example, the encoder neural network 310 can be similar to the one described above. Figure 1 The encoder neural network 110 is described and configured.

[0112] The training system 300 can use an audio feature embedding neural network 320 to process the ground truth audio features 304 to generate a set of embedded audio features, which includes the embedding of the corresponding audio feature 304 for each intermediate time step represented by the ground truth audio features 304. In embodiments where the ground truth audio features 304 include Mel spectrograms, the audio feature embedding neural network 320 is sometimes referred to as a spectrogram embedding neural network. For example, the audio feature embedding neural network 320 may include one or more convolutional neural network layers configured to apply convolutional kernels to a sequence of ground truth audio features 304 (or a processed version thereof) to generate a sequence of embedded audio features 322. As another example, the audio feature embedding neural network 320 may include one or more recurrent neural network layers, such as one or more bidirectional LSTM neural network layers configured to recursively process the sequence of ground truth audio features 304 to generate a sequence of embedded audio features.

[0113] The training system 300 can then use an attention-based neural network 330 to process (i) the modified input sequence 312 and (ii) the embedded audio features 322 to generate a set of latent features 332 for the output audio sequence. This set of latent features 332 includes a corresponding set of features for each input time step—that is, for each representation of the modified input sequence 312. Because the attention-based neural network 330 “combines” the modified input sequence 312 and the embedded audio features 322 to generate the latent features 332, the attention-based neural network 330 is sometimes referred to as a combinatorial neural network.

[0114] The attention-based neural network 330 can use one or more attention neural network layers to process the modified input sequence 312 and the embedded audio features 322 to align the two inputs. Specifically, for each representation in the modified input sequence 312, the attention-based neural network can generate a corresponding attention output by applying an attention mechanism to the corresponding embedded audio feature 322 corresponding to each intermediate time step. That is, the attention-based neural network 330 can determine for each input time step i:

[0115] c i =Attn(h i f spec (Y * ))

[0116] Among them, c i It is the attention output for input time step i; Attn is an attention mechanism, such as dot product attention or scaled dot product attention; h i Y* is the modified representation of the input sequence 312 corresponding to the input time step i, and is used as a query for the attention mechanism; Y* is the set of ground truth audio features 304; and f spec This represents the output of the audio feature embedding neural network 320 and is used as the value for the attention mechanism.

[0117] In some implementations, one or more attention neural network layers may be multi-head attention neural network layers, which receive layer inputs, apply multiple different attention mechanisms to the layer inputs to generate corresponding attention output sets, and combine the corresponding attention outputs to generate a final attention output set. For example, a multi-head attention neural network layer may apply multiple different attention mechanisms in parallel.

[0118] The attention-based neural network 330 can then process (i) the modified input sequence 312 and (ii) the generated attention output corresponding to the corresponding representation in the modified input sequence 312 to generate latent features 332. For example, the attention-based neural network 330 can process both inputs using a variational autoencoder. As a specific example, the variational autoencoder could be a conditional variational autoencoder, conditioned on the modified input sequence 312. The variational autoencoder can have a Gaussian prior, such as N(0,1).

[0119] For example, the attention-based neural network 330 can use the lower bound of evidence (ELBO) to optimize the variational autoencoder:

[0120]

[0121] Where H is the modified input sequence 312, Z is the posterior latent feature 332, the first term is the KL divergence between the prior and posterior, and the second term can be approximated by drawing samples from the posterior.

[0122] Then, the training system 300 can, for example, combine (i) the modified input sequence 312 and (ii) the latent features 332 of the true value audio features 304 (and optionally, the identifier of the class to which the output audio sequence should belong, as described above) by concatenation to generate the input for the duration prediction neural network 340, which can be referred to as the training modified input sequence.

[0123] The duration prediction neural network 340 can then process the modified input sequence to generate the predicted duration for each representation in the modified input sequence 312, and upsample the modified input sequence 312 according to the predicted duration to generate an intermediate sequence 342, which includes the corresponding intermediate element for each intermediate time step. For example, it can be similar to the above reference. Figure 1 The duration prediction neural network described in 120 or the above reference Figure 2 The duration prediction neural network 200 is described to configure the duration prediction neural network 340.

[0124] Therefore, even without the true duration to guide training, the training system 300 can use the true audio features 304 to provide information to the duration prediction neural network 340 to help train the duration prediction neural network 340.

[0125] In some implementations, the training system 300 may perform a “teacher coercion” regarding the total duration of the intermediate elements of the intermediate sequence 342. Although the true duration for each text element in the input text sequence 302 is not available, the total duration of the output audio sequence may be known, and the training system 300 may force the duration of the intermediate sequence 342 to match the true total duration of the output audio sequence. If the sum of the predicted durations of the representations in the modified input sequence 312 determined by the duration prediction neural network 340 differs from the desired total duration of the intermediate sequence 342, the duration neural network 340 may modify the corresponding predicted duration of the representation before upsampling. For example, the duration prediction neural network 340 may scale the predicted duration of each representation based on the ratio between the desired duration of the intermediate sequence 342 and the sum of the predicted durations. Thus, the training system 300 may force the Mel spectrogram 352 generated by the neural network to have the same size as the corresponding true Mel spectrogram, which is helpful in identifying errors in the Mel spectrogram 352.

[0126] Because the true set of audio features 304 is unavailable during inference, in these embodiments, the neural network can generate the input to the duration prediction neural network 340 during inference by combining—for example, connecting—(i) the modified input sequence 312 and (ii) the feature set determined according to the prior distribution of latent features 332. The prior distribution can be, for example, the prior of the variational autoencoder described above, such as a Gaussian prior.

[0127] For example, the feature set can be a pattern of a prior distribution, or it can be randomly sampled from a prior distribution. As a specific example, the pattern of the prior distribution can be a zero vector.

[0128] The decoder neural network 350 can process the intermediate sequence to generate a Mel spectrogram 352 representing the output audio sequence. For example, it can be compared with the above reference... Figure 1 The decoder neural network 130 described is similarly configured with the decoder neural network 350. As mentioned above, although in Figure 3 The diagram depicts the generation of Mel spectrogram 352, but typically the decoder neural network 350 can be configured to process intermediate sequences 342 to generate corresponding sequences for any appropriate set of audio features.

[0129] Training engine 360 ​​can determine parameter updates 362 for each of one or more of the following: encoder neural network 310, audio feature embedding neural network 320, attention-based neural network 330, duration prediction neural network 340, or decoder neural network 350. In some embodiments, training engine 360 ​​determines parameter updates 362 for each neural network. In some other embodiments, one or more neural networks have been pre-trained, and training system 300 freezes their parameter values ​​during the training of other neural networks.

[0130] The training engine 360 ​​can use a loss function to determine parameter updates, which includes a first term representing the error in the predicted Mel spectrogram (and optionally, the initial Mel spectrogram). As a specific example, the first term could be (or proportional to) the reference above. Figure 1 The sign One of the terms. As another specific example, the first term could be (or proportional to) the second term of the ELBO expression identified above, that is,

[0131]

[0132] As an alternative or supplement, the loss function may include a second term characterizing the error in the total predicted duration of the output audio sequence (i.e., the sum of all predicted durations). That is, although the true duration of each text element in the input text sequence 302 is unavailable, the total duration of the output audio sequence may be known and can be compared to, for example, the total duration of the Mel spectrogram 352. As a specific example, the second term may be (or proportional to):

[0133]

[0134] Where N is the number of modified input sequences represented in 312, T is the number of intermediate time steps, and d i Let ||i|| represent the predicted duration of i in the modified input sequence, and ||·||2 is the L2 loss.

[0135] As an alternative or supplement, the loss function may include a third term characterizing the KL divergence loss of the variational autoencoder in the attention-based neural network 330. As a specific example, the third term may be (or proportional to) the first term of the ELBO expression identified above, i.e.

[0136]

[0137] In some implementations, the training data repository 301 includes one or more labeled training examples that include real-value durations, in addition to training examples that do not contain real-value durations. In these implementations, when processing labeled training examples, the training system 300 can determine a loss function that includes at least a fourth term characterizing errors in the predicted durations generated by the duration prediction neural network 340, for example, as referenced above. Figure 1 The sign item.

[0138] After the neural network has been trained on the training system 300, it can be deployed in any suitable setup, such as in a data center or on an edge device, as described in the reference above. Figure 1 As stated above.

[0139] Figure 4 This is a flowchart of an example process 400 for processing an input text sequence using a neural network system to generate an output audio sequence. For convenience, process 400 will be described as being executed by a system of one or more computers located in one or more locations. For example, a neural network system appropriately programmed according to this specification, such as... Figure 1 The neural network system 100 described herein can execute process 400.

[0140] The system obtains the input text sequence (step 402). The input text sequence may include corresponding text elements at each of multiple input time steps. For example, each text element represents a character, phoneme, or word. Alternatively or supplementarily, each text element may include a set of language features derived from the text data represented by the input text sequence.

[0141] The system uses a first neural network to process the input text sequence to generate a modified input sequence (step 404). For each of the multiple input time steps, the modified input sequence may include representations of corresponding text elements in the input text sequence. For example, the first neural network may be the one referenced above. Figure 1 The encoder neural network 110 is described. As another example, the first neural network may include the referenced above. Figure 3 The encoder neural network 310, the audio feature embedding neural network 320, and the attention-based neural network 330 are described.

[0142] For example, for each text element in the input text sequence, the system can obtain a predefined embedding for that text element. The system can then use a first neural network to process the predefined embeddings of the text elements to generate a modified input sequence.

[0143] In some implementations, the first neural network includes one or more of the following: one or more convolutional neural network layers, one or more unidirectional LSTM neural network layers, or one or more bidirectional LSTM neural network layers.

[0144] In some implementations, the system may use a first neural network to process the input text sequence to generate an initial modified input sequence, which includes a corresponding initial representation at each input time step. The system can then combine the initial representation with an identifier of the class to which the output audio sequence should belong, for example, by concatenating the initial representation with an identifier of the class. For example, each class could correspond to a speaker that the output audio sequence should sound like.

[0145] The system uses a second neural network to process the modified input sequence to generate the predicted duration of the corresponding text element in the output audio sequence for each input time step (step 406).

[0146] For example, the second neural network could be the reference mentioned above. Figure 1 The duration neural network 120 described above; referenced above. Figure 2 The duration neural network 200 described; or refer to the above. Figure 3 The duration prediction neural network 340 is described.

[0147] For example, the second neural network may include one or more of the following: one or more convolutional neural network layers; one or more unidirectional LSTM neural network layers; one or more bidirectional LSTM neural network layers; or, an output projection layer configured to receive corresponding layer inputs for each representation in the modified input sequence and generate a predicted duration for each representation in the modified input sequence.

[0148] The system upsamples the modified input sequence based on the predicted duration to generate an intermediate sequence (step 408). The intermediate sequence may include corresponding intermediate elements at each of multiple intermediate time steps.

[0149] In some implementations, the system can determine a distribution over an intermediate sequence for each representation in the modified input sequence, based on the predicted duration of that representation. The system can then generate each intermediate element in the intermediate sequence by determining a weighted sum of the representations, wherein each representation is weighted according to the value of an intermediate element in the distribution over the intermediate sequence corresponding to that representation. For example, the distribution of each corresponding representation could be a Gaussian distribution, where the center of the Gaussian distribution corresponds to the center of the predicted duration of the representation.

[0150] In some such implementations, a fourth neural network can be used—for example, as referenced above. Figure 2The duration prediction neural network 200 described herein has a range prediction subnetwork 220 that processes the modified input sequence to generate the variance of a Gaussian distribution for each corresponding representation. For example, the system may combine each representation in the modified input sequence with the predicted duration of that representation to generate a corresponding combined representation. The system may use a fourth neural network to process the combined representations to generate the corresponding variance of a Gaussian distribution for each representation. As a specific example, the fourth neural network may include one or more of the following: one or more convolutional neural network layers; one or more unidirectional LSTM neural network layers; one or more bidirectional LSTM neural network layers; or, an output projection layer configured to receive a corresponding layer input for each representation in the modified input sequence and generate the corresponding variance of a Gaussian distribution for each representation in the modified input sequence.

[0151] In some implementations, the system can upsample the modified input sequence to generate an upsampled sequence comprising corresponding upsampled representations at each intermediate time step. By combining the upsampled representation with a positional embedding within it for each upsampled representation in the upsampled text sequence, the system can generate intermediate sequences from the upsampled sequence. For example, the positional embedding of the upsampled representation can identify the position of the upsampled representation in a subsequence of the upsampled representation corresponding to the same representation in the modified input sequence.

[0152] The system uses the intermediate sequence to generate the output audio sequence (step 410). The output audio sequence may include corresponding audio samples at each of a plurality of output time steps. In some implementations, the output time step is the same as the intermediate time step.

[0153] For example, the system can use a third neural network to process the intermediate sequences to generate a Mel spectrogram, which includes corresponding spectrogram frames at each intermediate time step. The system can then process the Mel spectrogram to generate an output audio sequence. For example, the third neural network could be the one referenced above. Figure 1 The decoder neural network described is 130 or the above reference. Figure 3 The decoder neural network described is 350.

[0154] At a first processing time step, the system may use a third neural network to process a first intermediate element in the intermediate sequence to generate a first frame of the Mel spectrogram. At each of a plurality of subsequent processing time steps, the system may use the third neural network to process i) the corresponding subsequent intermediate element in the intermediate sequence and ii) a previous frame of the Mel spectrogram generated in the previous processing time step to generate subsequent frames of the Mel spectrogram. As a specific example, the third neural network may include one or more of the following: one or more convolutional neural network layers; one or more unidirectional LSTM neural network layers; one or more bidirectional LSTM neural network layers; or, an output projection layer configured to receive layer input and generate subsequent frames of the Mel spectrogram.

[0155] For example, the system can use one or more fully connected neural network layers to process previous spectrogram frames to generate embeddings of the previous frames, for example, as referenced above. Figure 3 The described decoder neural network 130 consists of one or more fully connected neural network layers in its "front-end". The system can then use a third neural network to process i) subsequent intermediate elements in the intermediate sequence and ii) embeddings of previous frames to generate subsequent frames of the Mel spectrogram.

[0156] As a specific example, the system can concatenate i) subsequent intermediate elements in the intermediate sequence and ii) the embeddings of previous frames to generate a first concatenated representation; and process the first concatenated representation using a first subnet of a third neural network to generate an embedding of the first concatenated representation. For example, the first subnet may include one or more unidirectional LSTM neural network layers. The system can then concatenate i) subsequent intermediate elements in the intermediate sequence and ii) the embeddings of the first concatenated representation to generate a second concatenated representation; and process the second concatenated representation using a second subnet of a third neural network to generate subsequent frames of the Mel spectrogram. For example, the second subnet may include an output projection neural network layer, such as a feedforward neural network layer, configured to generate subsequent frames of the Mel spectrogram.

[0157] In some implementations, to generate a Mel spectrogram, the system can use a third subnetwork of a third neural network to process intermediate sequences to generate an initial Mel spectrogram. For example, the third subnetwork may include a first subnetwork and a second subnetwork. The system can then use a fourth subnetwork of the third neural network to process the initial Mel spectrogram to generate the Mel spectrogram. For example, the fourth subnetwork may include one or more convolutional neural network layers, as described above. Figure 3 The decoder neural network 130 described consists of one or more convolutional neural network layers in its "post-network".

[0158] In some implementations, the first, second, and third neural networks have been trained concurrently. In some such implementations, the neural networks can be trained for any real-value duration of the representation in the input sequence without modification.

[0159] For example, the training system can obtain a sequence of training input texts, which includes corresponding training text elements at each of multiple training input time steps. The training system can use a first subnetwork of a first neural network (e.g., see reference above). Figure 3 The described encoder neural network 310 processes the training input text sequence to generate an embedding of the training input text sequence. The training system can obtain a Mel-spectrum map of the ground truth corresponding to the training input text sequence. The training system can use a second subnetwork of the first neural network (e.g., referenced above). Figure 3 The described audio feature embedding neural network 320 processes the ground truth Mel spectrogram to generate embeddings of the ground truth Mel spectrogram. The training system can combine i) the embeddings of the training input text sequence and ii) the embeddings of the ground truth Mel spectrogram to generate a training modified input sequence, which, for each training input time step, includes representations of corresponding training text elements in the training input text sequence. The training system can then process the training modified input sequence using a second neural network (e.g., a duration prediction neural network 340) to generate a predicted duration for each representation in the training modified input sequence.

[0160] As a specific example, the training system can be achieved by using a first neural network—for example, as referenced above. Figure 3 The third subnet of the attention-based neural network 330 described processes two embeddings, i) an embedding of the training input text sequence and ii) an embedding of the ground truth Mel spectrogram, to combine the two embeddings.

[0161] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers configured to perform a specific operation or action, this means that software, firmware, hardware, or a combination thereof have been installed on the system, which, in operation, causes the system to perform the operation or action. For one or more computer programs configured to perform a specific operation or action, this means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0162] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuit systems, in tangibly implemented computer software or firmware, in computer hardware—including the structures disclosed in this specification and their structural equivalents—or in combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof. Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0163] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or further include a dedicated logic circuit system, such as a FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0164] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages; and the computer program may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not need to, correspond to a file in a file system. A program may be stored as a portion of a file containing other programs or data, such as in one or more scripts stored in a markup language document, in a single file dedicated to the program involved, or in multiple collaborating files, such as in a file storing one or more modules, subroutines, or code portions. A computer program may be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple locations and interconnected via a data communication network.

[0165] In this specification, the term "database" is used broadly to refer to any collection of data: data that does not need to be structured in any particular way, or not structured at all, and which can be stored on storage devices in one or more locations. Thus, for example, an indexed database may include multiple collections of data, each of which can be organized and accessed differently.

[0166] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0167] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by a dedicated logic circuit system—such as an FPGA or ASIC—or a combination of a dedicated logic circuit system and one or more programmable computers.

[0168] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into a special-purpose logic circuit system. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for receiving data from or transferring data to, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer does not require such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0169] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0170] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user—e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor—and a keyboard and pointing device—e.g., a mouse or trackball—through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to and from a device used by the user; for example, sending a webpage to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device—e.g., a smartphone running a messaging application—and receiving response messages in return from the user.

[0171] The data processing apparatus used to implement machine learning models may also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, namely inference and workloads.

[0172] Machine learning frameworks—such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet—can be used to implement and deploy machine learning models.

[0173] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes, for example, a back-end component as a data server, or a middleware component as an application server, or a front-end component as a client computer having, for example, a graphical user interface, web browser, or application through which a user can interact with embodiments of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. Components of the system can be interconnected via digital data communication of any form or medium—for example, a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0174] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data—e.g., HTML pages—to a user device, for purposes such as displaying data to a user interacting with the device acting as a client and receiving user input from the user. Data generated at the user device, such as the results of user interactions, may be received at the server from the device.

[0175] In addition to the above embodiments, the following embodiments are also innovative:

[0176] Example 1 is a method for generating an output audio sequence from an input text sequence, wherein the input text sequence includes corresponding text elements at each of a plurality of input time steps, and the output audio sequence includes corresponding audio samples at each of a plurality of output time steps, the method comprising:

[0177] The input text sequence is processed using a first neural network to generate a modified input sequence, wherein for each of the plurality of input time steps, the input sequence includes a representation of the corresponding text element in the input text sequence;

[0178] The modified input sequence is processed using a second neural network to generate a predicted duration for the corresponding text element in the output audio sequence for each input time step.

[0179] The modified input sequence is upsampled according to the predicted duration to generate an intermediate sequence, the intermediate sequence including a corresponding intermediate element at each of a plurality of intermediate time steps; and

[0180] The intermediate sequence is used to generate the output audio sequence.

[0181] Example 2 is the method of Example 1, wherein processing the input text sequence using the first neural network to generate the modified input sequence includes:

[0182] For each text element in the input text sequence, obtain the predetermined embedding of the text element; and

[0183] The first neural network is used to process the predetermined embeddings of the plurality of text elements to generate the modified input sequence.

[0184] Example 3 is a method of any one of Examples 1 or 2, wherein the first neural network includes one or more of the following:

[0185] One or more convolutional neural network layers;

[0186] One or more unidirectional LSTM neural network layers; or

[0187] One or more bidirectional LSTM neural network layers.

[0188] Example 4 is a method of any one of Examples 1 to 3, wherein processing the input text sequence using a first neural network to generate the modified input sequence includes:

[0189] The input text sequence is processed using the first neural network to generate an initial modified input sequence, the initial modified input sequence including a corresponding initial representation at each of the plurality of input time steps; and

[0190] Generating the modified input sequence from the initial modified input sequence includes, for each initial representation in the initial modified input sequence, combining the initial representation with an identifier of the class to which the output audio sequence should belong.

[0191] Example 5 is the method of Example 4, wherein each class corresponds to a speaker that the output audio sequence should sound like.

[0192] Example 6 is a method of any one of Examples 1 to 5, wherein the second neural network includes one or more of the following:

[0193] One or more convolutional neural network layers;

[0194] One or more unidirectional LSTM neural network layers;

[0195] One or more bidirectional LSTM neural network layers; or

[0196] An output projection layer is configured to receive a corresponding layer input for each representation in the modified input sequence and to generate the predicted duration for each representation in the modified input sequence.

[0197] Example 7 is a method of any one of Examples 1 to 6, wherein upsampling the modified input sequence according to the predicted duration to generate an intermediate sequence includes:

[0198] Based on the predicted duration of the representation, determine the distribution of each representation in the modified input sequence on the intermediate sequence; and

[0199] For each intermediate element in the intermediate sequence, the intermediate element is generated by determining a weighted sum of the representations, wherein each representation is weighted according to the value of the intermediate element in the distribution on the intermediate sequence corresponding to the representation.

[0200] Example 8 is the method of Example 7, wherein the distribution of each corresponding representation is a Gaussian distribution, wherein the center of the Gaussian distribution corresponds to the center of the predicted duration of the representation.

[0201] Example 9 is the method of Example 8, wherein the center of the Gaussian distribution specifically represented is:

[0202]

[0203] Among them, c i It is the center of the Gaussian distribution in the specific representation, d i It is the predicted duration of the specific representation, and each d j It is the predicted duration of the corresponding representation preceding the specific representation in the modified input sequence.

[0204] Example 10 is a method of any one of Examples 8 or 9, wherein the variance of the Gaussian distribution for each corresponding representation is generated by processing the modified input sequence using a fourth neural network.

[0205] Example 11 is the method of Example 10, wherein processing the modified input sequence using the fourth neural network includes:

[0206] For each representation in the modified input sequence, the representation is combined with the predicted duration of the representation to generate a corresponding combined representation; and

[0207] The combined representations are processed using the fourth neural network to generate the corresponding variance of the Gaussian distribution for each representation.

[0208] Example 12 is a method of any one of Examples 10 or 11, wherein the fourth neural network includes one or more of the following:

[0209] One or more convolutional neural network layers;

[0210] One or more unidirectional LSTM neural network layers;

[0211] One or more bidirectional LSTM neural network layers; or

[0212] An output projection layer is configured to receive a corresponding layer input for each representation in the modified input sequence and to generate the corresponding variance of the Gaussian distribution for each representation in the modified input sequence.

[0213] Example 13 is a method of any one of Examples 1 to 12, wherein upsampling the modified input sequence to generate an intermediate sequence includes:

[0214] The modified input sequence is upsampled to generate an upsampled sequence including a corresponding upsampled representation at each of the plurality of intermediate time steps; and

[0215] Generating the intermediate sequence from the upsampled sequence includes combining the upsampled representation with the positional embedding of the upsampled representation for each upsampled representation in the upsampled text sequence.

[0216] Example 14 is the method of Example 13, wherein the position embedding of the upsampled representation identifies the position of the upsampled representation in a subsequence of the upsampled representation corresponding to the same representation in the modified input sequence.

[0217] Example 15 is a method of any one of Examples 1 to 14, wherein the first neural network and the second neural network have been trained concurrently.

[0218] Example 16 is a method of any one of Examples 1 to 15, wherein generating the output audio sequence using the intermediate sequence includes:

[0219] The intermediate sequences are processed using a third neural network to generate Mel spectrograms, the Mel spectrograms including corresponding spectrogram frames at each of the plurality of intermediate time steps; and

[0220] The Mel spectrogram is processed to generate the output audio sequence.

[0221] Example 17 is a method of Example 16, wherein processing the intermediate sequence using a third neural network to generate a Mel spectrogram includes:

[0222] At the first processing time step in the processing time step sequence, the third neural network is used to process the first intermediate element in the intermediate sequence to generate the first frame of the Mel spectrogram; and

[0223] At each subsequent processing time step in the sequence of processing time steps, a third neural network is used to process i) subsequent intermediate elements in the intermediate sequence and ii) previous frames of the Mel spectrogram generated in the previous processing time step to generate subsequent frames of the Mel spectrogram.

[0224] Example 18 is the method of Example 17, wherein the third neural network includes one or more of the following:

[0225] One or more convolutional neural network layers;

[0226] One or more unidirectional LSTM neural network layers;

[0227] One or more bidirectional LSTM neural network layers; or

[0228] An output projection layer is configured to receive the input from the receiving layer and generate the subsequent frames of the Mel spectrogram.

[0229] Example 19 is a method of any one of Examples 17 or 18, wherein processing i) the subsequent intermediate elements in the intermediate sequence and ii) the previous frames of the Mel spectrogram using the third neural network includes:

[0230] The previous frame is processed using one or more fully connected neural network layers to generate an embedding of the previous frame; and

[0231] The third neural network is used to process i) the subsequent intermediate elements in the intermediate sequence and ii) the embeddings of the previous frames to generate the subsequent frames of the Mel spectrogram.

[0232] Example 20 is a method of Example 19, wherein processing i) the subsequent intermediate elements in the intermediate sequence and ii) the embedding of the previous frame using the third neural network includes:

[0233] Connect i) the subsequent intermediate elements in the intermediate sequence and ii) the embedding of the previous frame to generate a first linked representation;

[0234] The first connection representation is processed using a first subnet of the third neural network to generate an embedding of the first connection representation;

[0235] Connecting i) the subsequent intermediate elements in the intermediate sequence and ii) the embedding of the first link representation to generate a second link representation; and

[0236] The second connection representation is processed using the second subnet of the third neural network to generate the subsequent frames of the Mel spectrogram.

[0237] Example 21 is a method of any one of Examples 16 to 20, wherein processing the intermediate sequence using the third neural network to generate the Mel spectrogram includes:

[0238] The intermediate sequence is processed using a third subnetwork of the third neural network to generate an initial Mel spectrogram; and

[0239] The initial Mel spectrogram is processed using the fourth subnet of the third neural network to generate the Mel spectrogram.

[0240] Example 22 is the method of Example 21, wherein the fourth subnetwork of the third neural network includes one or more convolutional neural network layers.

[0241] Example 23 is a method of any one of Examples 16 to 22, wherein the first neural network, the second neural network, and the third neural network have been trained concurrently.

[0242] Example 24 is the method of Example 23, wherein the neural network is trained using a loss term including one or more of the following:

[0243] The first term characterizes an error in the predicted duration of the representation in the modified input sequence; or

[0244] The second term characterizes errors in the generated Mel spectrogram.

[0245] Example 25 is the method of Example 24, wherein the first item is:

[0246]

[0247] Where N is the number of representations in the modified input sequence, d represents the prediction duration, d* represents the true duration, and ||·||2 is the L2 loss.

[0248] Example 26 is a method of any one of Examples 24 or 25, wherein the second example is:

[0249]

[0250] Where T is the number of intermediate time steps, K is the number of frequencies represented in the Mel spectrogram, and y t The generated Mel spectrum, y t* represents the Mel spectrum of the true value, ||·||1 is the L1 loss, and ||·||2 is the L2 loss.

[0251] Example 27 is a method of any one of Examples 24 or 25, wherein the second example represents an error in either of the following: i) the generated Mel spectrogram and ii) the initial Mel spectrogram generated by the third neural network, wherein the Mel spectrogram is generated by processing the initial Mel spectrogram using a fourth subnet of the third neural network.

[0252] Example 28 is the method of Example 27, wherein the second item is:

[0253]

[0254] Where T is the number of intermediate time steps, K is the number of frequencies represented in the Mel spectrum, and y t ' is the initial Mel spectrum, y t The generated Mel spectrum, y t * is the true value Mel spectrum, ‖·‖1 is the L1 loss, and ‖·‖2 is the L2 loss.

[0255] Example 29 is a method of any one of Examples 23 to 28, wherein the training includes a teacher forcing the use of the true value duration for each representation in the modified input sequence.

[0256] Example 30 is a method of Example 23, wherein the training includes training the neural network without targeting any true value duration of the representation in the modified input sequence.

[0257] Example 31 is the method of Example 30, wherein the training includes:

[0258] At each of the multiple training input time steps, obtain the training input text sequence including the corresponding training text elements;

[0259] The training input text sequence is processed using a first subnet of the first neural network to generate an embedding of the training input text sequence;

[0260] Obtain the Mel-spectrum map of the true values ​​corresponding to the training input text sequence;

[0261] The real-value Mel spectrogram is processed using a second subnetwork of the first neural network to generate an embedding of the real-value Mel spectrogram;

[0262] Combining i) the embedding of the training input text sequence and ii) the embedding of the ground truth Mel-spectrum to generate a modified training input sequence, wherein for each of the plurality of training input time steps, the modified training input sequence includes a representation of the corresponding training text element in the training input text sequence; and

[0263] The second neural network is used to process the training modified input sequence to generate a predicted duration for each representation in the training modified input sequence.

[0264] Example 32 is a method of Example 31, wherein combining i) the embedding of the training input text sequence and ii) the embedding of the real value Mel spectrogram includes: processing i) the embedding of the training input text sequence and ii) the embedding of the real value Mel spectrogram using a third subnet of the first neural network.

[0265] Example 33 is a method of Example 32, wherein processing i) the embedding of the training input text sequence and ii) the embedding of the ground truth Mel-spectrum using the third subnet of the first neural network includes:

[0266] The embeddings of i) the training input text sequence and ii) the ground truth Mel spectrogram are aligned using one or more attention neural network layers;

[0267] The alignment embedding of the ground truth Mel spectrogram is processed using a variational autoencoder to generate alignment latent features of the ground truth Mel spectrogram; and

[0268] Connect i) the embedding of the training input text sequence and ii) the aligned latent features of the ground truth Mel spectrogram.

[0269] Example 34 is the method of Example 33, wherein the variational autoencoder is a conditional variational autoencoder conditioned on the embedding of the training input text sequence.

[0270] Example 35 is a method of any one of Examples 31 to 34, wherein, during inference, generating the modified input sequence includes:

[0271] The input text sequence is processed using the first subnetwork of the first neural network to generate an embedding of the input text sequence; and

[0272] The modified input sequence is generated by combining i) the embedding of the input text sequence and ii) the pattern of the prior distribution of the Mel spectrogram.

[0273] Example 36 is the method of Example 35, wherein the mode of the prior distribution of the Mel spectrogram is a zero vector.

[0274] Example 37 is a method of any one of Examples 31 to 36, wherein the neural network is trained using a loss term including one or more of the following:

[0275] The first item characterizes the errors in the generated Mel spectrogram;

[0276] The second item characterizes the error in the total prediction duration of the output audio sequence; or

[0277] The third term characterizes the KL divergence loss of the variational autoencoder of the third subnet of the first neural network.

[0278] Example 38 is the method of Example 37, wherein the first item is:

[0279]

[0280] Where T is the number of intermediate time steps, K is the number of frequencies represented in the Mel spectrogram, and y t The generated Mel spectrum, y t * is the Mel spectrum of the true value, ‖·‖1 is the L1 loss, and ‖·‖2 is the L2 loss.

[0281] Example 39 is the method of Example 37, wherein the first item characterizes an error in either of the following: i) the generated Mel spectrogram and ii) the initial Mel spectrogram generated by the third neural network, wherein the Mel spectrogram is generated by processing the initial Mel spectrogram using a fourth subnet of the third neural network.

[0282] Example 40 is the method of Example 39, wherein the first item is:

[0283]

[0284] Where T is the number of intermediate time steps, K is the number of frequencies represented in the Mel spectrogram, and y t ' is the initial Mel spectrum, y t The generated Mel spectrum, y t * is the Mel spectrum of the true value, ‖·‖1 is the L1 loss, and ‖·‖2 is the L2 loss.

[0285] Example 41 is a method of any one of Examples 37 to 40, wherein the second example is:

[0286]

[0287] Where N is the number of representations in the modified input sequence, T is the number of intermediate time steps, and d i Let represent the predicted duration of i in the modified input sequence, and let ||·|2 be the L2 loss.

[0288] Example 42 is a method of any one of Examples 1 to 41, wherein the plurality of intermediate time steps are the same as the plurality of output time steps.

[0289] Example 43 is a method of any one of Examples 1 to 42, wherein:

[0290] Each text element represents a character;

[0291] Each text element represents a phoneme;

[0292] Each text element represents a word; or

[0293] Each text element includes multiple linguistic features derived from the input text.

[0294] Example 44 is a system comprising one or more computers and one or more storage devices storing instructions, wherein when the instructions are executed by the one or more computers, the one or more computers perform the method of any one of Examples 1 to 43.

[0295] Example 45 is one or more non-transitory computer storage media encoded with a computer program, the program including instructions that, when executed by a data processing device, are operable to cause the data processing device to perform the method of any one of Examples 1 to 43.

[0296] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described herein in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from said combination may be removed from the claimed combination, and the claimed combination may be for sub-combinations or variations thereof.

[0297] Similarly, although operations are depicted in the accompanying drawings and recited in the claims in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or to perform all of the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0298] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for generating an output audio sequence from an input text sequence, wherein, The input text sequence includes corresponding text elements at each of multiple input time steps, and the output audio sequence includes corresponding audio samples at each of multiple output time steps. The method includes: The input text sequence is processed using a first neural network to generate a modified input sequence, wherein for each of the plurality of input time steps, the modified input sequence includes a representation of the corresponding text element in the input text sequence; The modified input sequence is processed using a second neural network to generate a predicted duration for the corresponding text element in the output audio sequence for each input time step; Upsampling the modified input sequence based on the predicted duration to generate an intermediate sequence, the intermediate sequence including a corresponding intermediate element at each of a plurality of intermediate time steps, wherein upsampling the modified input sequence includes: For each representation in the modified input sequence, using the predicted duration of the corresponding text element in the output audio sequence, parameters of the distribution of the representation are determined based on the predicted duration of the corresponding text element. The distribution of the representation assigns a corresponding value to each intermediate element, the corresponding value modeling the impact of the representation on the intermediate element based on the predicted duration of the corresponding text element. The distribution of the representation is a Gaussian distribution, and the center of the Gaussian distribution corresponds to the center of the predicted duration of the representation. Each intermediate element of the intermediate sequence is generated based on the distribution of the representations in the modified input sequence, and generating each intermediate element of the intermediate sequence includes, for each specific intermediate element: Based on the value assigned to the specific intermediate element in the distribution generated for each representation, the corresponding weight of the representation is determined; and The specific intermediate element is generated by determining a weighted sum of representations in the modified input sequence, wherein the representations are weighted according to the corresponding weights of each representation; and The intermediate sequence is used to generate the output audio sequence.

2. The method according to claim 1, wherein, Processing the input text sequence using the first neural network to generate the modified input sequence includes: For each text element in the input text sequence, obtain the predetermined embedding of the text element; and The first neural network is used to process the predetermined embeddings of the plurality of text elements to generate the modified input sequence.

3. The method according to claim 1, wherein, The first neural network includes one or more of the following: One or more convolutional neural network layers; One or more unidirectional LSTM neural network layers; or One or more bidirectional LSTM neural network layers.

4. The method according to claim 1, wherein, Processing the input text sequence using a first neural network to generate the modified input sequence includes: The input text sequence is processed using the first neural network to generate an initial modified input sequence, the initial modified input sequence including a corresponding initial representation at each of the plurality of input time steps; and Generating the modified input sequence from the initial modified input sequence includes: for each initial representation in the initial modified input sequence, combining the initial representation with an identifier of the class to which the output audio sequence should belong.

5. The method according to claim 4, wherein, Each class corresponds to a speaker whose output audio sequence should sound like.

6. The method according to claim 1, wherein, The second neural network includes one or more of the following: One or more convolutional neural network layers; One or more unidirectional LSTM neural network layers; One or more bidirectional LSTM neural network layers; as well as An output projection layer is configured to receive a corresponding layer input for each representation in the modified input sequence and to generate the predicted duration for each representation in the modified input sequence.

7. The method according to claim 6, wherein, The center of the Gaussian distribution in the specific representation is: ， in, c i It is the center of the Gaussian distribution in the specific representation. d i It is the predicted duration of the specific representation, and each d j It is the predicted duration of the corresponding representation in the modified input sequence preceding the specific representation.

8. The method according to claim 6, wherein, The variance of the Gaussian distribution for each corresponding representation is generated by processing the modified input sequence using a fourth neural network.

9. The method according to claim 8, wherein, Processing the modified input sequence using the fourth neural network includes: For each representation in the modified input sequence, the representation is combined with the predicted duration of the representation to generate a corresponding combined representation; and The combined representations are processed using the fourth neural network to generate the corresponding variance of the Gaussian distribution for each representation.

10. The method according to claim 8, wherein, The fourth neural network includes one or more of the following: One or more convolutional neural network layers; One or more unidirectional LSTM neural network layers; One or more bidirectional LSTM neural network layers; as well as An output projection layer is configured to receive a corresponding layer input for each representation in the modified input sequence and to generate the corresponding variance of the Gaussian distribution for each representation in the modified input sequence.

11. The method according to claim 1, wherein, Upsampling the modified input sequence to generate an intermediate sequence includes: The modified input sequence is upsampled to generate an upsampled sequence including a corresponding upsampled representation at each of the plurality of intermediate time steps; and Generating the intermediate sequence from the upsampled sequence includes: for each upsampled representation in the upsampled text sequence, combining the upsampled representation with the position embedding of the upsampled representation.

12. The method according to claim 11, wherein, The position embedding of the upsampled representation identifies the position of the upsampled representation in a subsequence of the upsampled representation corresponding to the same representation in the modified input sequence.

13. The method according to claim 1, wherein, The first neural network and the second neural network have been trained concurrently.

14. The method according to claim 1, wherein, Generating the output audio sequence using the intermediate sequence includes: The intermediate sequences are processed using a third neural network to generate Mel spectrograms, the Mel spectrograms comprising corresponding spectrogram frames at each of the plurality of intermediate time steps; and The Mel spectrogram is processed to generate the output audio sequence.

15. The method according to claim 14, wherein, Processing the intermediate sequence using a third neural network to generate a Mel spectrogram includes: At the first processing time step in the processing time step sequence, the third neural network processes the first intermediate element in the intermediate sequence to generate the first frame of the Mel spectrogram; and At each subsequent processing time step in the sequence of processing time steps, a third neural network is used to process i) subsequent intermediate elements in the intermediate sequence and ii) previous frames of the Mel spectrogram generated in the previous processing time step to generate subsequent frames of the Mel spectrogram.

16. The method according to claim 15, wherein, The third neural network includes one or more of the following: One or more convolutional neural network layers; One or more unidirectional LSTM neural network layers; One or more bidirectional LSTM neural network layers; as well as An output projection layer is configured to receive the input from the receiving layer and generate the subsequent frames of the Mel spectrogram.

17. The method according to claim 15, wherein, Processing i) the subsequent intermediate elements in the intermediate sequence and ii) the previous frames of the Mel spectrogram using the third neural network includes: The previous frame is processed using one or more fully connected neural network layers to generate an embedding of the previous frame; and The third neural network is used to process i) the subsequent intermediate elements in the intermediate sequence and ii) the embeddings of the previous frames to generate the subsequent frames of the Mel spectrogram.

18. The method according to claim 17, wherein, Processing i) the subsequent intermediate elements in the intermediate sequence and ii) the embeddings of the previous frames using the third neural network includes: Connect i) the subsequent intermediate elements in the intermediate sequence and ii) the embedding of the previous frame to generate a first linked representation; The first connection representation is processed using the first subnet of the third neural network to generate an embedding of the first connection representation; Connect i) the subsequent intermediate elements in the intermediate sequence and ii) the embedding of the first link representation to generate a second link representation; and The second connection representation is processed using a second subnet of the third neural network to generate the subsequent frames of the Mel spectrogram.

19. The method of claim 14, wherein, Processing the intermediate sequence using the third neural network to generate the Mel spectrogram includes: The intermediate sequence is processed using a third subnetwork of the third neural network to generate an initial Mel spectrogram; and The initial Mel spectrogram is processed using a fourth subnet of the third neural network to generate the Mel spectrogram.

20. The method according to claim 19, wherein, The fourth subnetwork of the third neural network includes one or more convolutional neural network layers.

21. The method according to claim 14, wherein, The first neural network, the second neural network, and the third neural network have been trained concurrently.

22. The method according to claim 21, wherein, The neural network is trained using a loss term that includes one or more of the following: The first term characterizes the error in the predicted duration represented by the modified input sequence; as well as The second term characterizes errors in the generated Mel spectrogram.

23. The method according to claim 22, wherein, The first item is: ， in, N It is the number of representations in the modified input sequence. d Indicates the predicted duration. d Indicates the duration of the actual value, and yes L 2 loss.

24. The method according to claim 22, wherein, The second item is: ， in, T It is the number of intermediate time steps. K It is the number of frequencies represented in the Mel spectrum. y t It is the generated Mel spectrum. y t It is the Mel spectrum of the true value. yes L 1 Loss, and yes L 2 loss.

25. The method according to claim 22, wherein, The second item characterizes an error in either of the following: i) the generated Mel spectrogram and ii) the initial Mel spectrogram generated by the third neural network, wherein the Mel spectrogram is generated by processing the initial Mel spectrogram using a fourth subnetwork of the third neural network.

26. The method of claim 25, wherein, The second item is: ， in, T It is the number of intermediate time steps. K It is the number of frequencies represented in the Mel spectrum. y t ’ This is the initial Mel spectrum. y t It is the generated Mel spectrum. yt It is the true value Mel spectrum. yes L 1 Loss, and yes L 2 loss.

27. The method according to claim 21, wherein, The training includes: the teacher forcing the use of the true value duration for each representation in the modified input sequence.

28. The method according to claim 21, wherein, The training includes training the neural network without any real-value duration for the representation in the modified input sequence.

29. The method according to claim 28, wherein, The training includes: At each of the multiple training input time steps, obtain the training input text sequence including the corresponding training text elements; The training input text sequence is processed using a first subnet of the first neural network to generate an embedding of the training input text sequence; Obtain the Mel-spectrum map of the true values ​​corresponding to the training input text sequence; The real-value Mel spectrogram is processed using a second subnetwork of the first neural network to generate an embedding of the real-value Mel spectrogram; Combining i) the embedding of the training input text sequence and ii) the embedding of the ground truth Mel-spectrum to generate a training modified input sequence, wherein for each of the plurality of training input time steps, the training modified input sequence includes a representation of the corresponding training text element in the training input text sequence; and The second neural network is used to process the training modified input sequence to generate a predicted duration for each representation in the training modified input sequence.

30. The method according to claim 29, wherein, Combining i) the embedding of the training input text sequence and ii) the embedding of the ground truth Mel spectrogram includes: using a third subnetwork of the first neural network to process i) the embedding of the training input text sequence and ii) the embedding of the ground truth Mel spectrogram.

31. The method according to claim 30, wherein, Using the third subnetwork of the first neural network to process i) the embedding of the training input text sequence and ii) the embedding of the ground truth Mel-spectrum image includes: One or more attention neural network layers are used to align i) the embedding of the training input text sequence and ii) the embedding of the ground truth Mel spectrogram; The aligned embeddings of the ground truth Mel spectrogram are processed using a variational autoencoder to generate aligned latent features of the ground truth Mel spectrogram; and Connect i) the embedding of the training input text sequence and ii) the aligned latent features of the ground truth Mel spectrogram.

32. The method according to claim 31, wherein, The variational autoencoder is a conditional variational autoencoder conditioned on the embedding of the training input text sequence.

33. The method according to claim 29, wherein, During inference, generating the modified input sequence includes: The input text sequence is processed using the first subnetwork of the first neural network to generate an embedding of the input text sequence; and The modified input sequence is generated by combining i) the embedding of the input text sequence and ii) the pattern of the prior distribution of the Mel spectrogram.

34. The method according to claim 33, wherein, The pattern of the prior distribution of the Mel spectrogram is a zero vector.

35. The method according to claim 29, wherein, The neural network is trained using a loss term that includes one or more of the following: The first term characterizes errors in the generated Mel spectrogram; The second term characterizes the error in the total prediction duration of the output audio sequence; as well as The third term characterizes the KL divergence loss of the variational autoencoder of the third subnet of the first neural network.

36. The method according to claim 35, wherein, The first item is: in, T It is the number of intermediate time steps. K It is the number of frequencies represented in the Mel spectrum. y t It is the generated Mel spectrum. y t It is the Mel spectrum of the true value. yes L 1 Loss, and yes L 2 loss.

37. The method of claim 35, wherein, The first item characterizes an error in either of the following: i) the generated Mel spectrogram and ii) the initial Mel spectrogram generated by the third neural network, wherein the Mel spectrogram is generated by processing the initial Mel spectrogram using a fourth subnetwork of the third neural network.

38. The method according to claim 37, wherein, The first item is: ， in, T It is the number of intermediate time steps. K It is the number of frequencies represented in the Mel spectrum. y t ’ This is the initial Mel spectrum. y t It is the generated Mel spectrum. y t It is the Mel spectrum of the true value. yes L 1 Loss, and yes L 2 loss.

39. The method according to claim 35, wherein, The second item is: ， in, N It is the number of representations in the modified input sequence. T It is the number of intermediate time steps. d i The modified input sequence represents i The predicted duration, and yes L 2 loss.

40. The method according to any one of claims 1 to 39, wherein, The plurality of intermediate time steps are the same as the plurality of output time steps.

41. The method according to any one of claims 1 to 39, wherein: Each text element represents a character; Each text element represents a phoneme; Each text element represents a word; or Each text element includes multiple linguistic features derived from the input text.

42. A system comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to perform the operation of the method according to any one of claims 1 to 41.

43. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of any one of claims 1 to 41.

Images