Parallel WaveGAN: A Fast and High-Quality Model to Generate Raw Audio

Abstract

A method (600) for training a non-autoregressive TTS model (300) includes receiving a reference audio signal (201) and a corresponding input text sequence (206). The method further includes encoding the reference audio signal into a variational embedding (220) that separates style / prosodic information from the reference audio signal, and encoding the input text sequence into an encoded text sequence (219). The method also includes predicting phoneme duration for each phoneme in the input text sequence (240) and determining a phoneme duration loss (244). The method further includes generating one or more predicted Mel frequency spectrogram sequences (302) for the input text sequence and determining a final spectrogram loss (280) based on the predicted Mel frequency spectrogram sequences and the reference Mel frequency spectrogram sequence (202). The method also includes training the TTS model based on the final spectrogram loss and the corresponding phoneme duration loss.

Description

Technical Field

[0001] This disclosure relates to parallelized tacotron non-autoregressive and controllable text-to-speech (TTS). Background Technology

[0002] Text-to-speech (TTS) systems read digital text aloud to users and are becoming increasingly popular on mobile devices. Some TTS models aim to synthesize various aspects of speech, such as speaking style, to produce natural, human-like speech. Synthesis in TTS models is a one-to-many mapping problem because there may be multiple possible speech outputs for different primitives of the text input. Many TTS systems utilize autoregressive models that predict current values ​​based on previous values. While autoregressive TTS models are capable of synthesizing text and generating highly natural speech outputs, the hundreds of computations required reduce efficiency during inference. Summary of the Invention

[0003] One aspect of this disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for training a non-autoregressive text-to-speech (TTS) model. The operations include receiving training data comprising a reference audio signal and a corresponding input text sequence. The reference audio signal comprises spoken utterance, and the input text sequence corresponds to a transcription of the reference audio signal. The operations also include encoding the reference audio signal into a variational embedding using a residual encoder, the variational embedding disentangles style / prosodic information from the reference audio signal, and encoding the input text sequence into an encoded text sequence using a text encoder. The operations further include using a duration decoder and predicting phoneme duration for each phoneme in the input text sequence based on the encoded text sequence and the variational embedding. The operations also include determining a phoneme duration loss based on the predicted phoneme duration and the reference phoneme duration from the reference audio signal used for each phoneme in the input text sequence. The operations further include generating one or more predicted Mel-frequency spectrogram sequences for the input text sequence as output from a spectrogram decoder comprising a stack of self-attention blocks and based on the output of the duration decoder. The operation also includes determining a final spectrogram loss based on one or more predicted Mel frequency spectrogram sequences and a reference Mel frequency spectrogram sequence sampled from a reference audio signal, and training a TTS model based on the final spectrogram loss and the corresponding phoneme duration loss determined for each phoneme in the input text sequence.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, each self-attention block in the stack of self-attention blocks includes the same lightweight convolutional (LConv) block. In other implementations, each self-attention block in the stack of self-attention blocks includes the same transformer block. The input text sequence may include words, each having one or more phonemes, silences at all word boundaries, and punctuation marks.

[0005] In some examples, the residual encoder includes a global variational autoencoder (VAE), and encoding the reference audio signal into a variational embedding includes: sampling a reference Mel frequency spectrogram sequence from the reference audio signal; and encoding the reference Mel spectrogram sequence into a variational embedding using the global VAE. In some implementations, the residual encoder includes a phoneme-level fine-grained variational autoencoder (VAE), and encoding the reference audio signal into a variational embedding includes: sampling a reference audio spectrogram sequence from the reference audio signal; aligning the reference Mel frequency spectrogram sequence with each phoneme in a phoneme sequence extracted from the input text sequence; and encoding the phoneme-level variational embedding sequence using the phoneme-level fine-grained VAE based on the alignment of the reference Mel frequency spectrogram sequence with each phoneme in the phoneme sequence.

[0006] Optionally, the residual encoder may include a stack of lightweight convolutional (LConv) blocks, wherein each LCoV block in the stack includes: a gate linear unit (GLU) layer; an LCoV layer configured to receive the output of the GLU layer; a residual connection configured to concatenate the output of the LCoV layer with the input of the GLU layer; and a final feedforward layer configured to receive the residual connection concatenating the output of the LCoV layer with the input of the GLU layer as input. The operation may also include upsampling the output of the duration decoder to multiple frames using reference phoneme durations sampled from a reference audio signal for each phoneme in the input text sequence, and obtaining a position embedding representing phoneme position information for each phoneme in the input text utterance. Here, generating one or more predicted Mel-frequency spectrogram sequences for the input text sequence is based on the position embedding and the upsampling of the duration decoder output to multiple frames.

[0007] In some examples, generating one or more predicted Mel-frequency spectrogram sequences for the input text sequence includes generating a corresponding Mel-frequency spectrogram sequence as output from each self-attention block in the stack of self-attention blocks of the spectrogram decoder. In these examples, determining the final spectrogram loss includes, for each corresponding predicted Mel-frequency spectrogram sequence, determining a corresponding spectrogram loss based on the predicted Mel-frequency spectrogram sequence and a reference Mel-frequency spectrogram sequence, and aggregating the corresponding spectrogram losses determined for the predicted Mel-frequency spectrogram sequences to generate the final spectrogram loss. In some implementations, the duration decoder includes a stack of self-attention blocks followed by two independent projections, and predicting the phoneme duration for each phoneme in the input text sequence includes: using a sigmoid activation after the first projection in the two independent projections to predict the probability of a non-zero duration for each phoneme; and using a softplus activation after the second projection in the two independent projections to predict the phoneme duration for each phoneme. This operation may further include, at each phoneme, determining whether the probability of a non-zero duration predicted for the corresponding phoneme is less than a threshold, and when the probability of a non-zero duration is less than the threshold, zeroing the phoneme duration predicted for the corresponding phoneme.

[0008] In some examples, the operation further includes concatenating the encoded text sequence, variational embeddings, and a reference speaker embedding representing the identity of a reference speaker emitting the reference audio signal, and generating the output of the duration decoder based on receiving the concatenation of the encoded text sequence, variational embeddings, and reference speaker embeddings as input. Optionally, the input text sequence may include a phoneme sequence, and encoding the input text sequence into an encoded text sequence includes: receiving the corresponding embedding for each phoneme in the phoneme sequence from a phoneme lookup table; for each phoneme in the phoneme sequence, processing the corresponding embedding using an encoder pre-net neural network of a text encoder to generate a corresponding transform embedding for that phoneme; processing the corresponding transform embedding using a stack of convolutional blocks to generate a convolutional output; and processing the convolutional output using a stack of self-attention blocks to generate the encoded text sequence.

[0009] Another aspect of this disclosure provides a system for training a non-autoregressive text-to-speech (TTS) model, the system including data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving training data comprising a reference audio signal and a corresponding input text sequence. The reference audio signal comprises spoken utterance, and the input text sequence corresponds to a transcription of the reference audio signal. The operations also include encoding the reference audio signal into a variational embedding using a residual encoder, the variational embedding separating style / prosodic information from the reference audio signal, and encoding the input text sequence into an encoded text sequence using a text encoder. The operations further include using a duration decoder and predicting phoneme duration for each phoneme in the input text sequence based on the encoded text sequence and the variational embedding. The operations also include determining a phoneme duration loss based on the predicted phoneme duration and the reference phoneme duration from the reference audio signal used for each phoneme in the input text sequence. The operations further include generating one or more predicted Mel-frequency spectrogram sequences for the input text sequence as output from a spectrogram decoder comprising a stack of self-attention blocks and based on the output of the duration decoder. The operation also includes determining a final spectrogram loss based on one or more predicted Mel frequency spectrogram sequences and a reference Mel frequency spectrogram sequence sampled from a reference audio signal, and training a TTS model based on the final spectrogram loss and the corresponding phoneme duration loss determined for each phoneme in the input text sequence.

[0010] Implementations of this disclosure may include one or more of the following optional features. In some implementations, each self-attention block in the stack of self-attention blocks includes the same lightweight convolutional (LConv) block. In other implementations, each self-attention block in the stack of self-attention blocks includes the same transformer block. The input text sequence may include words, each having one or more phonemes, silences at all word boundaries, and punctuation marks.

[0011] In some examples, the residual encoder includes a global variational autoencoder (VAE), and encoding the reference audio signal into a variational embedding includes: sampling a reference Mel frequency spectrogram sequence from the reference audio signal; and encoding the reference Mel spectrogram sequence into a variational embedding using the global VAE. In some implementations, the residual encoder includes a phoneme-level fine-grained variational autoencoder (VAE), and encoding the reference audio signal into a variational embedding includes: sampling a reference audio spectrogram sequence from the reference audio signal; aligning the reference Mel frequency spectrogram sequence with each phoneme in a phoneme sequence extracted from the input text sequence; and encoding the phoneme-level variational embedding sequence using the phoneme-level fine-grained VAE based on the alignment of the reference Mel frequency spectrogram sequence with each phoneme in the phoneme sequence.

[0012] Optionally, the residual encoder may include a stack of lightweight convolutional (LConv) blocks, wherein each LCoV block in the stack includes: a gate linear unit (GLU) layer; an LCoV layer configured to receive the output of the GLU layer; a residual connection configured to concatenate the output of the LCoV layer with the input of the GLU layer; and a final feedforward layer configured to receive the residual connection concatenating the output of the LCoV layer with the input of the GLU layer as input. The operation may also include upsampling the output of the duration decoder to multiple frames using reference phoneme durations sampled from a reference audio signal for each phoneme in the input text sequence, and obtaining a position embedding representing phoneme position information for each phoneme in the input text utterance. Here, generating one or more predicted Mel-frequency spectrogram sequences for the input text sequence is based on the position embedding and the upsampling of the duration decoder output to multiple frames.

[0013] In some examples, generating one or more predicted Mel-frequency spectrogram sequences for the input text sequence includes generating a corresponding Mel-frequency spectrogram sequence as output from each self-attention block in the stack of self-attention blocks of the spectrogram decoder. In these examples, determining the final spectrogram loss includes, for each corresponding predicted Mel-frequency spectrogram sequence, determining a corresponding spectrogram loss based on the predicted Mel-frequency spectrogram sequence and a reference Mel-frequency spectrogram sequence, and aggregating the corresponding spectrogram losses determined for the predicted Mel-frequency spectrogram sequences to generate the final spectrogram loss. In some implementations, the duration decoder includes a stack of self-attention blocks followed by two independent projections, and predicting the phoneme duration for each phoneme in the input text sequence includes: using a sigmoid activation after the first projection in the two independent projections to predict the probability of a non-zero duration for each phoneme; and using a softplus activation after the second projection in the two independent projections to predict the phoneme duration for each phoneme. This operation may further include, at each phoneme, determining whether the probability of a non-zero duration predicted for the corresponding phoneme is less than a threshold, and when the probability of a non-zero duration is less than the threshold, zeroing the predicted phoneme duration for the corresponding phoneme.

[0014] In some examples, the operation further includes concatenating the encoded text sequence, variational embeddings, and a reference speaker embedding representing the identity of a reference speaker emitting the reference audio signal, and generating the output of the duration decoder based on receiving the concatenation of the encoded text sequence, variational embeddings, and reference speaker embeddings as input. Optionally, the input text sequence may include a phoneme sequence, and encoding the input text sequence into an encoded text sequence includes: receiving the corresponding embedding for each phoneme in the phoneme sequence from a phoneme lookup table; processing the corresponding embedding for each phoneme in the phoneme sequence using an encoder pre-network neural network of a text encoder to generate a corresponding transform embedding for that phoneme; processing the corresponding transform embedding using a stack of convolutional blocks to generate a convolutional output; and processing the convolutional output using a stack of self-attention blocks to generate the encoded text sequence.

[0015] Another aspect of this disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations. The operations include receiving a text utterance to be synthesized into speech, and encoding a sequence of phonemes extracted from the text utterance into an encoded text sequence using a text encoder of a non-autoregressive TTS model. The operations also include selecting a variational embedding for the text utterance, the variational embedding specifying an expected prosody / style for synthesizing the text utterance into speech. For each phoneme in the phoneme sequence, the operations further include predicting a phoneme duration for the corresponding phoneme using a duration decoder of the non-autoregressive TTS model based on the encoded text sequence and the selected variational embedding. The operations also include generating a predicted Mel-frequency spectrogram sequence for the text utterance as output from a spectrogram decoder of the non-autoregressive TTS model, which includes a stack of self-attention blocks, based on the output of the duration decoder and the predicted phoneme duration. The predicted Mel-frequency spectrogram sequence has the expected prosody / style specified by the selected variational embedding.

[0016] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the selected variational embedding includes a previous variational embedding sampled from a residual encoder based on a variational autoencoder (VAE). The operation may also include using a synthesizer to convert a sequence of predicted Mel frequency spectrograms into synthesized speech. In some examples, the phoneme sequence in the encoded text sequence extracted from text utterance includes: receiving a corresponding embedding for each phoneme in the phoneme sequence from a phoneme lookup table; for each phoneme in the phoneme sequence, processing the corresponding embedding using an encoder pre-network neural network of a text encoder to generate a corresponding transform embedding for that phoneme; processing the corresponding transform embedding using a stack of convolutional blocks to generate a convolutional output; and processing the convolutional output using a stack of self-attention blocks to generate the encoded text sequence.

[0017] Optionally, each self-attention block in the stack of self-attention blocks of the spectrogram decoder may include one of the same lightweight convolutional (LConv) blocks or the same transformer blocks. In some implementations, the operation further includes upsampling the output of the duration decoder to multiple frames using the predicted phoneme duration and obtaining a positional embedding representing the phoneme position information of each phoneme in the phoneme sequence. Here, generating a predicted Mel-frequency spectrogram sequence for text utterance is based on the positional embedding and the upsampling of the duration decoder output to multiple frames. The operation may also include concatenating the encoded text sequence, a selected variational embedding, and a reference speaker embedding, the reference speaker embedding representing the identifier of a reference speaker associated with the selected variational embedding, and generating the duration decoder output based on the concatenation of the encoded text sequence, variational embedding, and reference speaker embedding received as input by the duration decoder.

[0018] Another aspect of the invention provides a system for executing a trained non-autoregressive text-to-speech (TTS) model, the system including data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving a text utterance to be synthesized into speech and encoding a sequence of phonemes extracted from the text utterance into an encoded text sequence using a text encoder of the non-autoregressive TTS model. The operations also include selecting a variational embedding for the text utterance, the variational embedding specifying an expected prosody / style for synthesizing the text utterance into speech. For each phoneme in the phoneme sequence, the operations further include predicting a phoneme duration for the corresponding phoneme using a duration decoder of the non-autoregressive TTS model based on the encoded text sequence and the selected variational embedding. The operations also include generating a predicted Mel-frequency spectrogram sequence for the text utterance as output from a spectrogram decoder of the non-autoregressive TTS model, which includes a stack of self-attention blocks, based on the output of the duration decoder and the predicted phoneme duration. The predicted Mel-frequency spectrogram sequence has the expected prosody / style specified by the selected variational embedding.

[0019] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the selected variational embedding includes a previous variational embedding sampled from a residual encoder based on a variational autoencoder (VAE). The operation may also include using a synthesizer to convert a sequence of predicted Mel frequency spectrograms into synthesized speech. In some examples, the phoneme sequence in the encoded text sequence extracted from text utterance includes: receiving a corresponding embedding for each phoneme in the phoneme sequence from a phoneme lookup table; for each phoneme in the phoneme sequence, processing the corresponding embedding using an encoder pre-network neural network of a text encoder to generate a corresponding transform embedding for that phoneme; processing the corresponding transform embedding using a stack of convolutional blocks to generate a convolutional output; and processing the convolutional output using a stack of self-attention blocks to generate the encoded text sequence.

[0020] Optionally, each self-attention block in the stack of self-attention blocks of the spectrogram decoder may include one of the same lightweight convolutional (LConv) blocks or the same transformer blocks. In some implementations, the operation further includes upsampling the output of the duration decoder to multiple frames using the predicted phoneme duration and obtaining a positional embedding representing the phoneme position information of each phoneme in the phoneme sequence. Here, generating a predicted Mel-frequency spectrogram sequence for text utterance is based on the positional embedding and the upsampling of the duration decoder output to multiple frames. The operation may also include concatenating the encoded text sequence, a selected variational embedding, and a reference speaker embedding, the reference speaker embedding representing the identity of a reference speaker associated with the selected variational embedding, and generating the duration decoder output based on the concatenation of the encoded text sequence, variational embedding, and reference speaker embedding received as input by the duration decoder.

[0021] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will be apparent from the description and drawings, as well as from the claims. Attached Figure Description

[0022] Figure 1 It is an example system for training deep neural networks to provide non-autoregressive TTS models for predicting spectrograms of text discourse.

[0023] Figure 2 This is a schematic diagram of a non-autoregressive neural network used to train a non-autoregressive TTS model.

[0024] Figure 3 Is executed by Figure 2 A schematic diagram of a trained non-autoregressive TTS model trained by a non-autoregressive deep neural network.

[0025] Figure 4 This is a schematic diagram of an example convolutional block.

[0026] Figure 5 This is a schematic diagram of an example lightweight convolutional block.

[0027] Figure 6 This is a flowchart illustrating an exemplary arrangement of operations for a computer-implemented method for training a non-autoregressive TTS model.

[0028] Figure 7 This is a flowchart illustrating an exemplary arrangement of operations for a computer-implemented method of executing a trained non-autoregressive TTS model.

[0029] Figure 8 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described in this paper.

[0030] In different figures, the same reference numerals indicate the same elements. Detailed Implementation

[0031] Synthesizing real human speech is an unpredictable problem because the same text input has an infinite number of plausible spoken implementations. While end-to-end neural network-based approaches are advancing to match human performance for short, assistant-like utterances, neural network models are sometimes perceived as less interpretable or more uncontrollable than more conventional models that involve multiple processing steps, each operating on fine-grained linguistic or speech representations. Sources of variability in speech include prosodic features such as intonation, stress, rhythm, and style, as well as speaker and channel features. The prosodic features of spoken utterance convey linguistic, semantic, and emotional meaning more effectively than the meaning present in lexical representations (e.g., transcription of spoken utterance).

[0032] Neural networks are machine learning models that employ one or more nonlinear units to predict outputs from received inputs. For example, neural network-based end-to-end text-to-speech (TTS) models can transform input text into output speech. Neural TTS models offer the potential to robustly synthesize speech by predicting linguistic factors that correspond to prosody not provided by the text input. As a result, many applications, such as audiobook narration, news reading, voice design software, and conversational assistants, are able to produce realistic, synthesized speech that is not monotonous utterances.

[0033] Many neural end-to-end TTS models utilize autoregressive models that predict current values ​​based on previous values. For example, many autoregressive models are based on recurrent neural networks, which use some or all of the network's internal states from previous time steps when computing the output at the current time step. An example of a recurrent neural network is an LSTM neural network that includes one or more Long Short-Term (LSTM) memory blocks. Each LSTM memory block can include one or more cells, each including an input gate, a forget gate, and an output gate that allows the cell to store previous states used for that cell, such as those used to generate current activations or other components fed into the LSTM neural network.

[0034] While autoregressive TTS models can synthesize text and generate highly natural speech output, their architecture—a series of unidirectional LSTM-based decoder blocks with soft attention—inherently results in lower training and inference efficiency compared to fully feedforward architectures when implemented on modern parallel hardware. Furthermore, because autoregressive models are trained by a teacher-forced application of ground-real labels for each time step, they are also prone to discrepancies between training and applying the trained model during inference. Together with the soft attention mechanism, these discrepancies can lead to synthesized speech output with reduced quality, such as synthesized speech exhibiting robustness errors (e.g., babbling, early truncation, word repetition, and word skipping). The degradation in synthesized speech quality in autoregressive TTS models can be further exacerbated by increasing the size of the synthesized text.

[0035] To mitigate the aforementioned drawbacks of autoregressive TTS models, the implementation described in this paper targets non-autoregressive neural TTS models enhanced with a residual encoder based on a variational autoencoder (VAE). As will become apparent, a VAE-based residual encoder can separate latent representations / states from a reference audio signal conveying residual information (such as style / prosodic information) that cannot be represented by the input text to be synthesized (e.g., a phoneme sequence) or the speaker identifier (ID) of the speaker uttering the reference audio signal. In other words, the latent representation enables the output synthesized speech produced by the TTS model to utter speech as if it were the reference audio signal input to the residual encoder.

[0036] A non-autoregressive neural TTS model enhanced with a VAE-based residual encoder provides a controllable model for predicting Mel-frequency spectrogram information (e.g., predicting Mel-frequency spectrogram sequences) for input text utterances, while effectively controlling the prosody / style represented in the Mel-frequency spectrogram information. For example, using a selected variational embedding learned by the VAE-based residual encoder to represent the expected prosody / style for synthesizing text utterances into speech expressions, the spectrogram decoder of the TTS model can predict a Mel-frequency spectrogram for the text utterances and provide this Mel-frequency spectrogram as input to a synthesizer (e.g., a waveform synthesizer or vocoder network) for conversion into a time-domain audio waveform indicating synthesized speech with the expected prosody / style. As will become apparent, the non-autoregressive TTS model is trained only on sample input text sequences of human speech and corresponding reference Mel-frequency spectrogram sequences, enabling the trained TTS model to convert input text utterances into Mel-frequency spectrogram sequences with the expected prosody / style conveyed by the learned previously learned variational embeddings.

[0037] Figure 1 An example system 100 is shown for training a deep neural network 200, which is augmented with a VAE-based residual encoder 180 to provide a non-autoregressive neural TTS model (or simply "TTS model") 300, and for using the TTS model 300 to predict spectrograms (i.e., Mel frequency spectrogram sequences) 302 for text utterances 320. System 100 includes a computing system 120 having data processing hardware 122 and memory hardware 124, the memory hardware 124 communicating with the data processing hardware 122 and storing instructions that cause the data processing hardware 122 to perform operations. In some embodiments, the computing system 120 (e.g., data processing hardware 122) or a user computing device 10 executing the trained TTS model 300 provides the spectrogram 302 predicted by the TTS model 300 from the input text utterances 320 to a synthesizer 155 for conversion into a time-domain audio waveform indicative of synthesized speech 152, which can be audibly output as a spoken representation of the input text utterances 320. Time-domain audio waveforms include audio waveforms that define the amplitude of an audio signal as it changes over time. Synthesizer 155 can be trained individually and tuned on a Mel frequency spectrogram for conversion into time-domain audio waveforms.

[0038] The Mel frequency spectrogram comprises a frequency domain representation of sound. The Mel frequency spectrogram emphasizes lower frequencies crucial for speech intelligibility while not emphasizing higher frequencies dominated by friction and other noise bursts, and generally does not require high-fidelity modeling. Synthesizer 155 may include a vocoder neural network that can include any network configured to receive the Mel frequency spectrogram and generate audio output samples (e.g., time-domain audio waveforms) based on it. For example, vocoder network 155 can be based on a parallelized feedforward neural network described in van den Oord, Parallel Wavenet: Fast High-Fidelity Speech Synthesis, available at https: / / arxiv.org / pdf / 1711.10433.pdf and incorporated herein by reference. Alternatively, vocoder network 155 can be an autoregressive neural network. Synthesizer 155 may include a waveform synthesizer, such as a Griffin-Lim synthesizer or a waveform inverter with a trainable spectrogram. The choice of synthesizer 155 has no effect on the prosody / style of the synthesized speech 152, and in practice, only affects the audio fidelity of the synthesized speech 152.

[0039] Since the input text utterance 320 cannot convey context, semantics, and pragmatics to guide the desired prosody / style of the synthesized speech 152, the TTS model 300 can apply variational embedding 220 as a latent variable specifying the desired prosody / style to predict the Mel frequency spectrogram 302 for the text utterance 320 conveying the desired prosody / style specified by the variational embedding 220. In some examples, the computing system 120 implements the TTS model 300. Here, a user can access the TTS model 300 through the user computing device 10 and provide the TTS model 300 with input text utterance 320 to synthesize the expressed speech 152 with the desired prosody / style specified by the variational embedding 220. The variational embedding 220 can be selected by the user and corresponds to a previous variational embedding 220 sampled from the residual encoder 180. The variational embedding 220 can be a variational embedding 220 for each speaker selected by the user by providing a speaker identifier (ID) that identifies the speaker speaking in the desired prosody / style (i.e., through an interface implemented on the user computing device 10). Here, each speaker ID can be mapped to a corresponding variable embedding 220 for each speaker previously learned by the residual encoder 180. Additionally or alternatively, the user can provide input specifying a particular vertical associated with a corresponding prosody / style. Here, different verticals (e.g., news, sports, etc.) can each be mapped to a corresponding variational embedding 220 previously learned by the corresponding prosody / style residual encoder 180 that conveys the prosody / style associated with that vertical. In these examples, the synthesizer 155 can reside on either the computing system 120 or the user computing device 10. When the synthesizer 155 resides on the computing system 120, the computing system 120 can transmit a time-domain audio waveform representing the synthesized speech 152 to the user computing device 10 for audible playback. In other examples, the user computing device 10 implements the TTS model 300. The computing system may include a distributed system (e.g., a cloud computing environment).

[0040] In some implementations, the deep neural network 200 is trained on a large set of reference audio signals 201. Each reference audio signal 201 may include spoken utterances of human speech recorded by a microphone and having a prosodic / style representation. During training, the deep neural network 200 may receive multiple reference audio signals 201 for the same spoken utterance, but with varying prosodic / style (i.e., capable of uttering the same utterance in a variety of different ways). Here, the reference audio signals 201 are of variable length, such that the duration of the spoken utterances varies even if the content is the same. The deep neural network 200 may also receive multiple sets of reference audio signals 201, each set including reference audio signals 201 for utterances with similar prosodic / style spoken by the same corresponding speaker but conveying different linguistic content. The deep neural network 200, enhanced with a VAE-based residual encoder 180, is configured to encode / compress the prosodic / style representation associated with each reference audio signal 201 into a corresponding variational embedding 220. The variational embedding 220 may include a fixed-length variational embedding 220. The deep neural network 200 can embed each variational embedding 220 along with the corresponding speaker embedding y s Together they are stored in storage device 185 (e.g., on memory hardware 124 of computing system 120), and the speaker is embedded in y s The speaker identity 205 represents the reference speaker who emits the reference audio signal 201 associated with variational embedding 220. Figure 2 Variational embedding 220 can be a variational embedding for each speaker, which includes an aggregation (e.g., averaging) of multiple variational embeddings 220 encoded by residual encoder 180 from a reference audio signal 201 spoken by the same speaker.

[0041] During inference, computing system 120 or user computing device 10 can use trained TTS model 300 to predict Mel frequency spectrogram sequence 302 for text utterance 320. TTS model 300 can select variational embeddings 220 representing the expected prosody / style of text utterance 320 from storage device 185. Here, variational embedding 220 may correspond to a previous variational embedding 220 sampled from VAE-based residual encoder 180. TTS model 300 can use the selected variational embedding 220 to predict Mel frequency spectrogram sequence 302 for text utterance 320. In the example shown, synthesizer 155 uses the predicted Mel frequency spectrogram sequence 302 to produce synthesized speech 152 with the expected prosody / style specified by variational embedding 220.

[0042] During inference, computing system 120 or user computing device 10 can use trained TTS model 300 to predict Mel frequency spectrogram sequence 302 for text utterance 320. TTS model 300 can select from storage device 185 a variational embedding 220 representing the expected prosody / style for text utterance 320. Here, variational embedding 220 may correspond to a previous variational embedding 220 sampled from VAE-based residual encoder 180. TTS model 300 can use the selected variational embedding 220 to predict Mel frequency spectrogram sequence 302 for text utterance 320. In the example shown, synthesizer 155 uses the predicted Mel frequency spectrogram sequence 302 to produce synthesized speech 152 with the expected prosody / style specified by variational embedding 220.

[0043] In a non-limiting example, an individual can train a deep neural network 200 to learn per-speaker variational embeddings that convey a prosodic / style representation associated with a particular speaker. For example, Brian McCullough, the host of the Techmeme Ride Home podcast, can train the deep neural network 200 on a reference audio signal 201 that includes previous episodes of the podcast, along with an input text sequence 206 corresponding to the transcription of the reference audio signal 201. During training, a VAE-based residual encoder 180 can learn per-speaker variational embeddings 220 representing the prosodic / style of Brian narrating the Ride Home podcast. Brian can then apply these per-speaker variational embeddings 220 to a trained TTS model 300 (executed on computing system 120 or user computing device 10) to predict a Mel frequency spectrogram sequence 302 for a text discourse 320 corresponding to a transcription of a new episode of the Ride Home podcast. The predicted mermaid frequency spectrogram sequence 302 can be provided as input to synthesizer 155 to generate synthesized speech 152 with Brian's unique prosody / style, as specified by his per-speaker variational embedding 220. That is, the resulting synthesized speech 152 can sound exactly like Brian's voice and possess the prosody / style of Brian used to narrate episodes of the Ride Home podcast. Therefore, to broadcast a new episode, Brian simply provides a transcription of the episode and uses a trained TTS model 300 to generate synthesized speech 302, which can then be streamed to loyal listeners of the Ride Home podcast (also known as the MutantPodcast Army).

[0044] Figure 2 A non-autoregressive neural network for training a non-autoregressive TTS model 300 is shown (e.g., Figure 1The deep neural network 200 includes a VAE-based residual encoder 180, a text encoder 210, a duration decoder 230, and a spectrogram decoder 250. The deep neural network 200 can be trained on training data comprising multiple reference audio signals 201 and corresponding input text sequences 206. Each reference audio signal 201 comprises spoken utterances of human speech, and the corresponding input text sequence 206 corresponds to a transcription of the reference audio signal 201. In the example shown, the VAE-based residual encoder 180 is configured to encode the reference audio signals 201 into variational embeddings (z) 220. Specifically, the residual encoder 180 receives a reference Mel-frequency spectrogram sequence 202 sampled from the reference audio signals 201 and encodes the reference Mel-frequency spectrogram sequence 202 into a variational embedding 220, whereby the variational embedding 220 separates style / prosodic information from the reference audio signals 201 corresponding to spoken utterances of human speech. Thus, variational embedding 220 corresponds to the latent states of the reference speaker, such as influences and intentions, which contribute to the reference speaker's prosody, mood, and / or speaking style. As used herein, variational embedding 220 includes both style information and prosodic information. In some examples, variational embedding 220 includes a digital vector with a capacity represented by multiple bits in variational embedding 220. The reference Mel-frequency spectrogram sequence 202 sampled from the reference audio signal 201 may have a length L. R and dimension D R As used herein, the reference Mel frequency spectrogram sequence 202 comprises multiple fixed-length reference Mel frequency spectrogram frames sampled / extracted from the reference audio signal 201. Each reference Mel frequency spectrogram frame may include a duration of 5 milliseconds.

[0045] The VAE-based residual encoder 180 corresponds to a posterior network that implements unsupervised learning of the latent representation of speaking style (i.e., variational embedding (z)). Compared to heuristic-based systems, learning variational embeddings using a VAE network provides advantageous properties for simplifying the separation, scaling, and combination of style control. In some implementations, and as... Figure 2 As shown, the VAE-based residual encoder 180 includes a global VAE network that encodes a global variational embedding 220 from the entire reference Mel frequency spectrogram sequence 202. Here, the global VAE network 400a includes two stacks 410, 420 of lightweight convolutional (LConv) blocks, each stack having multi-head attention. Figure 5A schematic diagram 500 of an example LCoV block is shown, comprising a gate linear unit (GLU) layer 502, an LCoV layer 504 configured to receive the output of the GLU layer 502, and a feedforward (FF) layer 506. The example LCoV block 500 also includes a first residual connection 508 (e.g., a first cascader 508) configured to cascade the output of the LCoV layer 504 with the input of the GLU layer 502. The FF layer 506 is configured to receive the first residual connection 508, which cascades the output of the LCoV layer 504 with the input of the GLU layer 502, as input. The example LCoV block 500 also includes a second residual connection 510 (e.g., a second cascader 510) configured to cascade the output of the FF layer 506 with the first residual connection 508. The example LCoV block 500 can perform FF blending in the FF layer 506 using the structure ReLU(W1X+b1)W2+b2, where W1 increases the dimension by a factor of 4. Each LConv block in the first stack 410 and the second stack 420 of the global VAE network 180 may include eight (8) heads. In some examples, the first stack 410 of the LConv blocks includes three (3) 17×1 LConv blocks, and the second stack 420 of the LConv blocks following the first stack 410 includes five 17×1 LConv blocks interleaved with 3×1 convolutions. This configuration of the two stacks 410, 420 of the LConv blocks 410, 420 allows the global VAE network 180 to continuously downsample the latent representation before applying global average pooling to obtain the final global variational embedding 220. The projection layer 430 may project the dimensions of the global variational embedding 220 output from the second stack 420 of the LConv blocks. For example, the global variational embedding 220 output from the second stack 420 of the LConv blocks may have eight (8) dimensions, and the projection layer 430 may project that dimension to thirty-two (32).

[0046] In other embodiments, the VAE-based residual encoder 180 includes a phoneme-level fine-grained VAE network, which can be employed by the non-autoregressive neural network 200, instead of... Figure 2 The global VAE network depicted in the diagram. Although... Figure 2The global VAE network depicted encodes the reference Mel frequency spectrogram sequence 202 into a utterance-level global variational embedding 220, but the phoneme-level fine-grained VAE network is configured to encode spectrogram frames from the reference Mel frequency spectrogram sequence 202 associated with each phoneme in the input text sequence 206 into corresponding phoneme-level variational embeddings 220. More specifically, the phoneme-level fine-grained VAE network can align the reference Mel frequency spectrogram sequence 202 with each phoneme in the phoneme sequence extracted from the input text sequence 206 and encode a sequence of phoneme-level variational embeddings 220. Thus, each phoneme-level variational embedding 220 in the sequence of phoneme-level variational embeddings 220 encoded by the phoneme-level fine-grained VAE network encodes a corresponding subset of one or more spectrogram frames from the reference Mel frequency spectrogram sequence 202, which includes the corresponding phoneme in the phoneme sequence extracted from the input text sequence 206. Phoneme-level fine-grained VAE networks can initially cascade a 202-series reference Mel frequency spectrogram to the speaker embedding y. s And position embedding 216, the speaker embedding y s The positional embedding 216 indicates the speaker who utters the utterance associated with the reference Mel frequency spectrogram sequence 202, and indicates the phoneme position information of each phoneme in the phoneme sequence extracted from the input text sequence 206. Each positional embedding 216 may include a fixed-length vector containing information about the specific position of the corresponding phoneme in the phoneme sequence extracted from the input text sequence 206. Subsequently, a stack of five (5) 8-head 17×1LConv blocks is concatenated to compute attention using the layer-normalized encoded text sequence 219 output from the text encoder 210.

[0047] Continue to refer to Figure 2The text encoder 210 encodes the input text sequence 206 into a text encoding sequence 219. The text encoding sequence 219 includes an encoded representation of the sequence of speech units (e.g., phonemes) extracted from the input text sequence 206. The input text sequence 206 may include words, each having one or more phonemes, silences at all word boundaries, and punctuation marks. Therefore, the input text sequence 206 comprises a phoneme sequence, and the text encoder 210 can receive a corresponding token embedding for each phoneme in the phoneme sequence from the token embedding lookup table 207. Here, the corresponding token embedding includes the phoneme embedding. However, in other examples, the token embedding lookup table 207 may obtain token embeddings for other types of speech input associated with the input text sequence 206, rather than phonemes, such as, but not limited to, subphonemes (e.g., multiphonemes), graphemes, word fragments, or words in a utterance. After receiving the corresponding lexical embedding for each phoneme in the phoneme sequence, the text encoder 210 uses an encoder pre-net neural network 208 to process each corresponding lexical embedding to generate a corresponding transform embedding 209 for each phoneme. Subsequently, a stack of convolutional (Conv) blocks 212 can process the corresponding transform embeddings 209 to generate a convolutional output 213. In some examples, this stack of Conv blocks 212 comprises three (3) identical 5×1 Conv blocks. Figure 4 A schematic diagram 400 of an example Conv block is shown, comprising a Conv layer 402, a batch normalization layer 404, and a dropout layer 406. During training, the batch normalization layer 404 may apply batch normalization to reduce internal covariate shifts. The dropout layer 406 may reduce overfitting. Finally, adder 214 combines the convolution output 213 with the position embedding 216 to generate input 215, and a stack of self-attention blocks 218 processes input 215 to generate an encoded text sequence 219. In the example shown, the stack of self-attention blocks 218 comprises six (6) transform blocks. In other examples, self-attention blocks 218 may include LConv blocks instead of transform blocks.

[0048] It is noteworthy that, since each convolutional output 213 simultaneously flows through the stack of self-attention blocks 218 and therefore the stack of self-attention blocks 218 is unaware of the position / order of each phoneme in the input text utterance, the positional embedding 216 combined with the convolutional outputs 213 injects the necessary positional information indicating the order of each phoneme in the input text sequence 206. In contrast, the autoregressive encoder incorporating a recurrent neural network (RNN) inherently considers the order of each phoneme, as each phoneme is parsed sequentially from the input text sequence. However, the text encoder 210 integrating the stack of self-attention blocks 218 employing multi-head self-attention avoids the repetition of the autoregressive encoder, resulting in a significantly reduced training time and, theoretically, capturing longer dependencies in the input text sequence 206.

[0049] Continue to refer to Figure 2 Cascade 222 concatenates variational embedding 220 from residual encoder 180, encoded text sequence 219 output from text encoder 210, and reference speaker embedding ys representing speaker identity 205 of the reference speaker emitting the reference audio signal. Duration decoder 230 is configured to decode concatenation 224 of encoded text sequence 219, variational embedding 220, and reference speaker embedding ys to generate output 238 of duration decoder 230 for predicting phoneme duration 240 for each phoneme in the phoneme sequence of input text sequence 206. In some embodiments, duration decoder 230 includes a stack of self-attention blocks 232 followed by two independent projections. In the example shown, the stack of self-attention blocks includes four (4) 3×1LConv blocks. Reference above Figure 5Each LCoV block in the stack of self-attention block 232 may include a GLU unit 502, an LCoV layer 504, and an FF layer 506 with residual connections. The duration decoder 230 predicts the phoneme duration 240 for each phoneme by predicting the probability of a non-zero duration 235 for each phoneme and predicting the consecutive phoneme duration 237 for each phoneme. Since the phoneme sequence in the input text sequence 206 includes regular phonemes, silences between word boundaries, and punctuation marks, although silences and punctuation marks are generally associated with non-zero durations, only regular phonemes are associated with non-zero durations. Therefore, the duration decoder 230 can use a sigmoid activation 234 following the first activation in two independent activations to predict the probability of a non-zero duration 235, and a softplus activation following the second projection in two independent projections to predict the consecutive phoneme duration for each phoneme. For each phoneme, the duration decoder can determine whether the probability of the predicted non-zero duration 235 for the corresponding phoneme is less than a threshold. When the probability of the non-zero duration is less than the threshold, the multiplier 239 can zero out the continuous phoneme duration 237 predicted by the softplus activation 236 for the corresponding phoneme. Otherwise, when the probability of the non-zero duration is not less than the threshold, the predicted phoneme duration 240 can be set to be equal to the continuous phoneme duration 237 predicted by the softplus activation 236.

[0050] An external aligner, such as one based on a Hidden Markov Model, can be used to sample the reference phoneme duration 245 for each phoneme from the reference audio signal 201. The duration decoder 230 can compute a cross-entropy (CE) loss term 242 between the probabilities of the reference phoneme duration 245 and the non-zero duration 235. The duration decoder 230 can also compute an L1 loss term 244 between the predicted phoneme duration 240 and the reference phoneme duration 245. Phoneme duration loss based on two levels of CE and L1 loss terms 242 and 244 can be computed for the input text sequence 206.

[0051] Continue to refer to Figure 2The upsampling layer 246 uses a reference phoneme duration 245 sampled from the reference audio signal to upsample the output 238 of the duration decoder 230 into a plurality of frames specified by the reference phoneme duration 245. Here, the number of frames corresponds to the length of the reference Mel frequency spectrogram sequence 202 (i.e., the number of frames generated by the upsampling layer 246 is equal to the number of reference spectrogram frames in the reference Mel frequency spectrogram sequence 202). One or more position embeddings 216 may be added to the output 238 of the upsampling layer 246 to inform the spectrogram decoder 250 of the corresponding frame position within each phoneme. One or more different types of position embeddings 216 may include at least one of the following: a transformer-style sine embedding of the frame position within the phoneme, a transformer-style sine embedding of the phoneme duration, or a fractional number of frames within the phoneme. In some embodiments, when several different types of the above-described position embeddings 216 are added, the weighting layer 248 learns position weights 249 to combine one or more position embeddings 216 with the output 238 of the duration decoder 230. For example, for each channel and location embedding, the weighted layer 248 can learn softmax normalized weights to teach the non-autoregressive neural network 200 to learn which location embedding 216 to copy to the spectrogram decoder 250.

[0052] Spectrum decoder 250 is configured to receive, as input, the upsampled output 238 of duration decoder 230 combined with weighted position embedding 216, and to generate one or more predicted Mel-frequency spectrogram sequences 302 for the input text sequence 206 as output. Spectrum decoder 250 may include a stack of multiple self-attention blocks 252, 252a-n with multi-head attention. In some examples, spectrum decoder 250 includes six (6) eight-head 17×1 LConv blocks with 0.1 exits. Spectrum decoder 250 may include more or fewer than six LConv blocks. (Refer to above) Figure 5 Each LCoV block in the stack of self-attention blocks 232 may include a GLU unit 502, an LCoV layer 504, and an FF layer 506 along with residual connections. In other examples, each self-attention block 252 in the stack includes the same transformer block.

[0053] In some implementations, the spectrogram decoder 250 generates a corresponding predicted Mel-frequency spectrogram sequence 302, 302a-n as output from each self-attention block 252 in the stack of self-attention blocks 252, 252a-n. The network 200 can be trained such that the number of frames in each corresponding predicted Mel-frequency spectrogram sequence 302 is equal to the number of frames in the reference Mel-frequency spectrogram sequence 202 input to the residual encoder 180. In the example shown, each self-attention block 252a-n is paired with a corresponding projection layer 254a-n that projects the output 253 from the self-attention block 252 to generate a corresponding predicted Mel-frequency spectrogram sequence 302a-n with dimensions matching those of the reference Mel-frequency spectrogram sequence 202. In some examples, projection layer 254 projects 128 bins of predicted Mel-frequency spectrogram sequence 302. By predicting multiple Mel-frequency spectrogram sequences 302a-n, an iterative spectrogram loss can be used to train the non-autoregressive neural network 200. In other words, for each corresponding predicted Mel frequency spectrum sequence 302a-n, the corresponding spectrum loss 270, 270a-n can be determined based on the corresponding predicted Mel frequency spectrum sequence 302 and the reference Mel frequency spectrum sequence 202. The corresponding spectrum loss 270 may include an L1 loss term. For example, the first spectrum loss 270a may be determined based on the first predicted Mel frequency spectrum sequence 302a and the reference Mel frequency spectrum sequence 202, the second spectrum loss 270b may be determined based on the first predicted Mel frequency spectrum sequence 302a and the reference Mel frequency spectrum sequence 202, and so on, until all corresponding spectrum losses 270a-n are iteratively determined based on the predicted Mel frequency spectrum sequences 302a-n. The spectrum loss 270 can be aggregated to generate a final spectrum loss 280. The final spectrum loss 280 may correspond to the iterative L1 loss term 280. Aggregating the spectrum losses 270a-n may include summing the spectrum losses 270a-n to obtain the final spectrum loss 280. Optionally, the aggregated spectrogram loss may include averaging the spectrogram loss.

[0054] A deep neural network 200 can be trained such that the number of frames in each corresponding predicted Mel frequency spectrogram sequence 302 is equal to the number of frames in the reference Mel frequency spectrogram sequence 202 input to the residual encoder 180. Furthermore, the deep neural network 200 is trained such that the data associated with the reference Mel frequency spectrogram sequence 202 and the predicted Mel frequency spectrogram sequence 302 substantially match each other. The predicted Mel frequency spectrogram sequence 302 can implicitly provide a prosodic / style representation of the reference audio signal 201.

[0055] Figure 3 It shows the result of Figure 2 An example of a non-autoregressive TTS model 300 trained with a non-autoregressive deep neural network 200. Specifically, Figure 3 A TTS model 300 is depicted, which uses a selected variational embedding 220 to predict a Mel frequency spectrogram sequence 302 for input text utterance 320, whereby the selected variational embedding 220 represents the expected prosody / style for the text utterance 320. During inference, the trained TTS model 300 is executed on computing system 120 or user computing device 10, and can use the selected variational embedding 220 to predict the corresponding Mel spectrogram sequence 302 for input text utterance 320. Here, the TTS model 300 selects the variational embedding 220 representing the expected prosody / style of the text utterance 320 from storage device 185. In some examples, the user provides a user input instruction indicating the desired choice of prosody / style for the synthesized speech 152 to convey for the text utterance 320, and the TTS model 300 selects the appropriate variational embedding 220 representing the desired prosody / style from data storage device 185. In these examples, the expected prosody / style can be selected by the user by indicating a speaker ID 205 associated with a specific speaker speaking in the expected rhythm / style and / or by specifying a specific prosody vertical industry (e.g., news broadcaster, sports broadcaster, etc.) corresponding to the expected prosody / style. The selected variational embedding 220 may correspond to a previous variational embedding 220 sampled from a VAE-based residual encoder 180. The trained TTS model 300 uses synthesizer 155 to generate synthesized speech 152 for the corresponding input text utterance 320 with the expected prosody / style (e.g., the selected variational embedding 220). That is, the selected variational embedding 220 may include the expected prosody / style (e.g., news broadcaster, sports broadcaster, etc.) stored on differential encoder 180. The selected variational embedding 220 conveys the expected prosody / style via synthesized speech 152 for the input text utterance 320.

[0056] In an additional implementation, the trained TTS model 300 employs a residual encoder 180 during inference to extract / predict variational embeddings 220 in real time for predicting Mel-frequency spectrogram sequences 302 for input text utterances 320. For example, the residual encoder 180 may receive a reference audio signal 201 (uttered by a human user) conveying the intended prosody / style (e.g., “say it like this”). Figure 2The TTS model 300 extracts / predicts the corresponding variational embedding 220 representing the expected prosody / style. Subsequently, the trained TTS model 300 can use the variational embedding 220 to efficiently transfer the expected prosody / style conveyed by the reference audio signal 201 to predict the Mel frequency spectrogram sequence 302 for the input text utterance 320. Therefore, the input text utterance 320 to be synthesized as expressing speech 152 and the reference audio signal 201 conveying the expected prosody / style to be transferred to express speech 152 can include different linguistic contents.

[0057] Specifically, text encoder 210 encodes the phoneme sequence extracted from text utterance 320 into an encoded text sequence 219. Text encoder 210 can receive the corresponding lexical embedding for each phoneme in the phoneme sequence extracted from text utterance 320 from lexical embedding lookup table 207. After receiving the corresponding lexical embedding for each phoneme in the phoneme sequence extracted from text utterance 320, text encoder 210 processes each corresponding lexical embedding using encoder pre-net neural network 208 to generate a corresponding transform embedding 209 for each phoneme. Subsequently, this stack of Conv blocks 212 (e.g., three (3) identical 5×1 Conv blocks) processes the corresponding transform embedding 209 to generate a convolutional output 213. Finally, adder 214 combines the convolutional output 213 with positional embedding 216 to generate input 215, and a stack of self-attention blocks 218 (e.g., six transform blocks) processes input 215 to generate the encoded text sequence 219. The positional embedding 216, combined with the convolution output 213, provides the necessary positional information to indicate the order of each phoneme in the phoneme sequence used for text utterance 320.

[0058] Continue to refer to Figure 3 Cascade 222 concatenates the selected variational embedding 220, the encoded text sequence 219, and optionally the reference speaker embedding ys to generate concatenation 224. Here, the reference speaker embedding ys may represent the speaker identity 205 of a reference speaker emitting one or more reference audio signals 201 associated with the selected variational embedding 220, or the speaker identity 205 of some other reference speaker having the vocal characteristics conveyed in the resulting synthesized speech 152. Duration decoder 230 is configured to decode the concatenation 224 of the encoded text sequence 219, the selected variational embedding 220, and the reference speaker embedding ys to generate the output 238 of duration decoder 230 predicting the phoneme duration 240 for each phoneme in the phoneme sequence of the input text utterance 320.

[0059] In some implementations, the duration decoder 230 includes a stack of self-attention blocks 232 (e.g., four (4) 3×1LConv blocks) followed by two independent projections 234, 236. The duration decoder 230 can predict the phoneme duration for each phoneme by predicting the probability of a non-zero duration 235 for each phoneme and predicting the duration of consecutive phonemes 237 for each phoneme. Since the phoneme sequence in the input text utterance 320 includes regular phonemes, silences between word boundaries, and punctuation marks, although silences and punctuation marks are generally associated with non-zero durations, only regular phonemes are associated with non-zero durations. Therefore, the duration decoder 230 can use a sigmoid activation 234 following the first activation of the two independent activations to predict the probability of a non-zero duration 235, and a softplus activation 236 following the second projection of the two independent projections to predict the duration of consecutive phonemes 237 for each phoneme. The duration decoder 230 determines whether the probability of a non-zero duration 235 for each phoneme is less than a threshold. When the probability of a non-zero duration 235 is less than the threshold, the multiplier 239 can zero out the continuous phoneme duration 237 predicted by the softplus activation 236 for the corresponding phoneme. Otherwise, when the probability of a non-zero duration 235 is not less than the threshold, the predicted phoneme duration 240 can be set to be equal to the continuous phoneme duration 237 predicted by the softplus activation 236.

[0060] The trained TTS model 300 may include an upsampling layer 246 that uses predicted phoneme durations 240 to upsample the output 238 of the duration decoder 230 into multiple frames. Here, the number of frames corresponds to the prediction length of the predicted spectrogram 302 determined by the predicted phoneme durations 240 and the output 238 for the corresponding input text utterance 320. The TTS model 300 may obtain one or more position embeddings 216, which represent the phoneme position information of each phoneme in the phoneme sequence for the input text utterance 320. One or more position embeddings 216 may be added to the number of frames of the output 238 generated by the upsampling layer 246 to inform the spectrogram decoder 250 of the corresponding frame position within each phoneme. That is, the stack of the self-attention block 252 may not know the position / order of each phoneme in the input text utterance 320, so the position embeddings 216 provide the necessary order of each phoneme in the phoneme sequence. In some implementations, several different types of positional embeddings 216 are added to the output 238, and a weighting layer 248 learns positional weights 249 to combine one or more positional embeddings 216 with the output 238 of the duration decoder 230. For example, for each channel and positional embedding 216, the weighting layer 248 may learn softmax normalized weights to teach the non-autoregressive neural network 200 to select which positional embedding 216 to copy to the spectrogram decoder 250. One or more positional embeddings 216 may include the following types of positional embeddings: transformer-style sinusoidal embeddings, each indicating the frame position within a phoneme; transformer-style sinusoidal embeddings, each indicating the predicted phoneme duration of the corresponding phoneme; and fractional levels of frames within a phoneme.

[0061] The spectrogram decoder 250 is configured to generate a predicted Mel-frequency spectrogram sequence 302 for text utterance 320 based on learned positional weights 249 derived from the output 238 of the duration decoder 230 and the upsampled predicted phoneme duration 240. Here, the predicted Mel-frequency spectrogram sequence 302 has an expected prosody / style specified by the selected variational embedding 220. The predicted Mel-frequency spectrogram sequence 302 for text utterance 320 is based on the positional embedding 216 and upsampling of the output 238 of the duration decoder 230 to multiple frames.

[0062] The spectrogram decoder 250 generates a corresponding predicted Mel frequency spectrogram sequence 302 as output based on the last self-attention block 252 in the stack of self-attention blocks 252a-n. Here, each self-attention block 252 in the stack of self-attention blocks 252 of the spectrogram decoder 250 includes one of the same LCoV block or the same transformer block. In some examples, the spectrogram decoder 250 includes six (6) eight-head 17×1 LCoV blocks with 0.1 exits. The last feedforward (FF) layer 506 of each self-attention block 252 ( Figure 5 The output of ) is provided as the input to subsequent self-attention blocks 252. That is, the GLU unit 502 of the first self-attention block 252a in the stack of self-attention blocks 252a-n ( Figure 5 ) and LConv layer 504 ( Figure 5 The output 238 from the duration decoder 230 and the predicted phoneme duration 240 are processed, and the output of the last FF layer 506 from the first self-attention block stack 252a is provided as input to the subsequent second self-attention block 252b in the stack of self-attention blocks 252. The output of the last FF layer of each self-attention block 252 is provided as input to the subsequent self-attention block 252 until the last self-attention block 252n in the stack of self-attention blocks 252 is reached. The last self-attention block 252n in the stack of self-attention blocks 252 (e.g., the sixth self-attention block 252) is paired with the corresponding projection layer 254 that projects the output 253 from the last self-attention block 252 to generate the corresponding predicted Mel frequency spectrogram sequence 302.

[0063] The predicted Mel-frequency spectrogram sequence 302 generated by spectrogram decoder 250 corresponds to the input text utterance 320 and conveys the expected prosody / style indicated by the selected variational embedding 220. The trained TTS model 300 provides the synthesizer 155 with the predicted Mel-frequency spectrogram sequence 302 for the input text utterance 320 for conversion into a time-domain audio waveform indicating the synthesized speech 152. The synthesized speech 152 can be audibly output as a spoken representation of the input text utterance 320, including the expected prosody / style as indicated by the selected variational embedding 220.

[0064] Figure 6 This is a flowchart illustrating an exemplary arrangement of the operation of a computer-implemented method 600 for training a non-autoregressive TTS model. In step 602, method 600 includes receiving training data comprising a reference audio signal 201 and a corresponding input text sequence 206. The reference audio signal 201 comprises spoken utterances, and the input text sequence 206 corresponds to a transcription of the reference audio signal 201. In step 604, method 600 includes encoding the reference audio signal 201 into a variational embedding 220 using a residual encoder 180. The variational embedding 220 separates style / prosodic information from the reference audio signal 201. In step 606, method 600 includes encoding the input text sequence 206 into an encoded text sequence 219 using a text encoder 210. In step 608, method 600 includes using a duration decoder 230 to predict the phoneme duration 240 of each phoneme in the input text sequence 206 based on the encoded text sequence 219 and the variational embedding 220.

[0065] In step 610, method 600 includes determining a phoneme duration loss 244 based on a predicted phoneme duration 240 and a reference phoneme duration 245 sampled from a reference audio signal 201 for each phoneme in the input text sequence 206. In step 612, method 600 includes generating one or more predicted Mel-frequency spectrogram sequences 302 as output from a spectrogram decoder 250 comprising a stack including self-attention blocks 252, based on the output 238 of a duration decoder 230. In step 614, method 600 includes determining a final spectrogram loss 280 based on one or more predicted Mel-frequency spectrogram sequences 302 and a reference Mel-frequency spectrogram sequence sampled from the reference audio signal 201. In step 616, method 600 includes training a TTS model 300 based on the final spectrogram loss 280 and the corresponding phoneme duration loss 244 determined for each phoneme in the input text sequence 206.

[0066] Figure 7 This is a flowchart illustrating an exemplary arrangement of the operation of a computer-implemented method 700 for performing a trained non-autoregressive TTS model 300. In step 702, method 700 includes receiving a text utterance 320 to be synthesized into speech 152. In step 704, method 700 includes encoding a phoneme sequence extracted from the text utterance 320 into an encoded text sequence 219 using a text encoder 210 of the non-autoregressive TTS model 300. In step 706, method 700 includes selecting a variational embedding 220 for the text utterance 320. The variational embedding 220 specifies the expected prosody / style for synthesizing the text utterance 320 into speech 152. For each phoneme in the phoneme sequence, in step 708, method 700 includes predicting a phoneme duration 240 for the corresponding phoneme using a duration decoder 230 of the non-autoregressive TTS model 300 based on the encoded text sequence 219 and the selected variational embedding 220. In step 710, method 700 includes generating a predicted Mel frequency spectrogram sequence 302 for text utterance 320 as output from the spectrogram decoder 250 of a non-autoregressive TTS model 300, which includes a stack of self-attention blocks 252, based on the output 238 of the duration decoder 230 and the predicted phoneme duration 240. The predicted Mel frequency spectrogram sequence 302 has an expected prosody / style specified by a selected variational embedding 220.

[0067] Figure 8This is a schematic diagram of an example computing device 800 that can be used to implement the systems and methods described in this document. The computing device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0068] Computing device 800 includes a processor 810, a memory 820, a storage device 830, a high-speed interface / controller 840 connected to the memory 820 and a high-speed expansion port 850, and a low-speed interface / controller 860 connected to a low-speed bus 870 and the storage device 830. Each of components 810, 820, 830, 840, 850, and 860 is interconnected using various buses and may be mounted on a common motherboard or otherwise as appropriate. Processor 810 is capable of processing instructions for execution within computing device 800, including instructions stored on memory 820 or storage device 830, to display graphical information for a graphical user interface (GUI) on an external input / output device such as a display 880 coupled to high-speed interface 840. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and multiple types of memory, may be used as appropriate. Moreover, multiple computing devices 800 may be connected, each providing a portion of the necessary operation (e.g., as a server library, blade server group, or multiprocessor system).

[0069] Memory 820 stores information non-transitorily within computing device 800. Memory 820 may be a computer-readable medium, volatile memory cells(s), or non-volatile memory cells(s). Non-transitory memory 820 may be a physical device for storing programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 800. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0070] Storage device 830 provides mass storage for computing device 800. In some embodiments, storage device 830 is a computer-readable medium. In various embodiments, storage device 830 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or an array of devices, including devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer or machine-readable medium, such as memory 820, storage device 830, or memory on processor 810.

[0071] High-speed controller 840 manages bandwidth-intensive operations of computing device 800, while low-speed controller 860 manages lower bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 840 is coupled to memory 820, display 880 (e.g., via a graphics processor or accelerator), and high-speed expansion port 850, which can receive various expansion cards (not shown). In some embodiments, low-speed controller 860 is coupled to storage device 830 and low-speed expansion port 890. Low-speed expansion port 890, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices, such as switches or routers, for example, via a network adapter.

[0072] As shown in the figure, the computing device 800 can be implemented in many different forms. For example, it can be implemented as a standard server 800a or multiple times in a set of such servers 800a, as a laptop computer 800b, or as part of a rack-mounted server system 800c.

[0073] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can be included in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive and transmit data and instructions from and to a storage system, at least one input device, and at least one output device.

[0074] A software application (i.e., a software resource) can refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0075] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including receiving machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0076] The processes and logic described in this specification can be implemented by one or more programmable processors (also known as data processing hardware) to execute one or more computer programs to perform functions by manipulating input data and generating output. Processes and logical flows can also be executed by dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magneto-optical, magneto-optical, or optical discs, or operatively coupled to receive data from or transfer data to one or more mass storage devices, or both. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory can be supplemented by or incorporated into dedicated logic circuits.

[0077] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen, and optionally a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0078] Many embodiments have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A computer-implemented method (600) that, when executed on data processing hardware (122), causes the data processing hardware (122) to perform operations for training a non-autoregressive text-to-speech TTS model (300), the operations comprising: The system receives training data including a reference audio signal (201) and a corresponding input text sequence (206), wherein the reference audio signal (201) includes spoken utterances and the input text sequence (206) corresponds to a transcription of the reference audio signal (201); The reference audio signal (201) is encoded into a variational embedding (220) using a residual encoder (180), which separates style / prosodic information from the reference audio signal (201); The input text sequence (206) is encoded into an encoded text sequence (219) using a text encoder (210); The duration decoder (230) is used to predict the phoneme duration (240) for each phoneme in the input text sequence (206) based on the encoded text sequence (219) and the variational embedding (220); The phoneme duration loss (244) is determined based on the predicted phoneme duration (240) and the reference phoneme duration (245) sampled from the reference audio signal (201) for each phoneme in the input text sequence (206). The spectrogram decoder (250), which includes a stack of self-attention blocks (252), generates one or more predicted Mel frequency spectrogram sequences (302) for the input text sequence (206) as output based on the output of the duration decoder (230); The final spectrogram loss (280) is determined based on the one or more predicted Mel frequency spectrogram sequences (302) and the reference Mel frequency spectrogram sequence (202) sampled from the reference audio signal (201); as well as The TTS model (300) is trained based on the final spectrogram loss (280) and the corresponding phoneme duration loss (244) determined for each phoneme in the input text sequence (206).

2. The computer-implemented method (600) according to claim 1, wherein, Each self-attention block (252) in the stack of the self-attention blocks (252) comprises the same lightweight convolutional LConv block.

3. The computer-implemented method (600) according to claim 1, wherein, Each self-attention block (252) in the stack of the self-attention blocks (252) includes the same transformer block.

4. The computer-implemented method (600) according to claim 1, wherein, The input text sequence (206) includes words, each having one or more phonemes, silences at all word boundaries, and punctuation marks.

5. The computer-implemented method (600) according to claim 1, wherein: The residual encoder (180) includes a global variational automatic encoder (VAE); as well as Encoding the reference audio signal (201) into the variational embedding (220) includes: Sample the reference Mel frequency spectrum sequence (202) from the reference audio signal (201); as well as The reference Mel frequency spectrum sequence (202) is encoded into the variational embedding (220) using the global VAE.

6. The computer-implemented method (600) according to claim 1, wherein: The residual encoder (180) includes a phoneme-level fine-grained variational automatic encoder (VAE). as well as Encoding the reference audio signal (201) into the variational embedding (220) includes: Sample the reference Mel frequency spectrum sequence (202) from the reference audio signal (201); Align the reference Mel frequency spectrogram sequence (202) with each phoneme in the phoneme sequence extracted from the input text sequence (206); and The phoneme-level fine-grained VAE is used to encode the sequence of phoneme-level variational embeddings (220) based on aligning the reference Mel frequency spectrogram sequence with each phoneme in the phoneme sequence.

7. The computer-implemented method (600) according to claim 1, wherein, The residual encoder (180) includes a stack of lightweight convolutional LCoV blocks (410, 420), each LCoV block (410, 420) in the stack comprising: Gate linear unit (GLU) layer (502); LConv layer (504), the LConv layer is configured to receive the output of the GLU layer (502); A residual connection (508) configured to cascade the output of the LConv layer (504) with the input of the GLU layer (502); and The final feedforward layer (506) is configured to receive the residual connection (508) as input, which concatenates the output of the LConv layer (504) with the input of the GLU layer (502).

8. The computer-implemented method (600) according to claim 1, wherein, The operation also includes: The output of the duration decoder (230) is upsampled into multiple frames using the reference phoneme duration (240) sampled from the reference audio signal (201) for each phoneme in the input text sequence (206); and Obtain the position embedding (216) representing the phoneme position information for each phoneme in the input text utterance (320). The one or more predicted Mel frequency spectrogram sequences (302) for generating the input text sequence (206) are upsampled to the plurality of frames based on the position embedding (216) and the output of the duration decoder (230).

9. The computer-implemented method (600) according to claim 1, wherein: Generating the one or more predicted Mel frequency spectrogram sequences (302) for the input text sequence (206) includes generating a corresponding predicted Mel frequency spectrogram sequence (302) as output from each self-attention block (252) in the stack of the spectrogram decoder (250); as well as Determining the final spectrogram loss (280) includes: For each corresponding predicted Mel frequency spectrogram sequence (302), a corresponding spectrogram loss (270) is determined based on the predicted Mel frequency spectrogram sequence (302) and the reference Mel frequency spectrogram sequence (202); and The corresponding spectrogram loss (270) determined for the predicted Mel frequency spectrogram sequence (302) is aggregated to generate the final spectrogram loss (280).

10. The computer-implemented method (600) according to claim 1, wherein: The duration decoder (230) includes a stack of self-attention blocks (232) followed by two independent projections; as well as Predicting the phoneme duration (240) for each phoneme in the input text sequence (206) includes: The sigmoid activation (234) following the first projection in the two independent projections is used to predict the probability of non-zero duration for each phoneme; as well as The softplus activation (236) following the second projection in the two independent projections is used to predict the duration of the phoneme for each phoneme (240).

11. The computer-implemented method (600) according to claim 10, wherein, The operation also includes, for each phoneme: Determine whether the probability of the predicted non-zero duration for the corresponding phoneme is less than a threshold; as well as When the probability of the non-zero duration is less than the threshold, the predicted phoneme duration (240) for the corresponding phoneme will be zeroed.

12. The computer-implemented method (600) according to claim 1, wherein, The operation also includes: The encoded text sequence (219), the variational embedding (220), and the reference speaker embedding representing the identity of the reference speaker emitting the reference audio signal (201) are concatenated; and The duration decoder (230) generates the output of the duration decoder (230) by receiving the cascade (224) of the encoded text sequence (219), the variational embedding (220), and the reference speaker embedding as input.

13. The computer-implemented method (600) according to any one of claims 1-12, wherein: The input text sequence (206) includes a phoneme sequence; and Encoding the input text sequence (206) into the encoded text sequence (219) includes: Receive the corresponding embedding of each phoneme in the phoneme sequence from the phoneme lookup table (208); For each phoneme in the phoneme sequence, the corresponding embedding is processed using the encoder pre-network neural network (207) of the text encoder (210) to generate the corresponding transform embedding (209) of the phoneme; The corresponding transform embedding (209) is processed using a set of convolutional blocks (212) to generate a convolutional output (213); and The convolution output (213) is processed using a stack of self-attention blocks (218) to generate the encoded text sequence (219).

14. A system (100) for training a non-autoregressive text-to-speech (TTS) model, the system comprising: Data processing hardware (122); as well as A memory hardware (124) communicating with the data processing hardware (122), the memory hardware (124) storing instructions that, when executed by the data processing hardware (122), cause the data processing hardware (122) to perform operations including the following: The system receives training data including a reference audio signal (201) and a corresponding input text sequence (206), wherein the reference audio signal (201) includes spoken utterances and the input text sequence (206) corresponds to a transcription of the reference audio signal (201); The reference audio signal (201) is encoded into a variational embedding (220) using a residual encoder (180), which separates style / prosodic information from the reference audio signal (201); The input text sequence (206) is encoded into an encoded text sequence (219) using a text encoder (210); The duration decoder (230) is used to predict the phoneme duration (240) for each phoneme in the input text sequence (206) based on the encoded text sequence (219) and the variational embedding (220); The phoneme duration loss (244) is determined based on the predicted phoneme duration (240) and the reference phoneme duration (245) sampled from the reference audio signal (201) for each phoneme in the input text sequence (206). The spectrogram decoder (250), which includes a stack of self-attention blocks (252), generates one or more predicted Mel frequency spectrogram sequences (302) for the input text sequence (206) as output based on the output of the duration decoder (230); The final spectrogram loss (280) is determined based on the one or more predicted Mel frequency spectrogram sequences (302) and the reference Mel frequency spectrogram sequence (202) sampled from the reference audio signal (201); as well as The TTS model (300) is trained based on the final spectrogram loss (280) and the corresponding phoneme duration loss (244) determined for each phoneme in the input text sequence (206).

15. The system (100) according to claim 14, wherein, Each self-attention block (252) in the stack of the self-attention blocks (252) comprises the same lightweight convolutional LConv block.

16. The system (100) according to claim 14, wherein, Each self-attention block (252) in the stack of the self-attention blocks (252) includes the same transformer block.

17. The system (100) according to claim 14, wherein, The input text sequence (206) includes words, each having one or more phonemes, silences at all word boundaries, and punctuation marks.

18. The system (100) according to claim 14, wherein: The residual encoder (180) includes a global variational automatic encoder (VAE); as well as Encoding the reference audio signal (201) into the variational embedding (220) includes: Sample the reference Mel frequency spectrum sequence (202) from the reference audio signal (201); as well as The reference Mel frequency spectrum sequence (202) is encoded into the variational embedding (220) using the global VAE.

19. The system (100) according to claim 14, wherein: The residual encoder (180) includes a phoneme-level fine-grained variational automatic encoder (VAE). as well as Encoding the reference audio signal (201) into the variational embedding (220) includes: Sample the reference Mel frequency spectrum sequence (202) from the reference audio signal (201); Align the reference Mel frequency spectrogram sequence (202) with each phoneme in the phoneme sequence extracted from the input text sequence (206); and The phoneme-level fine-grained VAE is used to encode the sequence of phoneme-level variational embeddings (220) based on aligning the reference Mel frequency spectrogram sequence (202) with each phoneme in the phoneme sequence.

20. The system (100) according to claim 14, wherein, The residual encoder (180) includes a stack of lightweight convolutional LCoV blocks (410, 420), each LCoV block (410, 420) in the stack comprising: Gate linear unit (GLU) layer (502); LConv layer (504), the LConv layer is configured to receive the output of the GLU layer (502); A residual connection (508) configured to cascade the output of the LConv layer (504) with the input of the GLU layer (502); and The final feedforward layer (506) is configured to receive the residual connection (508) as input, which concatenates the output of the LConv layer (504) with the input of the GLU layer (502).

21. The system (100) according to claim 14, wherein, The operation also includes: The output of the duration decoder (230) is upsampled into multiple frames using the reference phoneme duration (240) sampled from the reference audio signal (201) for each phoneme in the input text sequence (206); and Obtain the position embedding (216) representing the phoneme position information for each phoneme in the input text utterance (320). The one or more predicted Mel frequency spectrogram sequences (302) for generating the input text sequence (206) are upsampled to the plurality of frames based on the position embedding (216) and the output of the duration decoder (230).

22. The system (100) according to claim 14, wherein: Generating the one or more predicted Mel frequency spectrogram sequences (302) for the input text sequence (206) includes generating a corresponding predicted Mel frequency spectrogram sequence (302) as output from each self-attention block (252) in the stack of the spectrogram decoder (250); as well as Determining the final spectrogram loss (280) includes: For each corresponding predicted Mel frequency spectrogram sequence (302), a corresponding spectrogram loss (270) is determined based on the predicted Mel frequency spectrogram sequence (302) and the reference Mel frequency spectrogram sequence (202); and The corresponding spectrogram loss (270) determined for the predicted Mel frequency spectrogram sequence (302) is aggregated to generate the final spectrogram loss (280).

23. The system (100) according to claim 14, wherein: The duration decoder (230) includes a stack of self-attention blocks (232) followed by two independent projections; as well as Predicting the phoneme duration (240) for each phoneme in the input text sequence (206) includes: The sigmoid activation (234) following the first projection in the two independent projections is used to predict the probability of non-zero duration for each phoneme; as well as The softplus activation (236) following the second projection in the two independent projections is used to predict the duration of the phoneme for each phoneme (240).

24. The system (100) according to claim 23, wherein, The operation also includes, for each phoneme: Determine whether the probability of the predicted non-zero duration for the corresponding phoneme is less than a threshold; as well as When the probability of the non-zero duration is less than the threshold, the predicted phoneme duration (240) for the corresponding phoneme will be zeroed.

25. The system (100) according to claim 14, wherein, The operation also includes: The encoded text sequence (219), the variational embedding (220), and the reference speaker embedding representing the identity of the reference speaker emitting the reference audio signal (201) are concatenated; and The duration decoder (230) generates the output of the duration decoder (230) by receiving the cascade (224) of the encoded text sequence (219), the variational embedding (220), and the reference speaker embedding as input.

26. The system (100) according to any one of claims 14-25, wherein: The input text sequence (206) includes a phoneme sequence; and Encoding the input text sequence (206) into the encoded text sequence (219) includes: Receive the corresponding embedding of each phoneme in the phoneme sequence from the phoneme lookup table (208); For each phoneme in the phoneme sequence, the corresponding embedding is processed using the encoder pre-network neural network (207) of the text encoder (210) to generate the corresponding transform embedding (209) of the phoneme; The corresponding transform embedding (209) is processed using a set of convolutional blocks (212) to generate a convolutional output (213); and The convolution output (213) is processed using a stack of self-attention blocks (218) to generate the encoded text sequence (219).

Images