Unsupervised parallel tacotron non-autoregressive and controllable text-to-speech

Abstract

A method (600) for training a non-autoregressive TTS model (300) includes obtaining a sequence representation (224) of an encoded text sequence (219) concatenated with a variational embedding (220). The method further includes predicting a phoneme duration (240) for each phoneme represented by the encoded text sequence. The method further includes, based on the predicted phoneme durations, learning an interval representation and an auxiliary attention context representation and upsampling the sequence representation into an upsampling output (258) using the interval representation and the auxiliary attention context representation. The method further includes generating one or more predicted mel-spectrogram sequences (302) for the encoded text sequence based on the upsampling output. The method further includes determining a final spectrogram loss (280) based on the predicted mel-spectrogram sequences and a reference mel-spectrogram sequence (202) and training the TTS model based on the final spectrogram loss.

Description

Technical Field

[0001] This disclosure relates to unsupervised parallel tacotron non-autoregressive and controllable text-to-speech. Background Technology

[0002] Text-to-speech (TTS) systems read digital text aloud to users and are becoming increasingly popular on mobile devices. Some TTS models aim to synthesize various aspects of speech, such as speaking style, to produce human-like, natural-sounding speech. Synthesis in TTS models is a one-to-many mapping problem because different prosody for text input can result in multiple possible speech outputs. Many TTS systems utilize autoregressive models that predict current values ​​based on previous values. While autoregressive TTS models can synthesize text and generate highly natural-sounding speech outputs, the hundreds of computations required reduce the efficiency of the inference process. Summary of the Invention

[0003] One aspect of this disclosure provides a computer-implemented method that, when implemented on data processing hardware, causes the data processing hardware to perform operations for training a non-autoregressive text-to-speech (TTS) model. The operations include obtaining a sequence representation of an encoded text sequence concatenated with variational embeddings. The operations also include predicting the phoneme duration of each phoneme represented by the encoded text sequence based on the sequence representation using a duration model network. The operations include learning interval representations using a first function conditioned on the sequence representation based on the predicted phoneme durations, and learning auxiliary attention context representations using a second function conditioned on the sequence representation. The operations also include upsampling the sequence representations to an upsampled output of a specified number of frames using the interval representation matrix and the auxiliary attention context representation. The operations also include generating one or more predicted Mel spectrogram sequences for the encoded text sequence as output from a spectrogram decoder comprising one or more self-attention blocks, and based on the upsampled output. The operations also include determining a final spectrogram loss based on the one or more predicted Mel spectrogram sequences and a reference Mel spectrogram sequence, and training the TTS model based on the final spectrogram loss.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the first function and the second function each include a corresponding multi-layered perceptual-based learnable function. The operation may further include determining a global phoneme duration loss based on the predicted phoneme duration and the average phoneme duration. Here, training the TTS model is further based on the global phoneme duration loss. In some examples, training the TTS model based on the final spectrogram loss and the global phoneme duration loss includes training a duration model network to predict the phoneme duration for each phoneme without using supervised phoneme duration labels extracted from an external aligner.

[0005] In some implementations, the operation further includes using a duration model network to generate corresponding start and end boundaries based on the predicted phoneme duration for each phoneme represented by the encoded text sequence, and mapping the corresponding start and end boundaries generated for each phoneme to a corresponding grid matrix based on the number of phonemes represented by the encoded text sequence and the number of reference frames in the reference Mel spectrogram sequence. Here, the interval representation is learned based on the corresponding grid matrix mapped from the start and end boundaries, and the auxiliary attention context is learned based on the corresponding grid matrix mapped from the start and end boundaries. Upsampling the sequence representation to an upsampled output may include determining the product of the interval representation matrix and the sequence representation, determining the Einstein summation of the interval representation matrix and the auxiliary attention context representation, and projecting and summing the product of the interval representation matrix and the sequence representation and the einsum to generate the upsampled output.

[0006] In some implementations, the operation further includes: receiving training data comprising a reference audio signal and a corresponding input text sequence, the reference audio signal comprising spoken utterance and the input text sequence corresponding to a transcription of the reference audio signal; encoding the reference audio signal into a variational embedding using a residual encoder, the variational embedding decrypting style / prosodic information from the reference audio signal; and encoding the input text sequence into an encoded text sequence using a text encoder. In some examples, the residual encoder includes a global variational autoencoder (VAE). In these examples, encoding the reference audio signal into a variational embedding includes sampling a reference Mel spectrogram sequence from the reference audio signal and encoding the reference Mel spectrogram sequence into a variational embedding using a global VAE. Optionally, the residual encoder may include a phoneme-level fine-grained variational autoencoder (VAE). Here, encoding the reference audio signal into a variational embedding includes: sampling a reference Mel spectrogram sequence from the reference audio signal; aligning the reference Mel spectrogram sequence with each phoneme in the phoneme sequence extracted from the input text sequence; and encoding a phoneme-level variational embedding sequence using a phoneme-level fine-grained VAE and based on the alignment of the reference Mel spectrogram sequence with each phoneme in the phoneme sequence.

[0007] The residual encoder includes a stack of lightweight convolutional (LConv) blocks. Each LCoV block in the stack may include a gated linear unit (GLU) layer, an LCoV layer configured to receive the output of the GLU layer, a residual connection configured to cascade the output of the LCoV layer with the input of the GLU layer, and a final feedforward layer configured to receive the residual connection of the cascaded LCoV layer output and the GLU layer input as input. In some implementations, the operation further includes: cascading encoded text sequences, variational embeddings, and reference speaker embeddings representing the identity of a reference speaker emitting a reference audio signal; and a duration-based modeling network receiving the cascaded encoded text sequence, variational embeddings, and reference speaker embeddings as input to generate a sequence representation. In some examples, the input text sequence includes a phoneme sequence. In these examples, encoding the input text sequence into an encoded text sequence includes: receiving the corresponding embedding for each phoneme in the phoneme sequence from a phoneme lookup table; processing the corresponding embedding for each phoneme in the phoneme sequence using an encoder pre-network neural network of a text encoder to generate a corresponding transformed embedding for the phoneme; processing the corresponding transformed embedding using a set of convolutional blocks to generate a convolutional output; and processing the convolutional output using a stack of self-attention blocks to generate the encoded text sequence. Optionally, each self-attention block in the stack of self-attention blocks includes the same lightweight convolutional (LConv) block. Each self-attention block in the stack of self-attention blocks includes the same transformer block.

[0008] Another aspect of this disclosure provides a system for training a non-autoregressive text-to-speech (TTS) model, the system including data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining a sequence representation of an encoded text sequence concatenated with variational embeddings. The operations also include predicting the phoneme duration of each phoneme represented by the encoded text sequence based on the sequence representation using a duration model network. The operations include learning interval representations using a first function conditioned on the sequence representation, and learning auxiliary attention context representations using a second function conditioned on the sequence representations, based on the predicted phoneme durations. The operations also include upsampling the sequence representations to an upsampled output of a specified number of frames using the interval representation matrix and the auxiliary attention context representations. The operations also include generating one or more predicted Mel spectrogram sequences for the encoded text sequence as output from a spectrogram decoder comprising one or more self-attention blocks and based on the upsampled output. The operations also include determining a final spectrogram loss based on the one or more predicted Mel spectrogram sequences and a reference Mel spectrogram sequence, and training the TTS model based on the final spectrogram loss.

[0009] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the first function and the second function each include a corresponding multi-layered perceptual-based learnable function. The operation may also include determining a global phoneme duration loss based on the predicted phoneme duration and the average phoneme duration. Here, training the TTS model is further based on the global phoneme duration loss. In some examples, training the TTS model based on the final spectrogram loss and the global phoneme duration loss includes training a duration model network to predict the phoneme duration for each phoneme, without using supervised phoneme duration labels extracted from an external aligner.

[0010] In some implementations, the operation further includes using a duration model network to generate corresponding start and end boundaries based on the predicted phoneme duration for each phoneme represented by the encoded text sequence, and mapping the corresponding start and end boundaries generated for each phoneme to a corresponding grid matrix based on the number of phonemes represented by the encoded text sequence and the number of reference frames in the reference Mel spectrogram sequence. Here, the interval representation is learned based on the corresponding grid matrix mapped from the start and end boundaries, and the auxiliary attention context is learned based on the corresponding grid matrix mapped from the start and end boundaries. Upsampling the sequence representation to an upsampled output may include determining the product of the interval representation matrix and the sequence representation, determining the Einstein summation of the interval representation matrix and the auxiliary attention context representation, and projecting and summing the product of the interval representation matrix and the sequence representation and the einsum to generate the upsampled output.

[0011] In some implementations, the operation further includes: receiving training data comprising a reference audio signal and a corresponding input text sequence, the reference audio signal comprising spoken utterance and the input text sequence corresponding to a transcription of the reference audio signal; encoding the reference audio signal into a variational embedding using a residual encoder, the variational embedding decrypting style / prosodic information from the reference audio signal; and encoding the input text sequence into an encoded text sequence using a text encoder. In some examples, the residual encoder includes a global variational autoencoder (VAE). In these examples, encoding the reference audio signal into a variational embedding includes sampling a reference Mel spectrogram sequence from the reference audio signal and encoding the reference Mel spectrogram sequence into a variational embedding using a global VAE. Optionally, the residual encoder may include a phoneme-level fine-grained variational autoencoder (VAE). Here, encoding the reference audio signal into a variational embedding includes: sampling a reference Mel spectrogram sequence from the reference audio signal; aligning the reference Mel spectrogram sequence with each phoneme in the phoneme sequence extracted from the input text sequence; and encoding a phoneme-level variational embedding sequence using a phoneme-level fine-grained VAE and based on the alignment of the reference Mel spectrogram sequence with each phoneme in the phoneme sequence.

[0012] The residual encoder includes a stack of lightweight convolutional (LConv) blocks. Each LCoV block in the stack may include a gated linear unit (GLU) layer, an LCoV layer configured to receive the output of the GLU layer, a residual connection configured to cascade the output of the LCoV layer with the input of the GLU layer, and a final feedforward layer configured to receive the residual connection of the cascaded LCoV layer output and the GLU layer input as input. In some implementations, the operation further includes: cascading encoded text sequences, variational embeddings, and reference speaker embeddings representing the identity of a reference speaker emitting a reference audio signal; and a duration-based modeling network receiving the cascaded encoded text sequence, variational embeddings, and reference speaker embeddings as input to generate a sequence representation. In some examples, the input text sequence includes a phoneme sequence. In these examples, encoding the input text sequence into an encoded text sequence includes: receiving the corresponding embedding for each phoneme in the phoneme sequence from a phoneme lookup table; processing the corresponding embedding for each phoneme in the phoneme sequence using an encoder pre-network neural network of a text encoder to generate a corresponding transformed embedding for the phoneme; processing the corresponding transformed embedding using a set of convolutional blocks to generate a convolutional output; and processing the convolutional output using a stack of self-attention blocks to generate the encoded text sequence. Optionally, each self-attention block in the stack of self-attention blocks includes the same lightweight convolutional (LConv) block. Each self-attention block in the stack of self-attention blocks includes the same transformer block.

[0013] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description, the drawings, and the claims. Attached Figure Description

[0014] Figure 1 It is an example system for training deep neural networks to provide spectrograms of predicted text utterances in non-autoregressive TTS models.

[0015] Figure 2 This is a schematic diagram of a non-autoregressive neural network used to train a non-autoregressive TTS model.

[0016] Figure 3 Is implemented by Figure 2 A schematic diagram of a non-autoregressive TTS model trained by a non-autoregressive deep neural network.

[0017] Figure 4 This is a schematic diagram of an example convolutional block.

[0018] Figure 5 This is a schematic diagram of an example lightweight convolutional block.

[0019] Figure 6This is a flowchart illustrating an exemplary operational arrangement for a computer-implemented method of training a non-autoregressive text-to-speech model.

[0020] Figure 7 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described in this paper.

[0021] The same reference numerals in the various figures indicate the same elements. Detailed Implementation

[0022] The synthesis of realistic human speech is an unresolved problem because the same text input yields an infinite number of plausible spoken implementations. While end-to-end neural network-based approaches are continuously advancing to match the human performance of short, assistant-like utterances, neural network models are sometimes considered more difficult to interpret or control than more traditional models, which involve multiple processing steps, each operating on fine-grained linguistic or phonetic representations. Sources of speech variation include prosodic features such as intonation, stress, rhythm, and style, as well as speaker and vocal tract characteristics. The prosodic features of spoken utterance convey linguistic, semantic, and emotional meanings that extend beyond those present in lexical representations (e.g., transcription of spoken utterance).

[0023] Neural networks are machine learning models that use one or more nonlinear units to predict the output of a received input. For example, a neural network-based end-to-end text-to-speech (TTS) model can convert input text into output speech. Neural network TTS models offer the potential for robust synthesized speech by predicting linguistic factors that correspond to prosody not provided in the text input. As a result, many applications, such as audiobook narration, news readers, voice design software, and conversation assistants, are able to produce synthesized speech that sounds realistic and not monotonous.

[0024] Many neural end-to-end TTS models utilize autoregressive models that predict current values ​​based on previous values. For example, many autoregressive models are based on recurrent neural networks (RNNs), which use some or all of the network's internal state from the previous time step to compute the output for the current time step. An example of a RNN is a Long Short-Term Memory (LSTM) neural network that includes one or more LSTM memory blocks. Each LSTM memory block can include one or more units, each including an input gate, a forget gate, and an output gate that allows the unit to store its previous state, such as that used to generate the current activation or fed into other components of the LSTM neural network.

[0025] While autoregressive TTS models can synthesize text and generate highly natural speech output, their architecture, through a series of LSTM-based unidirectional decoder blocks (with soft attention), inherently makes training and inference less efficient when implemented on modern parallel hardware compared to fully feedforward architectures. Furthermore, because autoregressive models are trained by a teacher-forced application of ground truth labels at each time step, they are additionally prone to biases introduced between the training and inference processes. Along with the soft attention mechanism, these biases can lead to a deterioration in the quality of synthesized speech output, such as robust errors like gibberish, premature interruptions, word repetitions, and word skipping. This deterioration in synthesized speech quality in autoregressive TTS models can be further exacerbated by increasing the size of the synthesized text.

[0026] To mitigate the aforementioned drawbacks of autoregressive TTS models, the implementation described in this paper targets non-autoregressive neural TTS models enhanced with a residual encoder based on a variational autoencoder (VAE). It is evident that a VAE-based residual encoder can decipher latent representations / states from a reference audio signal conveying residual information, such as stylistic / prosodic information that cannot be represented by the input text to be synthesized (e.g., a phoneme sequence) or the speaker identifier (ID) of the speaker uttering the reference audio signal. In other words, the latent representation makes the output synthesized speech produced by the TTS model sound like the reference audio signal input to the residual encoder.

[0027] A non-autoregressive neural TTS model enhanced with a VAE-based residual encoder provides a controllable model for predicting Mel spectrogram information (e.g., predicting Mel spectrogram sequences) for input text utterances, while effectively controlling the prosody / style represented in the Mel spectrogram information. For example, using selected variational embeddings learned by a VAE-based residual encoder to represent the expected prosody / style for synthesizing text utterances into expressive speech, the spectrogram decoder of the TTS model can predict the Mel spectrogram of the text utterance and feed the Mel spectrogram as input to a synthesizer (e.g., a waveform synthesizer or vocoder network) to convert it into a temporal audio waveform indicating synthesized speech with the expected prosody / style. It is evident that the non-autoregressive TTS model is trained only on sample input text sequences and corresponding reference Mel spectrogram sequences of human speech, enabling the trained TTS model to convert input text utterances into Mel spectrogram sequences with the expected prosody / style conveyed through the previously learned variational embeddings.

[0028] Figure 1An example system 100 is shown for training a deep neural network 200 enhanced with a VAE-based residual encoder 180 to provide a non-autoregressive neural TTS model (or simply "TTS model") 300, and for using the TTS model 300 to predict spectrograms (i.e., Mel spectrogram sequences) 302 for text utterances 320. System 100 includes a computing system 120 having data processing hardware 122 and memory hardware 124 communicating with the data processing hardware 122 and storing instructions that cause the data processing hardware 122 to perform operations. In some embodiments, the computing system 120 (e.g., data processing hardware 122) or user computing device 10 implementing the trained TTS model 300 provides a synthesizer 155 with the predicted Mel spectrogram 302 predicted by the TTS model 300 based on the input text utterances 320 for conversion into a time-domain audio waveform indicative of synthesized speech 152, which can be audibly output as a spoken representation of the input text utterances 320. The time-domain audio waveform includes an audio waveform that defines the amplitude of the audio signal over time. The synthesizer 155 can be separately trained and tuned on a Mel spectrogram for conversion into a time-domain audio waveform.

[0029] The Mel spectrogram comprises a frequency domain representation of the sound. The Mel spectrogram emphasizes low frequencies, which are crucial for speech intelligibility, while neglecting high frequencies, which are dominated by fricatives and other burst noise and generally do not require high-fidelity modeling. Synthesizer 155 may include a vocoder neural network, which can include any network configured to receive the Mel spectrogram and generate audio output samples (e.g., time-domain audio waveforms) based on it. For example, vocoder network 155 can be based on a parallel feedforward neural network as described in van den Oord, Parallel WaveNet: Fast High-Fidelity Speech Synthesis, available at https: / / arxiv.org / pdf / 1711.10433.pdf, which is incorporated herein by reference. Alternatively, vocoder network 155 can be an autoregressive neural network. Synthesizer 155 may include a waveform synthesizer, such as a Griffm-Lim synthesizer or a trainable spectrogram-to-waveform inverter. The choice of synthesizer 155 has no effect on the prosody / style of the resulting synthesized speech 152, and in fact, only affects the audio fidelity of the synthesized speech 152.

[0030] Since the input text utterance 320 cannot convey the context, semantics, and pragmatics guiding the desired prosody / style of the synthesized speech 152, the TTS model 300 can apply variational embedding 220 as a latent variable specifying the desired prosody / style to predict the Mel spectrogram 302 of the text utterance 320 conveying the desired prosody / style specified by the variational embedding 220. In some examples, the computing system 120 implements the TTS model 300. Here, a user can access the TTS model 300 through the user computing device 10 and provide the TTS model 300 with input text utterance 320 to synthesize expressive speech 152 having the desired prosody / style specified by the variational embedding 220. The variational embedding 220 can be selected by the user and corresponds to a previous variational embedding 220 sampled from the residual encoder 180. The variational embedding 220 can be a per-speaker variational embedding 220 selected by the user by providing a speaker identifier (ID) that identifies the speaker speaking in the desired prosody / style (i.e., through an interface executed on the user computing device 10). Here, each speaker ID can be mapped to a corresponding per-speaker variational embedding 220 previously learned by the residual encoder 180. Additionally or alternatively, the user can provide input specifying a particular vertical type associated with a corresponding prosody / style. Here, different vertical types (e.g., news broadcaster, sports broadcaster, etc.) can each be mapped to a corresponding variational embedding 220 previously learned by the residual encoder 180, which conveys the corresponding prosody / style associated with the vertical type. In these examples, the synthesizer 155 can reside on either the computing system 120 or the user computing device 10. When the synthesizer 155 resides on the computing system 120, the computing system 120 can transmit a time-domain audio waveform representing the synthesized speech 152 to the user computing device 10 for audible playback. In other examples, the user computing device 10 implements the TTS model 300. The computing system may include a distributed system (e.g., a cloud computing environment).

[0031] In some implementations, the deep neural network 200 is trained on a large set of reference audio signals 201. Each reference audio signal 201 may include spoken utterances of human speech recorded by a microphone and having a prosodic / style representation. During training, the deep neural network 200 may receive multiple reference audio signals 201 that have the same spoken utterance but different prosodic / style (i.e., capable of uttering the same utterance in a variety of different ways). Here, the reference audio signals 201 have variable lengths, such that even if the content is the same, the duration of the spoken utterances may differ. The deep neural network 200 may also receive multiple sets of reference audio signals 201, each set including reference audio signals 201 for utterances spoken by the same corresponding speaker that have similar prosodic / style but convey different linguistic content. The deep neural network 200, enhanced with a VAE-based residual encoder 180, is configured to encode / compress the prosodic / style representation associated with each reference audio signal 201 into a corresponding variational embedding 220. The variational embedding 220 may include a fixed-length variational embedding 220. The deep neural network 200 can associate each variational embedding 220 with a speaker identity 205 representing the reference speaker emitting the reference audio signal 201 associated with the variational embedding 220. Figure 2 The corresponding speaker embedding y s They are stored together in storage 185 (e.g., on memory hardware 124 of computing system 120). Variational embedding 220 may be per-speaker variational embedding, which includes an aggregation (e.g., average) of multiple variational embeddings 220 encoded by residual encoder 180 based on a reference audio signal 201 spoken by the same speaker.

[0032] During inference, computing system 120 or user computing device 10 can use trained TTS model 300 to predict Mel spectrogram sequence 302 of text utterance 320. TTS model 300 can select variational embeddings 220 representing the expected prosody / style of text utterance 320 from storage 185. Here, variational embedding 220 may correspond to a previous variational embedding 220 sampled from VAE-based residual encoder 180. TTS model 300 can use the selected variational embedding 220 to predict Mel spectrogram sequence 302 of text utterance 320. In the example shown, synthesizer 155 uses the predicted Mel spectrogram sequence 302 to produce synthesized speech 152 with the expected prosody / style specified by variational embedding 220.

[0033] In a non-limiting example, an individual can train a deep neural network 200 to learn per-speaker variational embeddings that convey a prosodic / style representation associated with a particular speaker. For example, Brian McCullough, the host of the Techmeme Ride Home podcast, can train the deep neural network 200 on a reference audio signal 201, which includes previous episodes of the podcast and an input text sequence 206 corresponding to the transcription of the reference audio signal 201. During training, a VAE-based residual encoder 180 can learn a per-speaker variational embedding 220 representing the prosodic / style of Brian's narration of the Ride Home podcast. Brian can then apply this per-speaker variational embedding 220 to a trained TTS model 300 (implemented on computing system 120 or user computing device 10) to predict a Mel spectrogram sequence 302 of text utterances 320 corresponding to the transcription of a new episode of the Ride Home podcast. The predicted metronome sequence 302 can be fed as input to synthesizer 155 to generate synthesized speech 152, which has Brian's unique prosody / style as specified by his per-speaker variational embedding 220. That is, the resulting synthesized speech 152 may sound exactly like Brian's voice and has the prosody / style that Brian uses to narrate episodes of the Ride Home podcast. Therefore, to broadcast a new episode, Brian only needs to provide a transcription of the episode and use a trained TTS model 300 to generate synthesized speech 152, which can be streamed to the loyal listeners of the Ride Home podcast (also known as the MutantPodcast Army).

[0034] Figure 2 A non-autoregressive neural network for training a non-autoregressive TTS model 300 is shown (e.g., Figure 1The deep neural network 200 includes a VAE-based residual encoder 180, a text encoder 210, a duration model network 230, and a spectrogram decoder 260. The deep neural network 200 can be trained on training data comprising multiple reference audio signals 201 and corresponding input text sequences 206. Each reference audio signal 201 comprises spoken utterances of human speech, and the corresponding input text sequence 206 corresponds to a transcription of the reference audio signal 201. In the example shown, the VAE-based residual encoder 180 is configured to encode the reference audio signals 201 into variational embeddings (z) 220. Specifically, the residual encoder 180 receives a reference Mel spectrogram sequence 202 sampled from the reference audio signals 201 and encodes the reference Mel spectrogram sequence 202 into a variational embedding 220, whereby the variational embedding 220 decrypts style / prosodic information from the reference audio signals 201 corresponding to spoken utterances of human speech. Thus, variational embedding 220 corresponds to the latent states of the reference speaker, such as emotion and intention, which contribute to the reference speaker's prosody, emotion, and / or speaking style. As used herein, variational embedding 220 includes both style information and prosodic information. In some examples, variational embedding 220 includes a digital vector with a capacity represented by the number of bits in variational embedding 220. The reference Mel spectrogram sequence 202 sampled from the reference audio signal 201 may have a length L. R and dimension D R As used herein, the reference Mel spectrogram sequence 202 comprises multiple fixed-length reference Mel spectrogram frames sampled / extracted from the reference audio signal 201. Each reference Mel spectrogram frame may comprise a duration of five milliseconds.

[0035] The VAE-based residual encoder 180 corresponds to a posterior network capable of unsupervised learning of the latent representation of speaking style (i.e., variational embeddings (z)). Compared to heuristic-based systems, learning variational embeddings using a VAE network provides advantageous unwinding, scaling, and combination properties, thereby simplifying style control. Here, the residual encoder 180 includes a phoneme-level fine-grained VAE network comprising a projection layer with rectified linear units (ReLU) 410 and a stack of lightweight convolutional (LConv) blocks 420 with multi-head attention. The phoneme-level fine-grained VAE network 180 is configured to encode spectrogram frames from a reference Mel spectrogram sequence 202 associated with each phoneme in the input text sequence 206 into corresponding phoneme-level variational embeddings 220. More specifically, the phoneme-level fine-grained VAE network can align the reference Mel spectrogram sequence 202 with each phoneme in the phoneme sequence extracted from the input text sequence 206 and encode the phoneme-level variational embedding sequence 220. Therefore, each phoneme-level variational embedding 220 in the phoneme-level variational embedding 220 sequence encoded by the phoneme-level fine-grained VAE network 180 encodes a corresponding subset of one or more spectrogram frames from a reference Mel spectrogram sequence 202, which includes corresponding phonemes from the phoneme sequence extracted from the input text sequence 206. The phoneme-level fine-grained VAE network 180 can initially associate the reference Mel spectrogram sequence 202 with the speaker embedding y s And the sine position embedding 214 cascade, speaker embedding y s Representing the speaker uttering the utterance associated with the reference Mel spectrogram sequence 202, the sinusoidal positional embedding 214 indicates the phoneme position information for each phoneme in the phoneme sequence extracted from the input text sequence 206. In some examples, the residual encoder 180 includes positional embeddings (not shown) instead of the sinusoidal positional embedding 214. Each sinusoidal positional embedding 214 may include a fixed-length vector containing information about the specific position of the corresponding phoneme in the phoneme sequence extracted from the input text sequence 206. Subsequently, a concatenation is applied to a stack of five (5) 8-head 17x1 LConv blocks 420 to compute attention using the layer-normalized encoded text sequence 219 output from the text encoder 210.

[0036] Figure 5A schematic diagram 500 of an example LConov block (e.g., a stack of LConov block 420) is shown, having a gated linear unit (GLU) layer 502, an LConov layer 504 configured to receive the output of the GLU layer 502, and a feedforward (FF) layer 506. The example LConov block 500 also includes a first residual connection 508 (e.g., a first cascade 508) configured to cascade the output of the LConov layer 504 with the input of the GLU layer 502. The FF layer 506 is configured to receive the first residual connection 508 as input, which cascades the output of the LConov layer 504 with the input of the GLU layer 502. The example LConov block 500 also includes a second residual connection 510 (e.g., a second cascade 510) configured to cascade the output of the FF layer 506 with the first residual connection 508. Example LConv block 500 can perform FF blending in FF layer 506 using the structure ReLU(W1X+b1)W2+b2, where W1 increases the dimension by a factor of 4.

[0037] In other embodiments, the VAE-based residual encoder 180 may use a global VAE network instead of a non-autoregressive deep neural network 200. Figure 2 The text describes a phoneme-level fine-grained VAE network. The global VAE network encodes a reference Mel spectrogram sequence 202 into a utterance-level global variational embedding 220. Here, the global VAE network comprises two stacks of lightweight convolutional (LConv) blocks, each with multi-head attention. Each LConv block in the first and second stacks of the global VAE network 180 may include eight (8) heads. In some examples, the first stack of LConv blocks includes three (3) 17x1 LConv blocks, and the second stack of LConv blocks following the first stack includes five 17x1 LConv blocks interleaved with 3x1 convolutions. This configuration of the two LConv block stacks allows the global VAE network 180 to continuously downsample the latent representation before applying global average pooling to obtain the final global variational embedding 220. Projection layers can project the dimensions of the global variational embedding 220 from the output of the second stack of LConv blocks. For example, the global variational embedding 220 output from the second stack of the LConv block can have an eight (8) dimension, and the projection layer can project that dimension to a thirty-two (32).

[0038] Continue to refer to Figure 2The text encoder 210 encodes the input text sequence 206 into a text encoding sequence 219. The text encoding sequence 219 includes an encoded representation of the sequence of speech units (e.g., phonemes) extracted from the input text sequence 206. The input text sequence 206 may include words, each having one or more phonemes, silences at all word boundaries, and punctuation marks. Therefore, the input text sequence 206 comprises a phoneme sequence, and the text encoder 210 can receive the corresponding lexical embedding for each phoneme in the phoneme sequence from the lexical embedding lookup table 207. Here, the corresponding lexical embedding includes the phoneme embedding. However, in other examples, the lexical embedding lookup table 207 may obtain lexical embeddings instead of phonemes for other types of speech input associated with the input text sequence 206, such as, but not limited to, subphonemes (e.g., senomes), morphemes, word fragments, or words in a utterance. After receiving the corresponding lexical embedding for each phoneme in the phoneme sequence, the text encoder 210 uses the encoder pre-network neural network 208 to process each corresponding lexical embedding to generate a corresponding transformed embedding 209 for each phoneme. Subsequently, a set of convolutional (Conv) blocks 212 can process the corresponding transformed embeddings 209 to generate convolutional outputs 213. In some examples, a set of Conv blocks 212 includes three (3) identical 5x1 Conv blocks. Figure 4 A schematic diagram 400 of an example Conv block is shown, comprising a Conv layer 402, a batch normalization layer 404, and a dropout layer 406. During training, the batch normalization layer 404 may apply batch normalization to reduce internal covariate bias. The dropout layer 406 may reduce overfitting. Finally, the stack of self-attention blocks 218 processes the convolutional output 213 to generate an encoded text sequence 219. In the example shown, the stack of self-attention blocks 218 comprises six (6) transform blocks. In other examples, self-attention blocks 218 may include LConv blocks instead of transform blocks.

[0039] It is worth noting that since each convolutional output 213 flows simultaneously through the stack of self-attention blocks 218, the stack of self-attention blocks 218 is unaware of the position / order of each phoneme in the input text utterance 206. Therefore, in some examples, a sinusoidal positional embedding 214 is combined with the convolutional output 213 to inject the necessary positional information indicating the order of each phoneme in the input text sequence 206. In other examples, an encoded positional embedding is used instead of the sinusoidal positional embedding 214. In contrast, an autoregressive encoder incorporating a recurrent neural network (RNN) inherently considers the order of each phoneme, as each phoneme is parsed sequentially from the input text sequence. However, the text encoder 210, which integrates a stack of self-attention blocks 218 employing multi-head self-attention, avoids the loops of the autoregressive encoder, resulting in a significant reduction in training time and theoretically capturing longer dependencies in the input text sequence 206.

[0040] Continue to refer to Figure 2 Cascade 222 cascades variational embedding 220 from residual encoder 180, encoded text sequence 219 output from text encoder 210, and reference speaker embedding y representing the speaker identity 205 of the reference speaker who will emit the reference audio signal into cascade 224. s The duration model network 230 receives the concatenation 224 and is configured to generate an upsampled output 258 specifying the number of frames from the encoded text sequence 219 of the concatenation 224. In some implementations, the duration model network 230 includes a stack of self-attention blocks 232, followed by two separate small convolutional blocks 234, 238 and a projection with softplus activation 236. In the example shown, the stack of self-attention blocks includes four (4) 3x1LConv blocks 232. (Refer to above) Figure 5 As described, each LCoV block in the stack of self-attention block 232 may include a GLU unit 502, an LCoV layer 504, and an FF layer 506 with residual connections. The stack of self-attention block 232 is based on encoded text sequence 219, variational embedding 220, and reference speaker embedding y. s The cascaded 224 is used to generate the sequence representation V. Here, the sequence representation V represents a sequence of Mx1 column vectors (e.g., V = {v1,...,v...}). k}).

[0041] In some implementations, the first convolutional block 234 generates output 235 based on the sequence representation V, and the second convolutional block 238 generates output 239 based on the sequence representation V. Convolutional blocks 234 and 238 may each comprise a 3x1 Conv block with a kernel width of 3 and an output dimension of 3. A projection with softplus activation 236 can predict the phoneme duration 240 (e.g., {d1,...d2}) of each phoneme represented by the encoded text sequence 219. k Here, the softplus activation 236 receives the sequence representation V as input to predict the phoneme duration 240. The duration model network 230 calculates a global phoneme duration loss 241 (e.g., an L1 loss term 241) between the predicted phoneme duration 240 and the target average duration 245, which is expressed as:

[0042]

[0043] Among them, L dur d represents the global phoneme duration loss 241 (e.g., L1 loss term 241), K represents the total number of phonemes (e.g., lexical units) in the input text sequence 206, and d k The phoneme duration 240 represents a specific phoneme k in the total number of phonemes K, and T represents the total target frame duration from the reference Mel spectrogram sequence 202.

[0044] In some examples, the TTS model 300 is trained based on the global phoneme duration loss 241 in Equation 1. Since the individual target duration for each phoneme is unknown, the duration model network 230 determines the target average duration 245 based on the ratio of the total T frame durations from the entire reference Mel spectrogram sequence 202 to the total number of K phonemes (e.g., lexical units) in the input text sequence 206. That is, the target average duration 245 is the average duration of all phonemes using the reference Mel spectrogram sequence 202 and the input text sequence 206. An L1 loss term 241 is then determined between the predicted phoneme duration 240 and the target average duration 245 determined using the reference Mel spectrogram sequence 202 and the input text sequence 206. In this way, the duration model network 230 learns to predict phoneme duration 240 in an unsupervised manner, without using supervised phoneme duration labels provided from an external aligner. While the external aligner can provide a reasonable alignment between phonemes and Mel-spectrum frames, the length adjuster requires phoneme duration rounding to upsample the phonemes in the input text sequence 206 based on their durations, which can lead to persistent rounding errors. In some cases, supervised duration labels from the external aligner are used during training, and predicted durations are used during inference in training the TTS model 300 ( Figure 2 ) and TTS model 300 ( Figure 3 This creates a phoneme duration bias between the inferences. Furthermore, this rounding operation is non-differentiable, and therefore the error gradient cannot propagate through the duration model network 230.

[0045] Duration model network 230 includes matrix generator 242 to define lexical boundaries based on predicted phoneme duration 240. Matrix generator 242 determines word boundaries (e.g., phoneme boundaries) based on the predicted phoneme duration 240 as shown below:

[0046]

[0047] e k =s k +d k (3)

[0048] In equation 2, s k The beginning of the lexical boundary of a specific phoneme k (also referred to as the start boundary s in this article) k In equation 3, e k This indicates the end of the lexical boundary of a specific phoneme k (also referred to as the ending boundary e in this article). k ). Matrix generator 242 uses the start and end boundaries s from equations 2 and 3. k e k The following formula maps the lexical boundaries into two lexical boundary grid matrices S and E.

[0049] S tk =ts k (4)

[0050] E tk =e k -t (5)

[0051] Equation 4 will define each starting boundary s k Mapping to S tk A grid matrix, which gives the starting boundary s from time t to word k. k The distance. Equation 5 will define the distance for each ending boundary e. k Mapping to E tk A grid matrix, which gives the end boundary e from time t to word k. k The distance. Matrix generator 242 generates the start and end boundaries s. k e k (also collectively referred to as lexical boundaries) k e k And the word boundary s k e kThese are mapped to the start lexical boundary grid matrix S and the end lexical boundary grid matrix E (collectively referred to as grid matrix 243), respectively. Here, the size of grid matrix 243 is T x K, where K represents the number of phonemes in the input text sequence 206 and T represents the number of frames and the total frame duration in the reference Mel spectrogram sequence 202. Matrix generator 242 can generate corresponding start and end boundaries s for each phoneme in the phoneme sequence based on the number of phonemes represented by the encoded text sequence 219 and the number of reference frames in the reference Mel spectrogram sequence 202. k e k Mapped to the corresponding grid matrix 243 (e.g., start word boundary grid matrix S and end word boundary grid matrix E).

[0052] The duration model network 230 includes a first function 244 to learn an interval representation matrix W. In some implementations, the first function 244 includes two (2) projection layers with bias and Swish activation. In the example shown, both projections of the first function 244 are projected and output in dimension 16 (i.e., P = 16). The first function 244 receives a concatenation 237 as input from a concatenator 222. The concatenator 222 concatenates the output 235 from the convolutional block 234 and the grid matrix 243 to generate the concatenation 237. The first function 244 uses the concatenation 237 to project the output 247. Subsequently, the first function 244 utilizes the softplus activation 246 of the output 247 to generate the interval representation matrix W (e.g., a TxK attention matrix) from the projection. The interval representation matrix W can be based on the start and end boundaries s as shown below. k e k The corresponding grid matrix 243 is used for mapping to learn:

[0053] W=Softmax(MLP(S,E,Conv1D(V))) (6)

[0054] Equation 6 uses the Softmax function of the grid matrix 243 (e.g., the start-word boundary grid matrix S and the end-word boundary grid matrix E) and a multi-layer perceptual (MLP)-based learnable function, along with the sequence representation V, to compute the interval representation W (e.g., a TxK attention matrix). The MLP learnable function includes a third projection layer with an output dimension of 1, which is fed into the Softmax activation function, and Conv1D(V) includes a kernel width of 3, an output dimension of 8, batch normalization, and Swish activation. Here, the (k, t)th element of the grid matrix 243 gives the attention probability between the k-th word (e.g., a phoneme) and the t-th frame, and W() is the learnable function that maps the grid matrix 243 to the sequence representation V with small 1D convolutional layers.

[0055] The duration model network 230 can learn the auxiliary attention context tensor C (e.g., C = [C1, ..., C2]) using a second function 248 conditioned on the sequence representation V. p Here, C P This includes a TxK matrix from the auxiliary attention context tensor C. The auxiliary attention context tensor C may include auxiliary multi-head attention class information for the spectrogram decoder 260. The cascade 222 cascades the grid matrix 243 and the output 239 to generate the cascade 249. The second function 248 may include two (2) projection layers with bias and Swish activation. In the example shown, the projection dimension of the second function 248 is 2 (i.e., P = 2) for the output. The second function 248 receives the cascade 249 as input, based on the start and end boundaries s as shown in the following equation. k e k The corresponding grid matrix 243 and sequence representation V are used to generate the auxiliary attention context tensor C:

[0056] C=MLP(S,E,Conv1D(V)) (7)

[0057] Equation 7 uses a multilayer perceptual (MLP)-based learnable function and sequence representation V with a grid matrix 243 (e.g., start-word boundary grid matrix S and end-word boundary grid matrix E) to compute the auxiliary attention context tensor C. The auxiliary attention context tensor C helps smooth optimization and converges stochastic gradient descent (SGD). The duration model network 230 can upsample the sequence representation V to an upsampled output 258 with the number of frames (e.g., O = {o1, ..., o2}). T Here, the number of frames in the upsampled output 258 corresponds to the prediction length of the predicted Mel spectrogram 302, determined by the predicted phoneme duration 240 of the corresponding input text sequence 206. Upsampling the sequence representation V to the upsampled output 258 involves using a multiplier 253 to determine the product 254 of the interval representation matrix W and the sequence representation V.

[0058] The duration model network 230 uses the Einstein summation operator 255 to determine the einsum 256 of the interval representation matrix W and the auxiliary attention context tensor C. Projection 250 projects the einsum 256 as a projection output 252, which is added to the product 254 at adder 257 to generate an upsampled output 258. Here, the upsampled output 258 can be represented as follows:

[0059] O=WV+[(W⊙C1)1 k …(W⊙C P )1 k A (8)

[0060] Where O represents upsampling output 258, ⊙ represents element-wise multiplication, and 1 represents... k Let A represent a Kx1 column vector containing elements all equal to 1, and let A be a PxM projection matrix.

[0061] Continue to refer to Figure 2 The spectrogram decoder 260 is configured to receive the upsampled output 258 of the duration model network 230 as input and generate one or more predicted Mel spectrogram sequences 302 of the input text sequence 206 as output. The spectrogram decoder 260 may include a stack of multiple self-attention blocks 262, 262a-n with multi-head attention. In some examples, the spectrogram decoder 260 includes six (6) eight-head 17x1 LConv blocks with a 0.1 backoff. The spectrogram decoder 260 may include more or fewer than six LConv blocks. (See above reference...) Figure 5 As described, each LCoV block in the stack of self-attention block 232 may include a GLU unit 502, an LCoV layer 504, and an FF layer 506 with residual connections. In other examples, each self-attention block 262 in the stack includes the same transformer block.

[0062] In some implementations, the spectrogram decoder 260 generates corresponding predicted Mel spectrogram sequences 302, 302a-n as the output of each self-attention block 262 in the stack of self-attention blocks 262, 262a-n. The network 200 can be trained such that the number of frames in each corresponding predicted Mel spectrogram sequence 302 is equal to the number of frames in the reference Mel spectrogram sequence 202 input to the residual encoder 180. In the example shown, each self-attention block 262a-n is paired with a corresponding projection layer 264a-n, which projects the output 263 from the self-attention block 262 to generate a corresponding predicted Mel spectrogram sequence 302a-n with dimensions matching those of the reference Mel spectrogram sequence 202. In some examples, projection layer 264 projects 128-bin predicted Mel spectrogram sequences 302. By predicting multiple Mel spectrogram sequences 302a-n, the non-autoregressive neural network 200 can be trained using soft dynamic time warping (soft DTW) loss. In other words, because the predicted Mel spectrogram sequence 302 may have a different length (e.g., number of frames) than the reference Mel spectrogram sequence 202, the spectrogram decoder 260 cannot determine the conventional Laplace loss. Instead, the spectrogram decoder 260 determines the soft DTW loss between the reference Mel spectrogram sequence 202 and the predicted Mel spectrogram sequence 302, which may include different lengths. Specifically, for each corresponding predicted Mel spectrogram sequence 302a-n, the spectrogram decoder 260 determines the corresponding spectrogram losses 270, 270a-n based on the corresponding predicted Mel spectrogram sequence 302 and the reference Mel spectrogram sequence 202. The corresponding spectrogram loss 270 may include a soft DTW loss term recursively determined by the following formula:

[0063]

[0064] In equation 9, r i，j This represents the distance between the reference Mel spectrogram sequence frames from 1 to i and the predicted Mel spectrogram sequence frames from 1 to j, indicating the optimal alignment distance. Here, min γ Including the generalized minimum operation with a smoothing parameter γ, warp includes a warp penalty, and x i and These are the reference Mel spectrogram frame and the predicted Mel spectrogram sequence frame at times i and j, respectively.

[0065] The soft DTW loss term is recursive and computationally intensive, and can include O(T) loss terms. 2The complexity is determined by a fixed opposing bandwidth of 60, a distortion penalty of 128, and a smoothing parameter γ of 0.05. For example, a first spectrogram loss 270a can be determined based on a first predicted Mel spectrogram sequence 302a and a reference Mel spectrogram sequence 202. Here, the first predicted Mel spectrogram sequence 302a and the reference Mel spectrogram sequence 202 can have the same or different lengths. A second spectrogram loss 270b can be determined based on the first predicted Mel spectrogram sequence 302a and the reference Mel spectrogram sequence 202, and so on, until all corresponding spectrogram losses 270a-n are iteratively determined as predicted Mel frequency spectrogram sequences 302a-n. The spectrogram loss 270 including the soft DTW loss term can be aggregated to generate a final soft DTW loss 280. The final soft DTW loss 280 can correspond to the iterative soft DTW loss term 280. The final soft DTW loss 280 can be determined based on any combination of the same and / or different lengths of the predicted Mel spectrogram sequence 302 and the reference Mel spectrogram sequence 202, as shown below:

[0066]

[0067] In equation 10, Including the final soft DTW loss of 280, Including the soft DTW LI spectrogram reconstruction loss of the first iteration in the spectrogram decoder, Including the average duration L1 loss, and D KL This includes the KL divergence between the prior and posterior from the residual encoder. The goal of training the deep neural network 200 is to minimize the final soft DTW loss 280 to reduce the difference in phoneme duration between the predicted Mel spectrogram sequence 302 and the reference Mel spectrogram sequence 202. By minimizing the final soft DTW loss 280, the trained TTS model 300 can generate a predicted Mel spectrogram sequence 302 including the expected prosody / style based on the reference Mel spectrogram sequence 202. Aggregating spectrogram losses 270a-n may include summing the spectrogram losses 270a-n to obtain the final soft DTW loss 280. Optionally, aggregating spectrogram losses 270 may include averaging the spectrogram losses 270.

[0068] The deep neural network 200 can be trained such that the number of frames in each corresponding predicted Mel spectrogram sequence 302 is equal to the number of frames in the reference Mel spectrogram sequence 202 input to the residual encoder 180. Furthermore, the deep neural network 200 is trained such that the data associated with the reference Mel spectrogram sequence 202 and the predicted Mel spectrogram sequence 302 substantially match each other. The predicted Mel spectrogram sequence 302 can implicitly provide a prosodic / style representation of the reference audio signal 201.

[0069] Figure 3 It shows the result of Figure 2 An example of a non-autoregressive TTS model 300 trained with a non-autoregressive deep neural network 200. Specifically, Figure 3 A TTS model 300 is described that uses a selected variational embedding 220 to predict the Mel spectrogram sequence 302 of the input text utterance 320, whereby the selected variational embedding 220 represents the expected prosody / style of the text utterance 320. During inference, the trained TTS model 300 is implemented on the computing system 120 or the user computing device 10, and can use the selected variational embedding 220 to predict the corresponding Mel spectrogram sequence 302 of the input text utterance 320. Here, the TTS model 300 selects the variational embedding 220 representing the expected prosody / style of the text utterance 320 from storage 185. In some examples, the user provides user input instructions indicating the user's desired choice of the expected prosody / style conveyed by the text utterance 320 in the resulting synthesized speech 152, and the TTS model 300 selects the appropriate variational embedding 220 representing the expected prosody / style from the data storage 185. In these examples, the user can select the expected prosody / style by indicating the speaker identity 205 associated with a specific speaker speaking in the expected prosody / style and / or specifying a specific prosodic vertical type corresponding to the expected prosody / style (e.g., news broadcaster, sports broadcaster, etc.). The selected variational embedding 220 may correspond to a previous variational embedding 220 sampled from a VAE-based residual encoder 180. The trained TTS model 300 uses a synthesizer 155 to generate synthesized speech 152 with the expected prosody / style (e.g., the selected variational embedding 220) for the corresponding input text utterance 320. That is, the selected variational embedding 220 may include the expected prosody / style (e.g., news broadcaster, sports broadcaster, etc.) stored on the residual encoder 180. The selected variational embedding 220 conveys the expected prosody / style via the synthesized speech 152 of the input text utterance 320.

[0070] In an additional implementation, the trained TTS model 300 employs a residual encoder 180 during inference to dynamically extract / predict variational embeddings 220 for predicting input text utterances 320 against a Mel spectrogram sequence 302. For example, the residual encoder 180 may receive a reference audio signal 201 (pronounced by a human user) conveying the intended prosody / style (e.g., “say it like this”). Figure 2The TTS model 300 extracts / predicts the corresponding variational embedding 220 representing the expected prosody / style. Subsequently, the trained TTS model 300 can use the variational embedding 220 to efficiently transfer the expected prosody / style conveyed by the reference audio signal 201 to the Mel spectrogram sequence 302 predicted for the input text utterance 320. Therefore, the input text utterance 320 to be synthesized into expressive speech 152 and the reference audio signal 201 conveying the expected prosody / style to be transferred to expressive speech 152 can include different linguistic contents.

[0071] Specifically, text encoder 210 encodes the phoneme sequence extracted from text utterance 320 into an encoded text sequence 219. Text encoder 210 can receive the corresponding lexical embedding for each phoneme in the phoneme sequence extracted from text utterance 320 from lexical embedding lookup table 207. After receiving the corresponding lexical embedding for each phoneme in the phoneme sequence extracted from text utterance 320, text encoder 210 uses encoder pre-network neural network 208 to process each corresponding lexical embedding to generate a corresponding transformed embedding 209 for each phoneme. Subsequently, a set of Conv blocks 212 (e.g., three (3) identical 5x1 Conv blocks) processes the corresponding transformed embeddings 209 to generate a convolutional output 213. Finally, a stack of self-attention blocks 218 processes the convolutional output 213 to generate the encoded text sequence 219. In the example shown, the stack of self-attention blocks 218 includes six (6) transform blocks. In other examples, self-attention blocks 218 may include LConv blocks instead of transform blocks. It is worth noting that since each convolutional output 213 flows simultaneously through the stack of self-attention blocks 218, the stack of self-attention blocks 218 is unaware of the position / order of each phoneme in the input text utterance. Therefore, in some examples, a sinusoidal positional embedding 214 is combined with the convolutional outputs 213 to inject the necessary positional information indicating the order of each phoneme in the input text sequence 206. In other examples, an encoded positional embedding is used instead of the sinusoidal positional embedding 214.

[0072] Continue to refer to Figure 3 The cascade 222 cascades the selected variational embedding 220, the encoded text sequence 219, and the optional reference speaker embedding y. s To generate cascaded 224. Here, refer to the speaker embedding y s The speaker identity 205 can represent a reference speaker emitting one or more reference audio signals 201 associated with the selected variational embedding 220, or another reference speaker identity 205 having speech features to be conveyed in the resulting synthesized speech 152. The duration model network 230 is configured to process the encoded text sequence 219, the selected variational embedding 220, and the reference speaker embedding y. sThe cascade 224 is decoded to generate an upsampled output 258 of the duration model network 230, which is used to predict the phoneme duration 240 for each phoneme in the phoneme sequence of the input text utterance 320.

[0073] The duration model network 230 is configured to generate an upsampled output 258 that specifies the number of frames from the encoded text sequence 219 of the concatenation 224. In some implementations, the duration model network 230 includes a stack of self-attention blocks 232, followed by two separate small convolutional blocks 234, 238 and a projection with softplus activation 236. In the example shown, the stack of self-attention blocks includes four (4) 3x1 LConv blocks 232. (Refer to above...) Figure 5 As described, each LCoV block in the stack of self-attention block 232 may include a GLU unit 502, an LCoV layer 504, and an FF layer 506 with residual connections. The stack of self-attention block 232 is based on encoded text sequence 219, variational embedding 220, and reference speaker embedding y. s The cascaded 224 is used to generate the sequence representation V.

[0074] In some implementations, a first convolutional block 234 generates output 235 based on sequence representation V, and a second convolutional block 238 generates output 239 based on sequence representation V. Convolutional blocks 234 and 238 may include 3x1 Conv blocks having a kernel width of 3 and an output dimension of 3. The duration model network 230 includes projections with softplus activation 236 for processing encoded text sequences. 21 9 indicates that each phoneme predicts a phoneme duration of 240 (e.g., {d1, ..., d2}). K Here, softplus activation 236 receives the sequence representation V as input to predict the phoneme duration 240.

[0075] Duration model network 230 includes matrix generator 242 to define lexical boundaries from predicted phoneme durations 240. As referenced above Figure 2As described. The duration model network 230 includes a first function 244 to learn an interval representation matrix W. In some implementations, the first function 244 includes two (2) projection layers with bias and Swish activation. In the example shown, both projections of the first function 244 are projected and output in dimension 16 (i.e., P = 16). The first function 244 receives a concatenation 237 as input from a concatenator 222. The concatenator 222 concatenates the output 235 from the convolutional block 234 and the grid matrix 243 to generate the concatenation 237. Subsequently, the first function 244 uses Equation 6 to generate the interval representation matrix W (e.g., a TxK attention matrix) from the projection of the softplus activation 246 with the output 247.

[0076] The duration model network 230 can learn the auxiliary attention context tensor C (e.g., C = [C1, ..., C2]) using a second function 248 conditioned on the sequence representation V. P Here, C p It is a TxK matrix from the auxiliary attention context tensor C. The auxiliary attention context tensor C may include information about the auxiliary multi-head attention class used for the spectrogram decoder 260. The cascade 222 cascades the grid matrix 243 and the output 239 to generate the cascade 249. The second function 248 may include two (2) projection layers with bias and Swish activation. The second function 248 receives the cascade 249 as input to use Equation 7 based on the start and end boundaries s k e k The corresponding grid matrix 243 and sequence representation V are mapped to generate the auxiliary attention context tensor C.

[0077] The duration model network 230 can upsample the sequence representation V into an upsampled output 258 with the number of frames (e.g., O = {o1, ..., o2}). T Here, the number of frames in the upsampled output corresponds to the prediction length of the predicted Mel spectrogram 302, determined by the predicted phoneme duration 240 of the corresponding input text sequence 206. Upsampling the sequence representation V to the upsampled output 258 is based on determining the product 254 of the interval representation W and the sequence representation V using multiplier 253, and determining the einsum 256 of the interval representation matrix W and the auxiliary attention context tensor C using einsum operator 255. Projection 250 projects einsum 356 into the projection output 252, and adder 257 adds the projection output 252 to the product 254 to generate the upsampled output 258 represented by Equation 8.

[0078] The spectrogram decoder 260 is configured to generate a predicted Mel spectrogram sequence 302 of text utterance 320 based on the upsampled output 258 of the duration model network 230 and the predicted phoneme duration 240. Here, the predicted Mel spectrogram sequence 302 has an expected prosody / style specified by the selected variational embedding 220. The predicted Mel spectrogram sequence 302 for text utterance 320 is based on the auxiliary attention context tensor C, the sequence representation V, the interval representation w, and the upsampled output 258 of the duration model network 230 to the number of frames.

[0079] The spectrogram decoder 260 generates a corresponding predicted Mel spectrogram sequence 302 as the output of the last self-attention block 262 in the stack of self-attention blocks 262a_n. Here, each self-attention block 262 in the stack of self-attention blocks 262 of the spectrogram decoder 260 includes one of the same LConv blocks or the same transformer blocks. In some examples, the spectrogram decoder 260 includes six (6) eight-head 17x1 LConv blocks with a 0.1 backoff. The last feedforward (FF) layer 506 of each self-attention block 262 ( Figure 5 The output of ) is provided as the input to subsequent self-attention blocks 262. That is, the GLU unit 502 of the first self-attention block 262a in the stack of self-attention blocks 262a-n ( Figure 5 ) and LConv layer 504 ( Figure 5 The output 238 from the duration model network 230 and the predicted phoneme duration 240 are processed, and the output of the last FF layer 506 from the first self-attention block 262a is provided as input to the subsequent second self-attention block 262b in the stack of self-attention blocks 262. The output of the last FF layer of each self-attention block 262 is provided as input to the subsequent self-attention block 262, until the last self-attention block 262n in the stack of self-attention blocks 262 is reached. The last self-attention block 262n in the stack of self-attention blocks 262 (e.g., the sixth self-attention block 262) is paired with the corresponding projection layer 264, which projects the output 263 from the last self-attention block 262 to generate the corresponding predicted Mel spectrogram sequence 302.

[0080] The predicted Mel spectrogram sequence 302 generated by the spectrogram decoder 260 corresponds to the input text utterance 320 and conveys the expected prosody / style indicated by the selected variational embedding 220. The trained TTS model 300 provides the predicted Mel spectrogram sequence 302 of the input text utterance 320 to the synthesizer 155 for conversion into a time-domain audio waveform indicating the synthesized speech 152. The synthesized speech 152 can be audibly output as a spoken representation of the input text utterance 320, which includes the expected prosody / style indicated by the selected variational embedding 220.

[0081] Figure 6 This is a flowchart of an exemplary operational arrangement of a computer-implemented method 600 for training a non-autoregressive text-to-speech (TTS) model. At operation 602, method 600 includes obtaining a sequence representation V of an encoded text sequence 219 concatenated with variational embeddings 220. At operation 604, method 600 includes predicting a phoneme duration 240 for each phoneme represented by the encoded text sequence 219 based on the sequence representation V using a duration model network. At operation 606, method 600 includes learning an interval representation matrix W based on the predicted phoneme durations 240 using a first function 244 conditioned on the sequence representation V. At operation 608, method 600 includes learning an auxiliary attention context representation C using a second function 248 conditioned on the sequence representation V.

[0082] At operation 610, method 600 includes upsampling the sequence representation V to a specified number of frames using an interval representation matrix W and an auxiliary attention context representation C, resulting in an upsampled output 258. At operation 612, method 600 includes generating one or more predicted Mel spectrogram sequences 302 for the encoded text sequence 219 based on the upsampled output 258, as output from a spectrogram decoder 260 comprising one or more self-attention blocks 262, 262a-n. At operation 614, method 600 includes determining a final spectrogram loss 280 based on one or more predicted Mel spectrogram sequences 302 and a reference Mel spectrogram sequence 202. At operation 616, method 600 includes training a TTS model 300 based on the final spectrogram loss 280.

[0083] Figure 7 This is a schematic diagram of an example computing device 700 that can be used to implement the systems and methods described in this document. The computing device 700 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the embodiments of the invention described and / or claimed in this document.

[0084] Computing device 700 includes a processor 710, a memory 720, a storage device 730, a high-speed interface / controller 740 connected to the memory 720 and a high-speed expansion port 750, and a low-speed interface / controller 760 connected to a low-speed bus 770 and the storage device 730. Each of components 710, 720, 730, 740, 750, and 760 is interconnected using various buses and may be mounted on a common motherboard or otherwise, as appropriate. The processor 710 is capable of processing instructions for execution within computing device 700, including instructions stored in memory 720 or storage device 730, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 780 coupled to the high-speed interface 740. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, may be used as appropriate. Furthermore, multiple computing devices 700 may be connected, with each device providing a portion of the necessary operation (e.g., as a server group, a set of blade servers, or a multiprocessor system).

[0085] Memory 720 stores information non-temporarily within computing device 700. Memory 720 may be a computer-readable medium, volatile memory(s), or non-volatile memory(s). Non-temporarily stored memory 720 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 700. Examples of non-volatile memory include, but are not limited to: flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0086] Storage device 730 provides mass storage for computing device 700. In some embodiments, storage device 730 is a computer-readable medium. In various embodiments, storage device 730 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or device array, including devices in a storage area network or other configuration. In other embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 720, storage device 730, or memory on processor 710.

[0087] High-speed controller 740 manages bandwidth-intensive operations of computing device 700, while low-speed controller 760 manages less bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 740 is coupled to memory 720, display 780 (e.g., via a graphics processor or accelerator), and high-speed expansion port 750, which can accept various expansion cards (not shown). In some embodiments, low-speed controller 760 is coupled to storage device 730 and low-speed expansion port 790. Low-speed expansion port 790, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or network devices such as switches or routers, for example, via a network adapter.

[0088] The computing device 700 can be implemented in a variety of different forms, as shown in the figure. For example, it can be implemented as a standard server 700a or multiple times in a set of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

[0089] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can be included in one or more computer programs executable and / or interpretable on a programmable system comprising at least one programmable processor, which may be dedicated or general-purpose and coupled to receive and transmit data and instructions from and to a storage system, at least one input device, and at least one output device.

[0090] A software application (i.e., a software resource) can refer to computer software that instructs a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0091] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0092] The processes and logic described in this specification can be executed by one or more programmable processors (also known as data processing hardware) that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic can also be executed by special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are the processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, receiving or transferring data to one or more mass storage devices (such as disks, magneto-optical disks, or optical disks) for storing data, or both. However, a computer does not necessarily need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory can be supplemented by or incorporated into dedicated logic circuitry.

[0093] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer having a display device and optionally a keyboard and a pointing device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen, to display information to the user, and a pointing device such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user, for example, by sending web pages to a web browser on the user's client device in response to a request received from a web browser.

[0094] Various embodiments have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are also within the scope of the appended claims.

Claims

1. A computer-implemented method (600), when performed on data processing hardware (122), causing the data processing hardware (122) to perform operations for training a non-autoregressive text-to-speech TTS model (300), the operations comprising: Obtain the sequence representation of the encoded text sequence (219) concatenated with the variational embedding (220); Using a duration model network (230): Based on the sequence representation, predict the phoneme duration (240) of each phoneme represented by the encoded text sequence (219). Based on predicted phoneme duration (240): The interval representation matrix is ​​learned using a first function (244) conditioned on the sequence representation; and A second function (248) conditioned on the sequence representation is used to learn an auxiliary attention context representation; Determine the product of the interval representation matrix and the sequence representation; Determine the Einstein summation einsum of the interval representation matrix and the auxiliary attention context representation; as well as The sequence representation is upsampled to the specified number of upsampled outputs (258) based on the product of the interval representation matrix and the sequence representation and the projection sum of einsum. Based on the upsampled output (258), one or more predicted Mel spectrogram sequences (302) are generated for the encoded text sequence (219) as the output of a spectrogram decoder (260) that includes one or more self-attention blocks; The final spectrogram loss (280) is determined based on the one or more predicted Mel spectrogram sequences (302) and the reference Mel spectrogram sequence (202); and The TTS model (300) is trained based on the final spectrogram loss (280).

2. The computer-implemented method (600) according to claim 1, wherein, The first function (244) and the second function (248) each include corresponding multi-layer perceptual-based learnable functions.

3. The computer-implemented method (600) according to claim 1, wherein, The operation further includes: The global phoneme duration loss (241) is determined based on the predicted phoneme duration (240) and the average phoneme duration (240). The training of the TTS model (300) is further based on the global phoneme duration loss (241).

4. The computer-implemented method (600) according to claim 3, wherein, Training the TTS model (300) based on the final spectrogram loss (280) and the global phoneme duration loss (241) includes training the duration model network (230) to predict the phoneme duration for each phoneme (240) without using supervised phoneme duration labels extracted from an external aligner.

5. The computer-implemented method (600) according to claim 1, wherein, The operation further includes using the duration model network (230): Based on the predicted phoneme duration (240), corresponding start and end boundaries are generated for each phoneme represented by the encoded text sequence (219); Based on the number of phonemes represented by the encoded text sequence (219) and the number of reference frames in the reference Mel spectrogram sequence (202), the corresponding start and end boundaries generated for each phoneme are mapped to the corresponding grid matrix (243). Wherein, the interval representation is learned based on the corresponding grid matrix (243) mapped from the start boundary and the end boundary; and The learning of the auxiliary attention context representation is based on the corresponding grid matrix (243) mapped from the start boundary and the end boundary.

6. The computer-implemented method (600) according to claim 1, wherein, The operation further includes: Receive training data including a reference audio signal (201) and a corresponding input text sequence (206), wherein the reference audio signal (201) includes spoken utterances and the input text sequence (206) corresponds to a transcription of the reference audio signal (201); The reference audio signal (201) is encoded into a variational embedding (220) using a residual encoder (180), which decrypts style / prosodic information from the reference audio signal (201); and The input text sequence (206) is encoded into the encoded text sequence (219) using a text encoder (210).

7. The computer-implemented method (600) according to claim 6, wherein: The residual encoder (180) includes a global variational autoencoder (VAE); as well as Encoding the reference audio signal (201) into the variational embedding (220) includes: Sample the reference Mel spectrogram sequence (202) from the reference audio signal (201); as well as The reference Mel spectrogram sequence (202) is encoded into the variational embedding (220) using the global VAE.

8. The computer-implemented method (600) according to claim 6, wherein: The residual encoder (180) includes a phoneme-level fine-grained variational automatic encoder (VAE); as well as Encoding the reference audio signal (201) into the variational embedding (220) includes: Sample the reference Mel spectrogram sequence (202) from the reference audio signal (201); Align the reference Mel spectrogram sequence (202) with each phoneme in the phoneme sequence extracted from the input text sequence (206); and Based on aligning the reference Mel spectrogram sequence (202) with each phoneme in the phoneme sequence, the phoneme-level fine-grained VAE is used to encode the phoneme-level variational embedding (220) sequence.

9. The computer-implemented method (600) according to claim 6, wherein, The residual encoder (180) includes a stack of lightweight convolutional LCoV blocks (420), each LCoV block (420) in the stack comprising: Gated linear unit (GLU) layer (502); LConv layer (504), the LConv layer is configured to receive the output of the GLU layer (502); Residual connections, configured to cascade the output of the LConv layer (504) with the input of the GLU layer (502); and The final feedforward layer is configured to receive the residual connection of the output of the cascaded LConv layer (504) and the input of the GLU layer (502) as input.

10. The computer-implemented method (600) according to claim 6, wherein, The operation further includes: Cascade the encoded text sequence (219), the variational embedding (220), and the reference speaker embedding representing the identity of the reference speaker emitting the reference audio signal (201); and The sequence representation is generated by receiving the cascade of the encoded text sequence (219), the variational embedding (220), and the reference speaker embedding as inputs from the duration model network (230).

11. The computer-implemented method (600) according to claim 6, wherein: The input text sequence (206) includes a phoneme sequence; and Encoding the input text sequence (206) into the encoded text sequence (219) includes: Receive the corresponding embedding for each phoneme in the phoneme sequence from the phoneme lookup table; For each phoneme in the phoneme sequence, the encoder pre-network neural network of the text encoder (210) is used to process the corresponding embedding to generate the corresponding transformed embedding (209) of the phoneme. A set of convolutional blocks is used to process the corresponding transformed embedding (209) to generate a convolutional output (213); and The convolution output (213) is processed using a stack of self-attention blocks to generate the encoded text sequence (219).

12. The computer-implemented method (600) according to claim 1, wherein, Each self-attention block in the stack of self-attention blocks comprises the same lightweight convolution (LConv) block.

13. The computer-implemented method (600) according to claim 1, wherein, Each self-attention block in the stack of self-attention blocks includes the same transformer block.

14. A system (100) for training a non-autoregressive text-to-speech (TTS) model (300), the system (100) comprising: Data processing hardware (122); as well as Memory hardware (124) communicating with data processing hardware (122), the memory hardware (124) storing instructions that, when executed by the data processing hardware (122), cause the data processing hardware (122) to perform operations including: Obtain the sequence representation of the encoded text sequence (219) concatenated with the variational embedding (220); Using a duration model network (230): Based on the sequence representation, predict the phoneme duration (240) of each phoneme represented by the encoded text sequence (219). Based on predicted phoneme duration (240): The interval representation matrix is ​​learned using a first function (244) conditioned on the sequence representation; and A second function (248) conditioned on the sequence representation is used to learn an auxiliary attention context representation; Determine the product of the interval representation matrix and the sequence representation; Determine the Einstein summation einsum of the interval representation matrix and the auxiliary attention context representation; as well as The sequence representation is upsampled to the specified number of frames based on the product of the interval representation matrix and the sequence representation, and the projection sum of einsum; Based on the upsampled output, one or more predicted Mel spectrogram sequences are generated for the encoded text sequence (219) as the output of a spectrogram decoder (260) that includes one or more self-attention blocks; The final spectrogram loss (280) is determined based on the one or more predicted Mel spectrogram sequences and the reference Mel spectrogram sequence. as well as The TTS model (300) is trained based on the final spectrogram loss (280).

15. The system (100) according to claim 14, wherein, The first function (244) and the second function (248) each include corresponding multi-layer perceptual-based learnable functions.

16. The system (100) according to claim 14, wherein, The operation further includes: The global phoneme duration loss (241) is determined based on the predicted phoneme duration (240) and the average phoneme duration (240). The training of the TTS model (300) is further based on the global phoneme duration loss (241).

17. The system (100) according to claim 16, wherein, Training the TTS model (300) based on the final spectrogram loss (280) and the global phoneme duration loss (241) includes training the duration model network (230) to predict the phoneme duration for each phoneme (240) without using supervised phoneme duration labels extracted from an external aligner.

18. The system (100) according to claim 14, wherein, The operation further includes using the duration model network (230): Based on the predicted phoneme duration (240), corresponding start and end boundaries are generated for each phoneme represented by the encoded text sequence (219); Based on the number of phonemes represented by the encoded text sequence (219) and the number of reference frames in the reference Mel spectrogram sequence, the corresponding start and end boundaries generated for each phoneme are mapped to the corresponding grid matrix (243). Wherein, the interval representation is learned based on the corresponding grid matrix (243) mapped from the start boundary and the end boundary; and The learning of the auxiliary attention context representation is based on the corresponding grid matrix (243) mapped from the start boundary and the end boundary.

19. The system (100) according to claim 14, wherein, The operation further includes: Receive training data including a reference audio signal (201) and a corresponding input text sequence (206), wherein the reference audio signal (201) includes spoken utterances and the input text sequence (206) corresponds to a transcription of the reference audio signal (201); The reference audio signal (201) is encoded into a variational embedding (220) using a residual encoder (180), which decrypts style / prosodic information from the reference audio signal (201); and The input text sequence (206) is encoded into the encoded text sequence (219) using a text encoder (210).

20. The system (100) according to claim 19, wherein: The residual encoder (180) includes a global variational autoencoder (VAE); as well as Encoding the reference audio signal (201) into the variational embedding (220) includes: The reference Mel spectrogram sequence is sampled from the reference audio signal (201); as well as The reference Mel spectrogram sequence is encoded into the variational embedding (220) using the global VAE.

21. The system (100) according to claim 19, wherein: The residual encoder (180) includes a phoneme-level fine-grained variational autoencoder (VAE); and Encoding the reference audio signal (201) into the variational embedding (220) includes: The reference Mel spectrogram sequence is sampled from the reference audio signal (201); Align the reference Mel spectrogram sequence with each phoneme in the phoneme sequence extracted from the input text sequence (206); and Based on aligning the reference Mel spectrogram sequence with each phoneme in the phoneme sequence, the phoneme-level fine-grained VAE is used to encode the phoneme-level variational embedding (220) sequence.

22. The system (100) according to claim 19, wherein, The residual encoder (180) includes a stack of lightweight convolutional LCoV blocks, each LCoV block in the stack of LCoV blocks (420) comprising: Gated Linear Unit (GLU) layer; LConv layer (504), the LConv layer is configured to receive the output of the GLU layer; Residual connections, configured to cascade the output of the LConv layer (504) with the input of the GLU layer; and The final feedforward layer is configured to receive the residual connection of the output of the cascaded LConv layer (504) and the input of the GLU layer as input.

23. The system (100) according to claim 19, wherein, The operation further includes: Cascade the encoded text sequence (219), the variational embedding (220), and the reference speaker embedding representing the identity of the reference speaker emitting the reference audio signal (201); and The sequence representation is generated by receiving the cascade of the encoded text sequence (219), the variational embedding (220), and the reference speaker embedding as inputs from the duration model network.

24. The system (100) according to claim 19, wherein: The input text sequence (206) includes a phoneme sequence; and Encoding the input text sequence (206) into the encoded text sequence (219) includes: Receive the corresponding embedding for each phoneme in the phoneme sequence from the phoneme lookup table; For each phoneme in the phoneme sequence, the encoder pre-network neural network of the text encoder (210) is used to process the corresponding embedding to generate the corresponding transformed embedding (209) of the phoneme. A set of convolutional blocks is used to process the corresponding transformed embedding (209) to generate a convolutional output (213); and The convolution output (213) is processed using a stack of self-attention blocks to generate the encoded text sequence (219).

25. The system (100) according to claim 14, wherein, Each self-attention block in the stack of self-attention blocks comprises the same lightweight convolution (LConv) block.

26. The system (100) according to claim 14, wherein, Each self-attention block in the stack of self-attention blocks includes the same transformer block.

Images