Residual Adapter for Few-Shot Text-to-Speech Speaker Adaptation

Abstract

A method for residual adapters for few-shot text-to-speech speaker adaptation includes obtaining a text-to-speech (TTS) model configured to convert text into a representation of synthetic speech, where the TTS model is pre-trained with an initial training dataset. The method also includes augmenting the TTS model with a stack of residual adapters. The method includes receiving an adaptation training dataset including one or more spoken utterances spoken by a target speaker, where each spoken utterance in the adaptation training dataset is paired with a corresponding input text associated with a transcription of the spoken utterance. The method also includes adapting the TTS model augmented with the stack of residual adapters to learn how to synthesize speech in the voice of the target speaker by optimizing the stack of residual adapters using the adaptation training dataset while parameters of the TTS model are frozen.

Description

[Technical Field]

[0001] The present disclosure relates to a residual adapter for few-shot text-to-speech speaker adaptation. [Background technology]

[0002] Speech synthesis systems use text-to-speech (TTS) models to generate speech from text input. The generated / synthesized speech should accurately convey the message (intelligibility) and sound like a human voice (naturalness) with the intended prosody (expressiveness). While traditional speech synthesis models are capable of providing intelligible speech, recent advances in neural modeling of speech have significantly improved the naturalness and fidelity of synthesized speech. These neural models are large, with millions of parameters that can be fine-tuned during training. Due to the large size of these models, training can be a long and computationally expensive process. Summary of the Invention

[0003] One aspect of the present disclosure provides a computer-implemented method for implementing a residual adapter for few-shot text-to-speech speaker adaptation. The computer-implemented method is executed by data processing hardware, causing the data processing hardware to perform operations including obtaining a text-to-speech (TTS) model configured to convert text into a representation of synthetic speech, the TTS model being pre-trained with an initial training dataset. The operations further include augmenting the TTS model with a stack of residual adapters. The operations include receiving an adaptation training dataset including one or more spoken utterances spoken by a target speaker, each spoken utterance in the adaptation training dataset being paired with a corresponding input text associated with a transcription of the spoken utterance. The operations include adapting the TTS model augmented with the stack of residual adapters to learn how to synthesize speech in the voice of the target speaker by optimizing the stack of residual adapters using the adaptation training dataset while parameters of the TTS model are frozen.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the TTS model includes a decoder including a corresponding stack of multi-head self-attention layers. In some implementations, augmenting the TTS model with the stack of residual adapters includes inserting the stack of residual adapters into the decoder, wherein each respective residual adapter in the stack of residual adapters is disposed between each multi-head self-attention layer in a corresponding stack of multi-head self-attention layers of the decoder. In these implementations, the selected decoder may include a conformer decoder including a first feedforward module, a multi-head self-attention module, a convolution module, a second feedforward module, and a layer normalization module.

[0005] In some implementations, each spoken utterance in the adaptation training data set includes a corresponding non-synthetic speech representation of the target speaker speaking the spoken utterance. In these implementations, the stack of residual adapters is optimized by: for each spoken utterance in the adaptation training data set, generating a corresponding synthetic speech representation from a corresponding input text paired with the spoken utterance using a TTS model augmented with the stack of residual adapters; determining a loss based on the non-synthetic speech representation of the target speaker speaking the spoken utterance and the synthetic speech representation generated from the input text; and updating parameters of the stack of residual adapters based on the loss. In these implementations, the stack of residual adapters may be further optimized by optimizing a speaker embedding associated with the target speaker.

[0006] The initial training dataset used to pre-train the TTS model may not contain utterances spoken by the target speaker. The input text paired with each corresponding spoken utterance in the adaptation training dataset may include a corresponding sequence of phonemes and a corresponding sequence of graphemes. In some implementations, each respective residual adapter in the stack of residual adapters includes a layer normalization layer, a downprojection layer, a rectified linear unit (ReLU) layer, an upprojection layer, and a residual connection.

[0007] In some implementations, the TTS model includes a Transformer Bidirectional Encoder Representation (BERT) encoder configured to receive input text corresponding to each spoken utterance in the adaptation training dataset and generate an encoded text representation of the corresponding input text utterance. In these implementations, the TTS model may also include a dispersion adapter configured to receive a first concatenated output including a speaker embedding and a stress embedding concatenated with the encoded text representation generated by the BERT encoder and generate an output corresponding to the first concatenated output. In these implementations, the TTS model may also include a duration adapter configured to receive the first concatenated output and predict a phoneme duration for each phoneme in the corresponding input text based on the first concatenated output. Further, in these implementations, the TTS model may include an upsampler configured to receive a second concatenated output including the first concatenated output concatenated with the output generated by the dispersion adapter and upsample the second concatenated output. In these implementations, the TTS model may also include a decoder configured to receive the upsampled second concatenated output and generate a synthetic speech representation of the corresponding input text.

[0008] The initial training data set can pre-train the TTS model to synthesize speech in a first language, where each spoken utterance in the adaptive training data set is spoken in a second language that is different from the first language.

[0009] Another aspect of the present disclosure provides a system for implementing a residual adapter for few-shot text-to-speech speaker adaptation. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining a text-to-speech (TTS) model configured to convert text into a representation of synthetic speech, the TTS model being pre-trained with an initial training dataset. The operations further include augmenting the TTS model with a stack of residual adapters. The operations include receiving an adaptation training dataset including one or more spoken utterances spoken by a target speaker, each spoken utterance in the adaptation training dataset being paired with a corresponding input text associated with a transcription of the spoken utterance. The operations include adapting the TTS model augmented with the stack of residual adapters to learn how to synthesize speech in the voice of the target speaker by using the adaptation training dataset to optimize the stack of residual adapters while parameters of the TTS model are frozen.

[0010] This aspect may include one or more of the following optional features: In some implementations, the TTS model includes a decoder including a corresponding stack of multi-head self-attention layers. In some implementations, augmenting the TTS model with the stack of residual adapters includes inserting the stack of residual adapters into the decoder, wherein each respective residual adapter in the stack of residual adapters is disposed between each multi-head self-attention layer in a corresponding stack of multi-head self-attention layers of the decoder. In these implementations, the decoder may include a conformer decoder including a first feedforward module, a multi-head self-attention module, a convolution module, a second feedforward module, and a layer normalization module.

[0011] In some implementations, each spoken utterance in the adaptation training data set includes a corresponding non-synthetic speech representation of the target speaker speaking the spoken utterance. In these implementations, the stack of residual adapters is optimized by: for each spoken utterance in the adaptation training data set, generating a corresponding synthetic speech representation from a corresponding input text paired with the spoken utterance using a TTS model augmented with the stack of residual adapters; determining a loss based on the non-synthetic speech representation of the target speaker speaking the spoken utterance and the synthetic speech representation generated from the input text; and updating parameters of the stack of residual adapters based on the loss. In these implementations, the stack of residual adapters may be further optimized by optimizing a speaker embedding associated with the target speaker.

[0012] The initial training dataset used to pre-train the TTS model may not contain utterances spoken by the target speaker. The input text paired with each corresponding spoken utterance in the adaptation training dataset may include a corresponding sequence of phonemes and a corresponding sequence of graphemes. In some implementations, each respective residual adapter in the stack of residual adapters includes a layer normalization layer, a downprojection layer, a rectified linear unit (ReLU) layer, an upprojection layer, and a residual connection.

[0013] In some implementations, the TTS model includes a Transformer Bidirectional Encoder Representation (BERT) encoder configured to receive input text corresponding to each spoken utterance in the adaptation training dataset and generate an encoded text representation of the corresponding input text utterance. In these implementations, the TTS model may also include a dispersion adapter configured to receive a first concatenated output including a speaker embedding and a stress embedding concatenated with the encoded text representation generated by the BERT encoder and generate an output corresponding to the first concatenated output. In these implementations, the TTS model may also include a duration adapter configured to receive the first concatenated output and predict a phoneme duration for each phoneme in the corresponding input text based on the first concatenated output. Further, in these implementations, the TTS model may include an upsampler configured to receive a second concatenated output including the first concatenated output concatenated with the output generated by the dispersion adapter and upsample the second concatenated output. In these implementations, the TTS model may also include a decoder configured to receive the upsampled second concatenated output and generate a synthetic speech representation of the corresponding input text.

[0014] The initial training data set can pre-train the TTS model to synthesize speech in a first language, where each spoken utterance in the adaptive training data set is spoken in a second language that is different from the first language.

[0015] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a schematic diagram of an exemplary system for a residual adapter for few-shot text-to-speech speaker adaptation. [Figure 2]FIG. 1 is a schematic diagram of an exemplary text-to-speech model configured with a residual adapter. [Figure 3] FIG. 1 is a schematic diagram of an exemplary Conformer decoder configured with a residual adapter. [Figure 4] FIG. 1 is a schematic diagram of an exemplary residual adapter. [Figure 5] FIG. 1 is a schematic diagram of an exemplary training process for a text-to-speech model configured with a residual adapter. [Figure 6] 1 is a flowchart of an exemplary arrangement of operations for a method of implementing a residual adapter for few-shot text-to-speech speaker adaptation. [Figure 7] FIG. 1 is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0017] Like reference symbols in the various drawings indicate like elements.

[0018] Due to the growing interest in many different commercial custom voice applications (e.g., personalized voice assistants in low-resource scenarios), few-shot speaker adaptation remains an important research area in text-to-speech synthesis (TTS). Typical few-shot speaker adaptation approaches involve adapting a pre-trained multi-speaker backbone acoustic model to a target speaker by fine-tuning all or a partial set of model parameters. However, scaling such approaches to thousands of speakers is difficult because the cost of providing the model typically grows linearly with the number of supported target speakers. Furthermore, in some cases, they require supplementary data (e.g., training data used to pre-train the backbone model) during adaptation to avoid overfitting and / or catastrophic memory loss.

[0019] Embodiments herein are directed to a parameter-efficient, few-shot speaker adaptation approach based on residual adapters for text-to-speech. During adaptation, a pre-trained multi-speaker TTS backbone model is frozen but augmented with simple / lightweight neural modules called residual adapters, whose parameters are optimized to synthesize the voices of target speakers, thereby enabling the same backbone model to be shared among many different speakers in the estimation. The approach described herein is more flexible than the previously mentioned fine-tuning approaches because the layer size of the backbone model is fixed, but the size of the residual adapter can be adjusted depending on the current task. Furthermore, the residual adapters are typically very small, requiring 0.1% of the additional parameters to the backbone model, thereby preventing the provisioning cost from growing linearly with the number of target speakers. Furthermore, because the current approach freezes the backbone model during the speaker adaptation phase, the quality of the speakers included in the backbone model is, by design, unaffected.

[0020] 1 , in some embodiments, audio environment 100 includes a user 10 (also referred to herein as a target speaker 10) delivering spoken utterances 12 to a voice-enabled device 110 (also referred to as a device 110 or a user device 110). The user 10 (i.e., the speaker of the utterance 12) may speak the utterances 12 as queries or commands to solicit a response from the device 110 or to cause the device 110 to perform a task specified by the query. The device 110 is configured to capture sounds from one or more users 10 within audio environment 100. Here, audio sounds may refer to utterances 12 spoken by the user 10 that function as audible queries, commands to the device 110, or audible communications captured by the device 110. A voice-enabled system (e.g., a digital assistant interface) of or associated with the device 110 may address the command queries by responding to the queries and / or causing the commands to be executed.

[0021] Here, device 110 captures audio data 14 corresponding to utterances 12 spoken by user 10. Device 110 may correspond to any computing device associated with user 10 and capable of receiving audio data 14. Some examples of user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, e-book readers, etc.), computers, wearable devices (e.g., smart watches), music players, casting devices, smart appliances (e.g., smart TVs), Internet of Things (IoT) devices, remote controls, smart speakers, etc. Device 110 includes data processing hardware 112 and memory hardware 114 in communication with data processing hardware 112, which stores instructions that, when executed by data processing hardware 112, cause data processing hardware 112 to perform one or more operations related to voice and / or text processing. In some examples, device 110 includes one or more applications (i.e., software applications), each of which may utilize one or more voice processing systems / models 140, 150, 200 associated with device 110 to perform various functions within the application. For example, device 110 includes an assistant application configured to communicate synthesized playback audio 154 (also referred to as synthesized speech 154) to user 10 to converse with user 10 and assist in performing various tasks.

[0022] Device 110 further includes an audio subsystem with an audio capture device (e.g., a microphone) 116 for capturing and converting audio data 14 within audio environment 100 into an electrical signal, and an audio output device (e.g., a speaker) 118 for communicating an audible audio signal (e.g., a synthesized playback signal 154 from device 110). In the illustrated example, device 110 implements a single audio capture device 116, but device 110 may implement an array of audio capture devices 116 without departing from the scope of this disclosure, in which case one or more audio capture devices 116 of the array may communicate with the audio subsystem (e.g., peripherals of device 110) without being physically present on device 110. For example, device 110 may correspond to a vehicle infotainment system that utilizes an array of microphones located throughout the vehicle. Similarly, audio output device 118 may include one or more speakers that are either present on device 110 or in communication with the device, or a combination of one or more speakers that are present on device 110 and one or more other speakers that are physically remote from but in communication with device 110.

[0023] In some examples, device 110 is configured to communicate with remote system 130 over network 120. Remote system 130 may include remote resources 132, such as remote data processing hardware 134 (e.g., a remote server or CPU) and / or remote memory hardware 136 (e.g., a remote database or other storage hardware). Device 110 may utilize remote resources 132 to perform various functions related to voice processing and / or synthesis playback communications. For example, device 110 is configured to perform speech recognition using speech recognition system 140 and / or text-to-speech conversion using TTS system 150 (e.g., using TTS model 200). These systems / models 140, 150, 200 may reside on device 110 (referred to as on-device systems) or may reside remotely (e.g., reside on remote system 130) but be in communication with device 110. In some examples, some of these systems 140, 150, 200 reside locally or on-device, while others reside remotely. In other words, any of these systems 140, 150, 200 may be local, remote, or both in any combination. For example, if the size or processing requirements of a system 140, 150, 200 are significant, that system 140, 150, 200 may reside on a remote system 130. Furthermore, if the device 110 can support the size or processing requirements of one or more systems 140, 150, 200, one or more systems 140, 150, 200 may reside on the device 110 using data processing hardware 112 and / or memory hardware 114. Optionally, one or more systems 140, 150, 200 may reside both locally / on-device and remotely.For example, one or more systems 140, 150, 200 may run on the remote system 130 by default when a connection to the network 120 between the device 110 and the remote system 130 is available, but when the connection is lost or the network 120 is unavailable, the systems 140, 150, 200 instead run locally on the device 110. In some implementations, the TTS model 200 may be a large-scale pre-trained model residing on a server (i.e., the remote system 130) and further comprises a residual adapter 400, which is trained based on a target speaker (i.e., the user 10 associated with the device 110).

[0024] The speech recognition system 140 receives audio data 14 as input and transcribes the audio signal into a transcription 142 as output. Generally speaking, by converting the audio data 14 into a transcription 142, the speech recognition system 140 enables the device 110 to recognize when an utterance 12 spoken by the user 10 corresponds to a query, a command, or some other form of audio communication. That is, the speech recognition system 140 may include natural language understanding (NLU) functionality that performs query interpretation (e.g., semantic analysis) on the transcription 142. The transcription 142 refers to a string of text that the device 110 can then use to generate a response to the query or command. For example, if the user 10 asks the device 110 the question, "What's the weather going to be like today?", the device 110 passes audio data 14 corresponding to the question, "What's the weather going to be like today?" to the speech recognition system 140. The speech recognition system 140 converts the audio data 14 into a transcription 142 that includes the text, "What's the weather going to be like today?" Device 110 may then use the text or portions of the text to determine a response to the query. For example, to determine the weather for the current day (i.e., today), device 110 passes the text (e.g., "What's the weather going to be like today?") or identifies portions of the text (e.g., "weather" and "today") to a search engine. The search engine may then return one or more search results that device 110 interprets to generate a response for user 10.

[0025] In some implementations, device 110 or a system associated with device 110 identifies text 152 (also referred to as text string 152 or input text 152) that device 110 communicates to user 10 in response to a query of spoken utterance 12. Device 110 may then use TTS system 150 to convert text 152 into corresponding synthetic playback audio 154 for device 110 to communicate with user 10 in response to a query of spoken utterance 12 (e.g., audibly communicate with user 10). In other words, TTS system 150 receives text 152 as input and converts the text 152 (e.g., via a series of neural networks) into an output of synthetic playback audio 154, where synthetic playback audio 154 is an audio signal that determines an audible representation of text 152. For example, playback audio 154 is a spoken or narrated version of input text 152. In some examples, input text 152 refers to a string or character string of text in a particular natural language (e.g., English, Spanish, or French). The character string can include letters, numbers, punctuation marks, and / or other special characters. When TTS system 150 generates playback audio 154, playback audio 154 includes synthesized speech that approximates how a human would verbalize the character string that defines input text 152.

[0026] The TTS system 150 (or other speech synthesis system) includes a TTS model 200 (e.g., the TTS model 200 of FIG. 2 ) that utilizes a deep neural network (e.g., an attention-based tachotron network) to generate the synthetic playback audio 154. In some implementations, the TTS model 200 generates a synthetic speech representation 261 based on the input text 152, which is then further converted into the synthetic playback audio 154. In some implementations, the TTS model 200 processes an embedding, which is a coded representation of speech features (e.g., features of the input text 152), to generate an audio waveform (e.g., a time-domain audio waveform that defines the amplitude of an audio signal over time). Once generated, the TTS system 150 communicates the synthetic playback audio 154 to the device 110, enabling the device 110 to output the synthetic playback audio 154. For example, the device 110 audibly outputs the synthetic playback audio 154 of “It's sunny today” from one or more speakers 118. Herein, the TTS model 200 of the TTS system 150 is configured to control speech-related attributes of the synthesized speech 154. In other words, the TTS model 200 is configured to simulate the voice of a human speaker in terms of naturalness, while at the same time being capable of generating diverse synthesized speech by modeling fine-grained latent features.

[0027] In some implementations, the TTS model 200 is augmented with one or more residual adapters 400 (i.e., a stack of residual adapters 400). For example, the TTS model 200 may include a base / backbone model that is trained on a large user dataset for a large number of users. The base model portion of the TTS model 200 may then be frozen, and one or more residual adapters 400 may then be trained to learn how to synthesize speech in the voice of a target speaker (e.g., speaker 10 associated with device 110). The one or more residual adapters 400 may undergo a training process (FIG. 5), where the residual adapters are optimized for the target speaker. The residual adapters 400 may be inserted at various points within the TTS model 200.

[0028] Although FIG. 1 illustrates an embodiment of TTS system 150 in the context of an assistant application, TTS system 150 (e.g., using TTS model 200) may be applied to other text-to-speech scenarios, such as voice search, navigation, or document reading.

[0029] 2 illustrates an exemplary TTS model 200 adapted to synthesize speech in a target speaker's voice using a residual adapter 400. The TTS model 200 may include one or more components, including a speaker embedding and / or word stress embedding 205, a Transformer Bidirectional Representation of Transcription (BERT) encoder 210, a variance adapter 220, a duration adapter 230, an upsampler 240, and / or a decoder 300. The decoder 300 may include multiple multi-head attention layers. For example, the decoder 300 may include a conformer decoder. While this disclosure refers to the decoder 300 as a conformer decoder 300, other decoders with multi-head attention layers are contemplated without departing from the scope of this disclosure. TTS model 200 is configured to receive input text 152 (e.g., in the form of a sequence of phonemes and graphemes) and output a synthetic speech representation 261 (e.g., a mel spectrogram) of input text 152 using various components 205, 210, 220, 230, 240, and / or 300. In some implementations, TTS system 150 can implement a vocoder (not shown) to convert synthetic speech representation 261, which corresponds to the mel spectrogram in the frequency domain, into a time-domain audio waveform corresponding to synthetic playback audio 154. In some examples, TTS system 150 implements a neural network-based vocoder, such as a speaker-independent generic pre-trained WaveRNN-based neural vocoder.

[0030] The speaker embedding and / or word stress embedding 205 may affect the timbre or other elements of the final output synthesized speech 261. The BERT encoder 210 may receive the input text 152 and generate an encoded text representation 211 for the corresponding input text. The dispersion adapter 220 may receive a first concatenated output 212 including the speaker embedding and word stress embedding 205 (e.g., from the speaker embedding and / or word stress embedding 205). The speaker embedding and word stress embedding 205 is concatenated with the encoded text representation 211 from the BERT encoder 210. The dispersion adapter 220 may then generate an output 221 based on the first concatenated output 212. Furthermore, the duration adapter 230 may also be configured to receive the first concatenated output 212 and predict a phoneme duration 231 for each phoneme in the corresponding input text 152 based on the first concatenated output 212. The upsampler 240 may be configured to receive the second concatenated output 213 based on a concatenation of the outputs 221 produced by the distribution adapter 220 and the first concatenated output 212. Based on the predicted phoneme durations 231 for each phoneme in the corresponding input text 152, the upsampler 231 upsamples the second concatenated output 231, and the conformer decoder 300 may generate a synthetic speech representation 261 of the corresponding input text 152 based on the upsampled second concatenated output 241.

[0031] In some implementations, the components 205, 210, 220, 230, 240, 300 (also referred to as sub-models) of the TTS model 200 are optimized during pre-training of the TTS model 200 ( FIG. 5 ). The TTS model 200 can be further adapted to synthesize speech in a target speaker's voice by distributing one or more stacks of residual adapters 400 in the various components 205, 210, 220, 230, 240, 300 of the TTS model 200. While FIG. 2 shows the residual adapter 400 disposed in the dispersion adapter 220, duration adapter 230, and / or conformer decoder 300, the residual adapter 400 may be disposed in any of the components 205, 210, 220, 230, 240, 300. In some implementations, multiple residual adapters 400 can be distributed throughout the TTS model 200. During training of the residual adapter 400, the parameters of components 205, 210, 220, 230, 240, and 300 may be frozen. In other words, the TTS model 200 and the residual adapter 400 may be used to generate output based on training data, but only the residual adapter 400 is optimized based on the loss. Training of the residual adapter 400 is described in more detail below (FIG. 5).

[0032] In some implementations, the TTS model 200 is a non-autoregressive variant of the PnG NAT (Phoneme and Grapheme Non-Attentional Tacotron) that replaces the autoregressive LSTM-based decoder with a non-autoregressive Conformer-based decoder. In these implementations, a Conformer decoder 300 that implements multiple multi-head attention layers, instead of a decoder that implements a long short-term memory (LSTM), is more easily accommodated by the residual adapter 400.

[0033] The TTS model 200 may receive input text 152 corresponding to both phoneme and grapheme sequences and then output a synthetic speech representation 261 as a 128-bit Bimmel spectrogram with a 50-ms frame window and a 12.5-ms frame step. In some implementations, the TTS model 200 includes a PnG BERT (Transformer Bidirectional Encoder Representation) encoder (BERT encoder 210), a 6-stack Conformer decoder 300, a duration-based Gaussian upsampler 240, and convolution-based dispersion adapters 220, 230 that predict log-duration, log-F0, and energy at the phoneme level, such as FastPitch. In these implementations, the encoder output 211 is concatenated with utterance-level speaker and word stress embeddings 205 that are broadcast at the phoneme level before the dispersion adapter 220 and duration adapter 230 apply conditioning. Before the upsampler 240, the logF0 and energy are concatenated to the encoder output 211 with the speaker and word stress embedding 205.

[0034] In some implementations, the TTS model 200 is first pre-trained on a large text corpus, such as Wikipedia, with a masked language model (MLM) target, which is then used to initialize the encoder 210. To prevent possible overfitting, both the phoneme and grapheme embeddings and the bottom four encoder layers 210 are frozen while pre-training the TTS model 200.

[0035] 3 provides an example of a conformer block / layer implemented by the conformer decoder 300 of the TTS model 200. The conformer decoder 300 includes a first half-feedforward layer 310, a second half-feedforward layer 340, and a concatenation operator 305, with a multi-head self-attention block 320 and a convolutional layer 330 positioned between the first half-feedforward layer 310 and the second half-feedforward layer 340. The first half-feedforward layer 310 processes the upsampled second concatenated output 241 for a sequence of input text 152. The multi-head self-attention block 320 then receives the upsampled second concatenated output 241 concatenated with the output of the first half-feedforward layer 310. The convolutional layer 330 subsamples the output of the multi-head self-attention block 320 concatenated with the output of the first half-feedforward layer 310. The second half-feedforward layer 340 then receives the concatenation of the output of the convolutional layer 330 and the multi-head self-attention block 320. The layer normalization module 350 processes the output from the second half-feedforward layer 340. Mathematically, the Conformer block 300 transforms the input features x using the modulation features m to generate the output features y as follows:

number

[0036] In some implementations, the residual adapter 400 is inserted into the conformer decoder 300. Although the residual adapter 400 is shown inserted at the end of the conformer decoder 300, the residual adapter 400 may be inserted into any suitable component(s) of the conformer decoder 300. For example, a stack of residual adapters 400 may be inserted into the multi-head self-attention layer 320 of the conformer decoder 300. In this implementation, each residual adapter 400 in the stack of residual adapters 400 is disposed between each multi-head self-attention layer 320 in the corresponding stack of the multi-head self-attention layer 320 of the conformer decoder 300. Furthermore, as described above, the residual adapter 400 may be disposed in any suitable component(s) of the TTS model 200, thereby adapting the TTS model 200 to synthesize speech in the voice of a target speaker.

[0037] In the example of FIG. 3, a residual adapter 400 can be used to further adapt the output of the Conformer 300 so that the output is a synthesized speech representation 261 of the target speaker trained based on training data corresponding to the target speaker. FIG. 4 shows an example residual adapter 400. The residual adapter may include a layer normalization layer 410, a downprojection layer 420, a rectified linear unit (ReLU) layer 430, an upprojection layer 440, and a residual connection 450. In some implementations, the residual adapter 400 receives an input 405 (i.e., an output from a layer of the Conformer decoder 300). The residual adapter 400 then processes the input 405 via components 410, 420, 430, 440, and / or 450 to generate an output 455. In some implementations, the output 455 is a modified version of the input 405, such that the output 455 corresponds to synthesized speech in the target speaker's voice.

[0038] In some implementations, to adapt a pre-trained TTS model 200 to a new target speaker, the TTS model may be adapted using a residual adapter 400. A residual adapter may include a lightweight neural module inserted between layers of the TTS model 200. As shown in FIG. 4, each residual adapter 400 receives a d-dimensional input vector h∈R d Apply layer normalization to Wdown∈R d×r Down projection to the bottleneck dimension r by ReLU activation, Wup∈ R It may include an up-projection by r×d, residual connection, and has the following form: h0=h+ReLU(Layernom(h)Wdown)Wup (3)

[0039] The residual adapter 400 can be inserted anywhere in the TTS model 200, but in some implementations, the residual adapter 400 is most effective in the conformer decoder 300. Furthermore, in addition to optimizing the residual adapter 400, training may also optimize the speaker embeddings so that the variance adapters 220, 230 (FIG. 2) are properly conditioned by the learned target speaker embedding. The TTS model 200 may be frozen during training / optimization of the residual adapter 400, including updating the running average and variance.

[0040] 5 illustrates a training process 500 for adapting a text-to-speech (TTS) model 200 augmented with a stack of residual adapters 400 to learn how to synthesize speech in a target speaker's voice. In some implementations, the process 500 utilizes a two-step training technique. First, the TTS model 200 is pre-trained with an initial training data set 505 containing spoken utterances from a large pool of speakers to obtain a backbone TTS model 200 and generate synthesized speech in a generic voice. The next step in the two-step training technique involves training a stack of residual adapters 400 on adaptation training data 510, while the parameters of the backbone TTS model 200 are frozen. As a result, the TTS model 200 is adapted by the residual adapters 400 to synthesize speech in the target speaker's voice. The training data 510 may include one or more spoken utterances by a target speaker (e.g., utterance 12 in FIG. 1), with each spoken utterance paired with a corresponding input text 512 associated with a transcription of the spoken utterance.

[0041] 5, process 500 begins by pre-training TTS model 200 using pre-training data 505 (i.e., initial training data 505). Model pre-training is a technique used to initialize a model, which may then be further fine-tuned based on additional training data 510. In the case of TTS model 200, pre-training may include TTS model 200 beginning text-to-speech synthesis in a generic voice. In some implementations, pre-training data 505 does not include any utterances spoken by a target speaker (i.e., non-synthesized speech 513 from training data 510 used to adapt TTS model 200 to synthesize speech in the target speaker's voice). Pre-training data 505 may include spoken utterances paired with textual representations of the spoken utterances.

[0042] Next, the process 500 can adapt the TTS model 200 to synthesize speech in the target speaker's voice using the training data 510, fine-tuning one or more residual adapters 400 of the TTS model 200. Meanwhile, the parameters of the TTS model 200 are frozen after pre-training. That is, the TTS model 200 is used to generate output 515 based on input text 512, but only the residual adapter 400 is optimized based on the determined loss 540. The training process may include fine-tuning any of the components 410, 420, 430, and 440 of the residual adapter 400 (FIG. 4) individually or jointly in an appropriate combination. The process 500 includes providing training input 510 to the TTS model 200. In some implementations, the training input 510 includes multiple spoken utterances from the target speaker (e.g., the spoken utterances 12 of the target speaker 10 in FIG. 1). 1 , which includes a non-synthesized speech representation 513 of the target speaker 10 speaking the spoken utterance 12. Additionally, the training input 510 may include input training text 512 corresponding to a transcription of the target user's spoken utterance 12 (e.g., transcription 152 of utterance 12 in FIG. 1 ). The training data 510, including the input training text 512, may include corresponding phoneme and grapheme sequences. In some implementations, the training data set 510 is entirely different from the initial training data 505. For example, the initial training data 505 may be in a first language and the training data set 510 may be in a second language. This allows the backbone TTS model 200 to be trained with a more extensive set of pre-training data 505, while adaptation using the residual adapter 400 can be trained with a target speaker having a much smaller set of training data 510.

[0043] Upon receiving the training input 510, the TTS model 200, augmented with the residual adapter 400, may generate an output 515 (e.g., a synthesized speech representation 261 from the corresponding input training text 512). The TTS model 200 may process the input training text 512 in a manner described with respect to any of FIGS. 1-4 or in any other suitable manner for text-to-speech conversion. For example, with reference to FIG. 2, the input training text 512 may be received by a BERT encoder 210 of the TTS model 200. The BERT encoder 210 may generate an encoded text representation 211 for the corresponding input text 512. The dispersion adapter 220 may receive a first concatenated output 212 that includes the speaker embedding and stress embedding 205 concatenated with the encoded representation 211 generated by the BERT encoder. The dispersion adapter 220 may then generate an output 221 corresponding to the first concatenation 212. The duration adapter 230 may receive the first concatenated output 212 and, based on the first concatenated output 212, predict the phoneme duration 231 of each phoneme in the corresponding input training text 512. In some implementations, the upsampler 240 receives the second concatenated output 213, which includes the first concatenated output 212 concatenated with the output 221 produced by the dispersion adapter 220. The upsampler 240 may then upsample the second concatenation. The conformer decoder 300 may then receive the upsampled second concatenated output 241 and generate a synthesized speech representation 261 of the corresponding input text 512. In this example, the synthesized speech representation 261 in FIG. 2 may correspond to the output 515 in FIG. 5. Furthermore, when training the residual adapter 400, one or more stacks of the residual adapter 400 may be inserted into any of the components of the TTS model 200 as needed to adapt the TTS model 200 to synthesize speech in the voice of a target speaker.

[0044] In some implementations, the output 515 is used by a loss function 530 to generate a loss 540. That is, the loss function 530 compares the output 515 with an utterance spoken by a target speaker to generate the loss 540, where the loss 540 indicates the discrepancy between the unsynthesized speech 513 corresponding to the spoken utterance from the target speaker (i.e., the target output) and the synthesized speech generated by the TTS model 200 from the input text 512 (i.e., the output 515). The loss function 350 may implement any suitable technique for determining the loss, such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross entropy, hinge loss, multi-class loss, etc.

[0045] The loss 540 may then be fed directly to the TTS model 200. Here, the TTS model 200 is frozen, and thus processing the loss 540 involves only adjusting one or more parameters of the residual adapter 400 to account for the loss 540. In some implementations, one or more of the residual adapters 400 include the speaker embeddings used to condition the speakers in the TTS model 200. For example, the speaker embeddings may be extracted from a reference mel spectrogram of the target speaker, and / or a speaker embedding that most closely resembles the timbre of the target speaker may be adapted from a table of speaker embeddings. Here, optimizing the residual adapter 400 includes optimizing the speaker embedding associated with the target speaker.

[0046] 6 is a flowchart of an exemplary arrangement of operations for a computer-implemented method 600 performing a residual adapter for few-shot text-to-speech speaker adaptation. Method 600 may be performed, for example, by various elements of the exemplary voice environment 100 of FIG. 1 and / or the computing device 700 of FIG. 7. At operation 602, method 600 includes obtaining a text-to-speech (TTS) model 200 configured to convert text 152 into a representation of synthetic speech 261, where the TTS model 200 is pre-trained with an initial training dataset 505. At operation 604, method 600 includes augmenting the TTS model 200 with a stack of residual adapters 400. At operation 606, method 600 includes receiving an adaptation training dataset 510 including one or more spoken utterances 12 spoken by a target speaker 10, where each spoken utterance 12 in the adaptation training dataset 510 is paired with a corresponding input text 512 associated with a transcription of the spoken utterance 12. In operation 608, the method 600 includes adapting the TTS model 200 augmented with the stack of residual adapters 400 to learn how to synthesize speech in the voice of the target speaker 10 by using the adaptation training data set 510 to optimize the stack of residual adapters 400 while the parameters of the TTS model 200 are frozen.

[0047] 7 is a schematic diagram of an exemplary computing device 700 that can be used to implement the systems and methods described herein. Computing device 700 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functions are for illustrative purposes only and are not intended to limit the scope of the present invention as described and / or claimed herein.

[0048] Computing device 700 includes a processor 710, memory 720, a storage device 730, a high-speed interface / controller 740 connecting to memory 720 and a high-speed expansion port 750, and a low-speed interface / controller 760 connecting to a low-speed bus 770 and storage device 730. Each of components 710, 720, 730, 740, 750, and 760 are interconnected using various buses and may be mounted on a common motherboard or otherwise mounted as desired. Processor 710 is capable of processing instructions for execution within computing device 700, including instructions stored in memory 720 or storage device 730, for displaying graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 780 coupled to high-speed interface 740. In other implementations, multiple processors and / or multiple buses may be used as desired, along with multiple memories and memory types. Additionally, multiple computing devices 700 may be connected together, each providing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0049] Memory 720 stores information non-temporarily within computing device 700. Memory 720 may be a computer-readable medium, volatile memory unit(s), or non-volatile memory unit(s). Non-transient memory 720 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0050] The storage device 730 can provide mass storage for the computing device 700. In some implementations, the storage device 730 is a computer-readable medium. In various different implementations, the storage device 730 can be a series of devices including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In further implementations, a computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 720, the storage device 730, or memory in the processor 710.

[0051] The high-speed controller 740 manages the bandwidth-intensive operations of the computing device 700, while the low-speed controller 760 manages the bandwidth-intensive operations. This allocation of roles is merely exemplary. In some implementations, the high-speed controller 740 is coupled to memory 720, a display 780 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 750 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 760 is coupled to a storage device 730 and a low-speed expansion port 790. The low-speed expansion port 790 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be coupled to one or more input / output devices such as a keyboard, pointing device, scanner, etc., or to a network device such as a switch or router, for example, via a network adapter.

[0052] Computing device 700 may be implemented in a number of different forms, as shown in the figure: for example, it may be implemented as a standard server 700a, or multiple times within a group of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

[0053] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0054] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0055] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some embodiments, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0056] The processes and logic flows described herein may be implemented by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operably connected to receive data from or transmit data to them, or both. A computer need not, however, have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0057] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touch screen, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, such as acoustic input, voice input, or tactile input. Additionally, a computer can interact with a user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0058] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (600), when executed by data processing hardware (112), causing the data processing hardware (112) to: obtaining a text-to-speech (TTS) model (200) configured to convert text (152) into a representation of synthetic speech (261), the TTS model (200) being pre-trained with an initial training dataset (505); augmenting said TTS model (200) with a stack of residual adapters (400); receiving an adaptation training dataset (510) including one or more spoken utterances (12) spoken by a target speaker (10), wherein each spoken utterance (12) in the adaptation training dataset (510) is paired with a corresponding input text (512) related to a transcription of the spoken utterance (12); Adapting the TTS model (200) augmented with the stack of residual adapters (400) to learn how to synthesize speech in the voice of the target speaker (10) by optimizing the stack of residual adapters (400) using the adaptation training data set (510) while the parameters of the TTS model (200) are frozen; A computer-implemented method (600) for causing a computer to perform operations including:

2. 2. The computer-implemented method of claim 1, wherein the TTS model includes a decoder that includes a corresponding stack of multi-head self-attention layers.

3. 3. The computer-implemented method of claim 1, wherein extending the TTS model with the stack of residual adapters includes inserting the stack of residual adapters into the decoder, wherein each respective residual adapter in the stack of residual adapters is positioned between each multi-head self-attention layer in a corresponding stack of the multi-head self-attention layers of the decoder.

4. The decoder (300) includes a conformer decoder (300), and the conformer decoder (300) a first feedforward module (310); a multi-head self-attention module (320); a convolution module (330); a second feedforward module (340); A layer normalization module (350).

5. each spoken utterance (12) in the adaptation training data set (510) includes a corresponding unsynthesized speech representation (513) of the target speaker (10) speaking the spoken utterance (12); The stack of residual adapters (400) comprises: for each spoken utterance (12) in the adaptation training data set (510), generating a corresponding synthetic speech representation (515) from the corresponding input text (512) paired with the spoken utterance (12) using the TTS model (200) augmented with the stack of residual adapters (400); determining a loss (540) based on the unsynthesized speech representation (513) of the target speaker (10) speaking the spoken utterance (12) and the synthesized speech representation (515) generated from the input text (512); and updating parameters of a stack of the residual adapters based on the loss.

6. 6. The computer-implemented method (600) of any one of claims 1 to 5, wherein the stack of residual adapters (400) is further optimized by optimizing a speaker embedding associated with the target speaker.

7. 7. The computer-implemented method (600) of claim 1, wherein the initial training data set (505) used to pre-train the TTS model (200) does not include any utterances (12) spoken by the target speaker (10).

8. 8. The computer-implemented method (600) of any one of claims 1 to 7, wherein the input text (512) paired with each corresponding spoken utterance (12) in the adaptation training data set (510) includes a corresponding sequence of phonemes and a corresponding sequence of graphemes.

9. Each respective residual adapter (400) in the stack of residual adapters (400) a layer normalization layer (410); a down-projection layer (420); a rectified linear unit (ReLU) layer (430); an up-projection layer (440); A computer-implemented method (600) according to any one of claims 1 to 8, comprising: a residual connection (450).

10. The TTS model (200) A Transformer Bidirectional Encoder Representation (BERT) encoder (210), comprising: for each spoken utterance (12) in the adaptation training data set (510), receiving the corresponding input text (512); a Bidirectional Transformer Representation (BERT) encoder (210) configured to generate an encoded text representation (211) for the corresponding input text (512); A distribution adapter (220), receiving a first concatenated output (212) comprising speaker embeddings and stress embeddings (205) concatenated with the encoded text representation (211) produced by the BERT encoder (210); a distributing adapter (220) configured to generate an output (221) corresponding to the first concatenated output (212); A duration adapter (230), comprising: receiving the first combined output (212); a duration adapter (230) configured to predict a phoneme duration (231) for each phoneme of the corresponding input text (512) based on the first concatenated output (212); An upsampler (240), receiving a second combined output (213) comprising the first combined output (212) combined with the output (221) produced by the distribution adapter (220); an upsampler (240) configured to upsample the second concatenated output (213); A decoder (300), receiving the upsampled second concatenated output (241); and a decoder (300) configured to generate a synthetic speech representation (261) for the corresponding input text (512).

11. the initial training data set (505) pre-trains the TTS model (200) to synthesize speech in a first language; 11. The computer-implemented method (600) of any one of claims 1 to 10, wherein each spoken utterance (12) in the adaptation training data set (510) is spoken by the target speaker (10) in a second language different from the first language.

12. A system (100), comprising: data processing hardware (112); and memory hardware (114) in communication with the data processing hardware (112), the memory hardware (114) storing instructions that, when executed by the data processing hardware (112), cause the data processing hardware (112) to: obtaining a text-to-speech (TTS) model (200) configured to convert text (152) into a representation of synthetic speech (261), the TTS model (200) being pre-trained with an initial training dataset (505); augmenting said TTS model (200) with a stack of residual adapters (400); receiving an adaptation training dataset (510) including one or more spoken utterances (12) spoken by a target speaker (10), wherein each spoken utterance (12) in the adaptation training dataset (510) is paired with a corresponding input text (512) related to a transcription of the spoken utterance (12); and adapting the TTS model (200) augmented with the stack of residual adapters (400) to learn how to synthesize speech in the voice of the target speaker (10) by optimizing the stack of residual adapters (400) using the adaptation training dataset (510) while parameters of the TTS model (200) are frozen.

13. 13. The system of claim 12, wherein the TTS model includes a decoder that includes a corresponding stack of multi-head self-attention layers.

14. 14. The system of claim 12 or 13, wherein extending the TTS model with the stack of residual adapters includes inserting the stack of residual adapters into the decoder, wherein each respective residual adapter in the stack of residual adapters is positioned between each multi-head self-attention layer in a corresponding stack of the multi-head self-attention layers of the decoder.

15. The decoder (300) includes a conformer decoder (300), which: a first feedforward module (310); a multi-head self-attention module (320); a convolution module (330); a second feedforward module (340); A layer normalization module (350).

16. each spoken utterance (12) in the adaptation training data set (510) includes a corresponding unsynthesized speech representation (513) of the target speaker (10) speaking the spoken utterance (12); The stack of residual adapters (400) comprises: for each spoken utterance (12) in the adaptation training data set (510), generating a corresponding synthetic speech representation (515) from the corresponding input text (512) paired with the spoken utterance (12) using the TTS model (200) augmented with the stack of residual adapters (400); determining a loss (540) based on the unsynthesized speech representation (513) of the target speaker (10) speaking the spoken utterance (12) and the synthesized speech representation (515) generated from the input text (512); and updating parameters of a stack of the residual adapters based on the loss.

17. The system (100) of any one of claims 12 to 16, wherein the stack of residual adapters (400) is further optimized by optimizing a speaker embedding associated with the target speaker.

18. 18. The system (100) of claim 12, wherein the initial training dataset (505) used for pre-training the TTS model (200) does not include any utterances (12) spoken by the target speaker (10).

19. 19. The system (100) of any one of claims 12 to 18, wherein the input text (512) paired with each corresponding spoken utterance (12) in the adaptation training data set (510) includes a corresponding string of phonemes and a corresponding string of graphemes.

20. Each respective residual adapter (400) in the stack of residual adapters (400) a layer normalization layer (410); a down-projection layer (420); a rectified linear unit (ReLU) layer (430); an up-projection layer (440); A system (100) according to any one of claims 12 to 19, comprising: a residual connection (450).

21. The TTS model (200) A Transformer Bidirectional Encoder Representation (BERT) encoder (210), comprising: for each spoken utterance (12) in the adaptation training data set (510), receiving the corresponding input text (512); a Bidirectional Transformer Representation (BERT) encoder (210) configured to generate an encoded text representation (211) for the corresponding input text (512); A distribution adapter (220), receiving a first concatenated output (212) comprising speaker embeddings and stress embeddings (205) concatenated with the encoded text representation (211) produced by the BERT encoder (210); a distributing adapter (220) configured to generate an output (221) corresponding to the first concatenated output (212); A duration adapter (230), comprising: receiving the first combined output (212); a duration adapter (230) configured to predict, based on the first concatenated output (212), a phoneme duration (231) of each phoneme of the corresponding input text (512); An upsampler (240), receiving a second combined output (213) comprising the first combined output (212) combined with the output (221) produced by the distribution adapter (220); an upsampler (240) configured to upsample the second concatenated output (213); A decoder (300), receiving the upsampled second concatenated output (241); a decoder (300) configured to generate a synthetic speech representation (261) for the corresponding input text (512).

22. the initial training data set (505) pre-trains the TTS model (200) to synthesize speech in a first language; 22. The system (100) of claim 12, wherein each spoken utterance (12) in the adaptation training data set (510) is spoken by the target speaker (10) in a second language different from the first language.

Images