Two-stage prosodic transfer

Abstract

A method (500) includes receiving an input text utterance (320) to be synthesized into expressive speech (152) having an intended prosody and a target voice, and generating an intermediate synthesized speech representation (202) of the input text utterance using a first text-to-speech (TTS) model (212). The intermediate synthesized speech representation has the intended prosody. The method further includes providing the intermediate synthesized speech representation to a second TTS model (220) that includes an encoder portion (300) and a decoder portion (400). The encoder portion is configured to encode the intermediate synthesized speech representation into an utterance embedding (204) that specifies the intended prosody. The decoder portion is configured to process the input text utterance and the utterance embedding to generate an output audio signal (280) of the expressive speech having the intended prosody and speaker characteristics of the target voice specified by the utterance embedding.

Description

Technical Field

[0001] This disclosure relates to two levels of phonological prosody transfer. Background Technology

[0002] Speech synthesis systems use text-to-speech (TTS) models to generate speech from text input. The generated / synthesized speech should accurately convey the message (intelligibility) while sounding like human speech with the expected prosody (expressiveness) (naturalness). While traditional cascaded and parametric synthesis models can provide intelligible speech and recent advances in neural modeling of speech have significantly improved the naturalness of synthesized speech, most existing TTS models are ineffective at modeling a wide variety of prosodic styles, resulting in a lack of expressiveness in synthesized speech used in important applications. For example, applications such as conversation assistants and long-form readers expect to produce realistic speech by inductively synthesizing prosodic features not conveyed in the text input, such as intonation, stress, rhythm, and style. For instance, a simple statement can be spoken in many different ways, depending on whether the statement is a question, an answer to a question, whether there is uncertainty in the statement, or whether it conveys any other meaning about the context or situation not specified in the input text.

[0003] In some scenarios, it can be useful to transfer prosodic models, which are modeled from various speakers in a specific prosodic domain / vertical industry, such as news readers, sports commentators, and educational instructors, to existing target speech. Applying target speech to new prosodic domains / vertical industries in this way can be particularly difficult because of the insufficient amount of training data associated with the target speech in the new domain / vertical industry. Summary of the Invention

[0004] One aspect of this disclosure provides a method for synthesizing input text utterance into expressive speech having a desired prosody and a target speech. The method includes: receiving, at data processing hardware, the input text utterance to be synthesized into expressive speech having a desired prosody and a target speech. The method further includes: generating an intermediate synthesized speech representation of the input text utterance by the data processing hardware using a first text-to-speech (TTS) model. The intermediate synthesized speech representation possesses the desired prosody. The method further includes: providing the intermediate synthesized speech representation to a second TTS model by the data processing hardware, the second TTS model including an encoder portion and a decoder portion. The encoder portion is configured to encode the intermediate synthesized speech representation into a utterance embedding specifying the desired prosody. The decoder portion is configured to process the input text utterance and the utterance embedding to generate an output audio signal of expressive speech. The output audio signal has the desired prosody and speaker characteristics of the target speech specified by the utterance embedding.

[0005] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the method further includes sampling a sequence of fixed-length reference frames from an intermediate synthesized speech representation using data processing hardware, the prosodic features representing the expected prosodicity possessed by the intermediate synthesized speech representation. Here, providing the intermediate synthesized speech representation to a second TTS model includes providing a sequence of fixed-length reference frames sampled from the intermediate synthesized speech representation to an encoder portion, whereby the encoder portion is configured to encode the sequence of fixed-length reference frames into a speech embedding. The prosodic features representing the expected prosodicity possessed by the intermediate synthesized speech representation may include duration, pitch profile, energy profile, and / or Mel frequency spectrum profile. In these implementations, the encoder portion can be configured to encode a sequence of fixed-length reference frames into speech embeddings by: for each syllable in the intermediate synthesized speech representation: encoding phoneme-level linguistic features associated with each phoneme in the syllable into a phoneme-based syllable embedding; encoding the fixed-length reference frame associated with the syllable into a frame-based syllable embedding, the frame-based syllable embedding indicating the duration, pitch, and / or energy associated with the corresponding syllable; and encoding the phoneme-based syllable embedding and the frame-based syllable embedding having the following: syllable-level linguistic features associated with the syllable, sentence-level linguistic features associated with the intermediate synthesized speech representation, and word-level linguistic features associated with the word including the corresponding syllable.

[0006] Word-level language features can include word embeddings obtained from a sequence of word embeddings generated from the input text utterance by a bidirectional encoder representation (BERT) model from the transducer. In some examples, the decoder portion is configured to process the input text utterance and utterance embeddings to generate an output audio signal by decoding the corresponding utterance embeddings into a sequence of fixed-length prediction frames that provide a prosodic representation of the input text utterance. Here, the prosodic representation represents the expected prosody specified by the utterance embedding. The second TTS model can be trained such that the number of fixed-length prediction frames decoded by the decoder portion is equal to the number of fixed-length reference frames sampled from the intermediate synthesized speech representation.

[0007] In some examples, the utterance embedding comprises a fixed-length numerical vector. The intermediate synthesized speech representation may include an audio waveform or Mel-frequency spectrum sequence capturing the intended prosody, such that providing the intermediate synthesized speech representation to a second TTS model may include providing the audio waveform or Mel-frequency spectrum sequence to an encoder portion. Here, the encoder portion is configured to encode the audio waveform or Mel-frequency spectrum sequence into an utterance embedding.

[0008] In some implementations, the method further includes: obtaining a speaker embedding representing speaker characteristics of the target speech by data processing hardware; and providing the speaker embedding to a decoder portion of a second TTS model by the data processing hardware to process the input text utterance, the utterance embedding, and the speaker embedding to generate an output audio signal of expressive speech. The intermediate synthesized speech representation generated using the first TTS model may include an intermediate speech that lacks speaker characteristics of the target speech and includes undesirable acoustic artifacts.

[0009] The method may further include: receiving training data at data processing hardware, comprising multiple training audio signals and corresponding transcripts, each training audio signal comprising a utterance of human speech having the expected prosody spoken by a corresponding speaker in a prosodic domain / vertical industry associated with the expected prosody. Each transcript comprises a text representation of the corresponding training audio signal. For each corresponding transcript of the training data: the method further includes: training a first TTS model by the data processing hardware to generate a corresponding reference audio signal, the corresponding reference audio signal comprising a training synthesized speech representation capturing the expected prosody of the corresponding utterance of human speech; training an encoder portion of a second TTS model by the data processing hardware by encoding the corresponding trained synthesized speech representation into a corresponding utterance embedding representing the expected prosody captured by the training synthesized speech representation; training a decoder portion of the second TTS model by the data processing hardware using the corresponding transcript of the training data by decoding the corresponding utterance embedding encoded by the encoder portion into a predicted output audio signal of expressive speech with the expected prosody; generating a gradient / loss between the predicted output audio signal and the corresponding reference audio signal; and backpropagating the gradient / loss through the second TTS model.

[0010] The first TTS model and the second TTS model can be trained separately or jointly. In some examples, the first TTS model includes a first neural network architecture, and the second TTS model includes a second neural network architecture that is different from the first neural network architecture. In other examples, the first TTS model and the second TTS model include the same neural network architecture.

[0011] Another aspect of this disclosure provides a system for synthesizing input text utterance into expressive speech having a desired prosody and a target speech. The system includes data processing hardware and memory hardware, the memory hardware communicating with the data processing hardware and storing instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations. These operations include: receiving input text utterance to be synthesized into expressive speech having a desired prosody and a target speech. These operations also include: generating an intermediate synthesized speech representation of the input text utterance using a first text-to-speech (TTS) model. The intermediate synthesized speech representation possesses the desired prosody. These operations further include: providing the intermediate synthesized speech representation to a second TTS model, the second TTS model including an encoder portion and a decoder portion. The encoder portion is configured to encode the intermediate synthesized speech representation into a utterance embedding specifying the desired prosody. The decoder portion is configured to process the input text utterance and the utterance embedding to generate an output audio signal of expressive speech. The output audio signal has the desired prosody and speaker characteristics of the target speech specified by the utterance embedding.

[0012] This aspect may include one or more of the following optional features. In some implementations, these operations further include sampling a sequence of fixed-length reference frames from the intermediate synthesized speech representation that provides prosodic features representing the expected prosodicity possessed by the intermediate synthesized speech representation. Here, providing the intermediate synthesized speech representation to the second TTS model includes providing a sequence of fixed-length reference frames sampled from the intermediate synthesized speech representation to an encoder portion, whereby the encoder portion is configured to encode the sequence of fixed-length reference frames into a speech embedding. The prosodic features representing the expected prosodicity possessed by the intermediate synthesized speech representation may include duration, pitch profile, energy profile, and / or Mel frequency spectrum profile. In these implementations, the encoder portion can be configured to encode a sequence of fixed-length reference frames into speech embeddings by: for each syllable in the intermediate synthesized speech representation: encoding phoneme-level linguistic features associated with each phoneme in the syllable into a phoneme-based syllable embedding; encoding the fixed-length reference frame associated with the syllable into a frame-based syllable embedding, the frame-based syllable embedding indicating the duration, pitch, and / or energy associated with the corresponding syllable; and encoding the phoneme-based syllable embedding and the frame-based syllable embedding having the following: syllable-level linguistic features associated with the syllable, sentence-level linguistic features associated with the intermediate synthesized speech representation, and word-level linguistic features associated with the word including the corresponding syllable.

[0013] Word-level language features can include word embeddings obtained from a sequence of word embeddings generated from the input text utterance by a bidirectional encoder representation (BERT) model from the transducer. In some examples, the decoder portion is configured to process the input text utterance and utterance embeddings to generate an output audio signal by decoding the corresponding utterance embeddings into a sequence of fixed-length prediction frames that provide a prosodic representation of the input text utterance using the input text utterance. Here, the prosodic representation represents the expected prosody specified by the utterance embedding. The second TTS model can be trained such that the number of fixed-length prediction frames decoded by the decoder portion is equal to the number of fixed-length reference frames sampled from the intermediate synthesized speech representation.

[0014] In some examples, the utterance embedding comprises a fixed-length numerical vector. The intermediate synthesized speech representation may include an audio waveform or Mel-frequency spectrum sequence capturing the intended prosody, such that providing the intermediate synthesized speech representation to a second TTS model may include providing the audio waveform or Mel-frequency spectrum sequence to an encoder portion. Here, the encoder portion is configured to encode the audio waveform or Mel-frequency spectrum sequence into an utterance embedding.

[0015] In some implementations, these operations further include: obtaining a speaker embedding representing speaker characteristics of the target speech; and providing the speaker embedding to a decoder portion of a second TTS model to process the input text utterance, the utterance embedding, and the speaker embedding to generate an output audio signal of expressive speech. The intermediate synthesized speech representation generated using the first TTS model may include an intermediate speech that lacks speaker characteristics of the target speech and includes undesirable acoustic artifacts.

[0016] These operations further include: receiving training data comprising multiple training audio signals and corresponding transcripts, each training audio signal comprising a utterance of human speech having the expected prosody spoken by a corresponding speaker in a prosodic domain / vertical industry associated with the expected prosody. Each transcript comprises a text representation of the corresponding training audio signal. For each corresponding transcript of the training data: these operations further include: training a first TTS model to generate a corresponding reference audio signal comprising a training synthetic speech representation capturing the expected prosody of the corresponding utterance of human speech; training an encoder portion of a second TTS model by encoding the corresponding synthetic speech representation into a corresponding utterance embedding representing the expected prosody captured by the training synthetic speech representation; training a decoder portion of the second TTS model using the corresponding transcript of the training data by decoding the corresponding utterance embedding encoded by the encoder portion into a predicted output audio signal of expressive speech with the expected prosody; generating a gradient / loss between the predicted output audio signal and the corresponding reference audio signal; and backpropagating the gradient / loss through the second TTS model.

[0017] The first TTS model and the second TTS model can be trained separately or jointly. In some examples, the first TTS model includes a first neural network architecture, and the second TTS model includes a second neural network architecture that is different from the first neural network architecture. In other examples, the first TTS model and the second TTS model include the same neural network architecture.

[0018] Details of one or more embodiments of this disclosure are set forth in the following drawings and description. Further aspects, features, and advantages will become apparent from the specification and drawings, and from the claims. Attached Figure Description

[0019] Figure 1 This is a schematic view of an example system used to train a two-level prosodic transfer system to produce expressive speech with the desired prosodic rhythm in the target speech.

[0020] Figure 2A It is incorporated into a text-to-speech model based on variational autoencoders. Figure 1 A schematic view of the rhythmic transfer system.

[0021] Figure 2B It is incorporated into a text-to-speech model based on a sequence-to-sequence prediction network. Figure 1 A schematic view of the rhythmic transfer system.

[0022] Figure 3 yes Figure 2A A schematic view of the encoder part of a text-to-speech model based on a variational autoencoder.

[0023] Figure 4A yes Figure 2A A schematic view of the decoder portion of a text-to-speech model based on a variational autoencoder.

[0024] Figure 4B yes Figure 2B A schematic view of the decoder portion of a text-to-speech model based on a sequence-to-sequence prediction network.

[0025] Figure 5 This is a flowchart illustrating an example of the operational setup of a method for generating an output audio signal of expressive speech from text discourse with a desired prosody.

[0026] Figure 6 This is a schematic view of an example computing device that can be used to implement the systems and methods described in this paper.

[0027] In the various figures, the same reference numerals indicate the same elements. Detailed Implementation

[0028] Text-to-speech (TTS) models, frequently used in speech synthesis systems, are typically given only text input without any reference acoustic representation at runtime. They must then generalize numerous linguistic factors not provided by the text input to produce realistic-sounding synthesized speech. A subset of these linguistic factors is collectively referred to as prosody and can include intonation (pitch variation), stress (emphasis on syllables versus non-emphasis on syllables), duration, loudness, pitch, rhythm, and style. Prosody can indicate the emotional state of speech, the form of speech (e.g., statement, question, command, etc.), the presence of irony or sarcasm, uncertainty about the knowledge of speech, or other linguistic elements that cannot be encoded by the grammar or lexical selection of the input text. Therefore, a given text input with high prosodic variation can produce synthesized speech with local pitch and speech duration variations to convey different semantic meanings, as well as synthesized speech with global pitch trajectory variations to convey different moods and emotions.

[0029] Specific domains / vertical industries, such as but not limited to news readers (e.g., news anchors), sports commentators, and educational instructors, each include utterances spoken by various speakers / voices with different speech characteristics (e.g., male / female, language, accent, etc.) but possessing the same prosodic style associated with that specific domain / vertical industry. For example, the prosodic representation of utterances spoken by a sports commentator can convey a range of emotions, while the prosodic representation of utterances spoken by a news anchor can convey a slower speaking rate and clearer word pronunciation. While recent advances in neural modeling of speech have significantly improved the naturalness of synthesized speech and provided the possibility of stably synthesizing speech by predicting prosodic-corresponding linguistic factors not provided by text input, the task of modeling prosody alone is very difficult because deconstructing prosody from speaker characteristics such as accent is an unusual process. Furthermore, the task is to use a neural network-based prosodic model to model a specific prosodicity using training utterances from a large corpus of various voices belonging to a specific domain / vertical industry associated with a specific prosodicity. Due to the lack of a sufficient number of training utterances spoken by the same speaker in the large corpus, the synthesized speech is inherently generated with imperfect voices containing unwanted / undesirable acoustic artifacts.

[0030] Speech synthesis systems can employ TTS models capable of producing synthesized speech from a specific target speech. For example, in e-reader applications, it might be desirable to generate synthesized speech from input text in the voice of Bob Dylan, or, as another example, a text messaging application could generate a synthesized speech representation of the received text message from a synthesized speech representation of the voice of the person sending the text message. However, training a TTS model to produce synthesized speech with a specific target speech and also with the expected prosody is an unusual task, especially when sufficient training utterances spoken by the speaker of the target speech and with the expected prosody are unavailable. For instance, in the scenario of transcribing the live commentary of the Super Bowl LIV between the San Francisco 49ers and the Kansas City Chiefs to produce synthesized speech of the late news anchor Peter Jennings, it would be desirable for the resulting synthesized speech of Peter Jennings' voice to also possess the prosody characteristic of the sports commentary vertical. While it's possible to sample a large number of Peter Jennings' remarks from recordings of his ABC Tonight World News program from 1983 to 2005, these sampled remarks are of little value as training examples for modeling the prosody within the sports commentary vertical, as they are sampled from a completely different vertical (e.g., the news anchor vertical). To make matters worse, since Peter Jennings has been deceased since August 7, 2005, he cannot provide any valuable training remarks that would include his speech in the prosodic style characteristic of the sports commentary vertical.

[0031] The implementation described herein involves learning specific prosody from a corpus of training utterances spoken by different speakers in various tones and generating synthetic expressive speech from the input text, such that the synthesized expressive speech reproduces the learned specific prosody in the target utterance. Here, training does not require utterances possessing specific prosody in the target utterance. More specifically, the implementation involves a two-stage speech prosody transfer system, wherein the task of a first text-to-speech (TTS) model is to learn only the expected prosody from the corpus of training utterances, while the task of a second TTS model is to transfer the expected prosody learned by the first TTS model to the synthesized speech representation of the target utterance.

[0032] Described in more detail below, the first TTS model is trained to produce an intermediate speech representation that captures the expected prosody without attempting to unravel the expected prosody and speaker characteristics. Therefore, the resulting intermediate synthesized speech representation produced by the first TTS model captures the expected prosody (expressiveness) conveyed in the training utterance, but may include imperfect speech with reduced quality (e.g., noise artifacts) and a lack of speaker characteristics (e.g., accent). Thus, the intermediate synthesized speech representation is not suitable for human listening because it is not intended to accurately convey the message (intelligibility), nor is it intended to sound like human speech (naturalness). Despite the imperfect speech representation and therefore its failure to convey the speaker characteristics representing the target utterance, the second TTS model is trained to reproduce the expected prosody captured by the intermediate speech representation and generate expressive speech with the expected prosody in the target utterance. That is, the second TTS model generates expressive speech with the expected prosody and the speaker characteristics associated with the target utterance. Here, the target utterance can be associated with a male actor who has never spoken any training utterance possessing the expected prosody.

[0033] The second TTS model can correspond to a prosodic transfer model comprising an encoder and a decoder. Here, the prosodic transfer model can correspond to a variational autoencoder (VAE) architecture or a sequence-to-sequence feature prediction network architecture. The encoder is configured to encode the intermediate synthesized speech representation generated by the first TTS model into a utterance embedding specifying the expected prosodicity captured by the intermediate synthesized speech representation; however, the decoder is configured to decode the utterance embedding to predict prosodic features, such as the duration of phonemes in each syllable and pitch and energy profiles. In some examples, the decoder is configured to decode the utterance embedding to predict a Mel-spectral spectrum as a complement or alternative to the prosodic features. The Mel-spectral spectrum can inherently convey the expected prosodicity.

[0034] The first TTS system can be trained on training utterances of human speech and their corresponding transcriptions to generate a training synthetic speech representation from the transcriptions, which captures the prosody of the corresponding training utterances of human speech. The training utterances can be entirely sampled from a specific prosodic vertical industry, such that each training utterance possesses an expected prosody associated with that specific prosodic vertical industry. The encoder portion of the second TTS model can be trained on utterance embeddings representing the captured prosody by encoding numerous training synthetic speech representations generated by the first TTS model conditioned on prosodic features and linguistic feature embeddings representing the training synthetic speech representations. Prosodic features can represent acoustic information about a reference audio signal in terms of pitch (F0), phoneme duration, and energy (C0). For example, prosodic features can include phoneme durations sampled from the reference audio signal, as well as fixed-length frames of pitch and energy. Language features can include, but are not limited to: phoneme-level language features, which contain information about the position, phoneme identification, and number of phonemes in a syllable; syllable-level language features, which contain information such as whether a syllable is identified and whether the syllable is stressed or unstressed; word-level language features that encode syntactic information about each word; and sentence-level language features that contain information about the speaker, the speaker's gender, and / or whether the utterance is a question or a phrase. Language features can be extracted from the corresponding transcription for each training utterance. In some examples, a second TTS model is incorporated into a bidirectional encoder representation (BERT) model from a transformer, configured to output word embeddings. In these examples, word embeddings can replace word-level language features that otherwise explicitly encode syntactic information about each word.

[0035] Each utterance embedding encoded by the encoder portion can be represented by a fixed-length numerical vector. In some implementations, the fixed-length numerical vector includes values ​​equal to 256. However, other implementations may use fixed-length numerical vectors with values ​​greater than or less than 256. For a given input text utterance, the decoder portion can process the input text utterance and the fixed-length utterance embedding to generate an output audio signal of expressive speech. Here, the output audio signal has the expected prosody specified by the utterance embedding. The output audio signal may include a sequence of predicted fixed-length frames (e.g., five milliseconds) of pitch, energy, and / or phoneme duration, or the output audio signal may include Mel-frequency spectrum frames conveying the expected prosody. Additionally, the decoder portion can receive a speaker embedding that provides speaker characteristics of the target speech. Therefore, the output audio signal with the expected prosody may also include speaker characteristics of the target speech. The synthesizer can receive the output audio signal generated by the second TTS model as input and generate a synthesized speech representation of the input text utterance with the expected prosody, spoken by the target speech, as output.

[0036] Figure 1An example system 100 is shown for training a two-level prosodic transfer system 200 to synthesize text utterance 320 into an expressive speech 152 having a prosodic representation 302, the prosodic representation 302 representing an expected prosodicity associated with a specific prosodic vertical industry 20. System 100 includes a computing system (interchangeably referred to as a “computing device”) 120 having data processing hardware 122 and memory hardware 124, the memory hardware 124 communicating with and storing instructions that can be executed by the data processing hardware 122 to cause the data processing hardware 122 to perform operations. In some embodiments, the computing system 120 (e.g., data processing hardware 122) provides a two-level prosodic transfer system 200 that is trained to generate an output audio signal 280 of expressive speech from input text utterance 320, such that the output audio signal 280 has the expected prosodicity and speaker characteristics of the target speech from a specific prosodic vertical industry 20. The output audio signal 280 conveys a prosodic representation 302 representing the expected prosodic rhythm so that the speech synthesizer 150 can generate an audio waveform of synthesized speech 152 with the expected prosodic rhythm in the target speech.

[0037] The prosody transfer system 200 includes a first text-to-speech (TTS) system 210 having a first TTS model 212 and a second TTS system 220 having a second TTS model 222. The first TTS system 210 and the second TTS system 220 may each include a speech synthesizer 150. The first TTS model 212 and the second TTS model 222 may each include corresponding neural network architectures that may be the same or different. The first TTS system 210 is configured to use the first TTS model 212 to generate an intermediate synthesized speech representation 202 of the input text utterance 320. For example, the first TTS model 212 may generate an intermediate output signal 201, such as a Mel frequency spectrum sequence, which captures / possesses the expected prosody from a specific prosodic vertical industry 20. The speech synthesizer 150 can then generate the intermediate synthesized speech representation 202 from the intermediate output signal 201 and provide the intermediate synthesized speech representation 202 to the second TTS model 222. The second TTS system 220 is configured to use a second TTS model 222 to transfer or reproduce the expected prosody captured by the intermediate synthesized speech representation 202 into the output audio signal 280 of the expressive speech to convey the input text utterance 320 as a spoken representation with the expected prosody in the target speech. The second TTS model 222 may receive a speaker embedding Z that includes speaker characteristics of the target speech. In some examples, the second TTS model 222 receives an intermediate output signal 201 (e.g., a Mel frequency spectrum) generated by the first TTS model 222 as a supplement or alternative to the intermediate synthesized speech representation 202 used to transfer the expected prosody into the output audio signal 280 of the expressive speech. Since the input text utterance 320 lacks the ability to convey context, semantics, and pragmatics to guide the expected prosodicity of the synthesized speech 152, the prosodic transfer system 200 can predict the prosodic representation 302 of the input text utterance 320 by conditioning the second TTS model 222 on linguistic features extracted from the text utterance 320 and using a fixed-length utterance embedding 204 as a latent variable representing the expected prosodicity of the text utterance 320. Described in more detail below, the task of the intermediate synthesized speech representation 202 generated by the first TTS system 210 is to capture / possess only the expected prosodicity from a specific prosodic vertical industry 20, enabling the second TTS model 222 to reproduce the expected prosodicity of the text utterance 320 by encoding the intermediate synthesized speech representation 202 into the utterance embedding 204. The computing system 120 may include a distributed system (e.g., a cloud computing environment). The synthesizer 150 may include a vocoder 155.

[0038] In some implementations, system 100 trains multiple prosodic transfer systems 200, 200A-N, each configured to transfer different corresponding expected prosodices from a corresponding prosodic vertical industry 20, 20A-N to the expressive speech 152 of the target speech. For example, each of the different prosodic vertical industries 20 may include utterances spoken by various different humans with different speech characteristics (e.g., male / female, language, accent, etc.) and possessing the same prosodic style associated with the corresponding prosodic vertical industry 20. For example, prosodic vertical industry 20A may correspond to utterances of human speech associated with news readers (e.g., news anchors), prosodic vertical industry 20B may correspond to utterances of human speech associated with sports commentators, and prosodic vertical industry 20N may correspond to utterances of human speech associated with educational lecturers. While sports commentator vertical industry 20B can broadly include utterances sampled from speakers commenting on many different sports, each particular prosodic vertical industry 20 can convey expected prosodices sampled from a narrower set of utterances. For example, it is possible to have many different sports commentary prosodic verticals 20, each corresponding to utterances of human speech associated with a specific sport. This can be advantageous because the prosodic style can vary between utterances spoken by a sports commentator for curling, compared to utterances spoken by a sports commentator for rugby.

[0039] Continue to refer to Figure 1For each prosodic vertical industry 20, 20A-N, computing device 120 (e.g., data processing hardware 122) receives corresponding training data 10 comprising multiple training audio signals 104 and corresponding transcriptions 106. Each training audio signal 104 comprises a utterance of human speech with the expected prosodic rhythm spoken by a corresponding speaker in the prosodic vertical industry (interchangeably referred to as the "prosodic domain") associated with the expected prosodic rhythm. Each transcription 106 comprises a text representation of the corresponding training audio signal 104. For each corresponding transcription 106 of the training data 10, computing device 120 trains a first TTS model 212 to generate a corresponding reference audio signal 202T, which comprises a trained synthesized speech representation capturing the expected prosody of the corresponding utterance 104 of human speech. Subsequently, computing device 120 trains a second TTS model 222 of a second TTS system 220. More specifically, training the second TTS model 222 may include training both the encoder portion 300 and the decoder portion 400 of the second TTS model 222 for each corresponding transcription 106 of the training data 10. Training the encoder portion 300 includes encoding the corresponding training synthetic speech representation 202T (e.g., an audio waveform or Mel frequency spectrum) into a corresponding utterance embedding 204 representing the expected prosody captured by the training synthetic speech representation 202T. Notably, the parameters of the first TTS system 220, and more specifically the first TTS model 222, are optimized to produce only the synthetic speech representation 202T, which accurately captures the prosody of the real-value utterance 104 of human speech, and thus is permitted to include intermediate speech lacking sufficient speech features and having reduced audio quality (e.g., containing acoustic artifacts).

[0040] The training decoder section 400 includes using the corresponding transcription 106 of the training data 10 to decode the utterance embedding 204 encoded by the encoder section 300 into a predicted output audio signal 280 of expressive speech with the expected prosody. That is, the predicted output audio signal 280 is associated with a prosodic representation 302 that reproduces the expected prosody captured from the corresponding ground truth utterance 104 of human speech through training the intermediate synthesized speech representation 202T. The decoder section 400 can be further trained to learn speaker characteristics of a specific target utterance, such that the audio signal 280 of the expressive speech possesses both the expected prosody and the speaker characteristics of the target utterance. In some examples, the first TTS model 212 and the second TTS model 222 are trained jointly. In other examples, the first TTS model 212 and the second TTS model 222 are trained separately.

[0041] Finally, for each training audio signal 104 and corresponding transcription 106 in the training data 10 for the corresponding prosodic vertical industry 20, the computing device 120 (e.g., data processing hardware 122) generates a gradient / loss between the predicted output audio signal 280 and the corresponding reference audio signal 202T, and backpropagates the gradient / loss through the second TTS model 222. Therefore, the computing device 120 can train corresponding prosodic transfer systems 200, 200A-N for each of many different prosodic vertical industries 20, 20A-N, such that each prosodic transfer system 200 is configured to synthesize text utterance 320 into expressive speech 152 of the target speech and having a prosodic representation 302, which represents the expected prosodicity associated with the corresponding specific prosodic vertical industry 20. For example, applying the above examples, the trained prosody transfer system 200A can synthesize expressive speech 152 from target speech with an expected prosody associated with the news reader prosody vertical industry 120A; the trained prosody transfer system 200B can synthesize expressive speech from target speech with an expected prosody associated with the sports commentator prosody vertical industry 20B; and the trained prosody transfer system 200N can synthesize expressive speech from target speech with an expected prosody associated with the education lecturer prosody vertical industry 20N. The computing device 120 can store each trained prosody transfer system 200 on a data storage device 180 (e.g., memory hardware 124) for later use during inference.

[0042] During inference, computing device 120 can use the trained prosodic transfer system 200 to synthesize text utterance 320 into expressive speech 152 with a target speech and a prosodic representation 302, which represents the expected prosodicity associated with a specific prosodic vertical industry 120. The prosodic representation 302 can correspond to predicted prosodic features of pitch, energy, and duration for each phoneme. Specifically, during the first level / stage, the trained prosodic transfer system 200 uses a first TTS model 212 to generate an intermediate synthesized speech representation 202 of the input text utterance 320, whereby the intermediate synthesized speech representation 202 possesses the expected prosodicity and is permitted to have reduced audio quality and lack speaker characteristics. During the second level / stage, the trained prosodic transfer system 200 provides the intermediate synthesized speech representation 202 to a second TTS model 222. Here, the encoder portion 300 of the second TTS model 222 is configured to encode the intermediate synthesized speech representation 202 into a utterance embedding 204 specifying the expected prosody. However, the decoder portion 400 of the second TTS model 222 is configured to process the input text utterance 320 and the utterance embedding 204 to generate an output audio signal 280 of expressive speech. The output audio signal 280 has the expected prosody and speaker characteristics of the target speech specified by the utterance embedding 204. The decoder portion 400 may receive a speaker embedding Z associated with the target speech that conveys speaker characteristics (e.g., accent, male / female, and accent). In the example shown, the speech synthesizer 150 uses the output audio signal 280 to generate synthesized speech 152 from the text utterance 320 and having the expected prosody in the target speech.

[0043] Figure 2AA schematic view of prosodic transfer systems 200, 200a is provided, wherein a second TTS model 222a at the second TTS system 220 corresponds to a second TTS model 222a based on a variational autoencoder (VAE). More specifically, the second TTS model 222a can provide a hierarchical language structure for a clock-device hierarchical variational autoencoder (CHiVE). However, the second TTS model 222a can include other types of VAEs. In the example shown, the first TTS system 210 receives text utterance 320 and optional additional inputs 325 as inputs, which may include speaker characteristics of the target utterance (e.g., speaker embedding Z). The additional inputs 325 may additionally or alternatively include one or more of language identifiers, text normalization, or prosodic vertical industry identifiers corresponding to the prosodic domain. Using the input text utterance and the optional additional inputs 325, the first TTS model 212 generates an intermediate output audio signal 201 comprising a Mel frequency spectrum sequence inherently possessing the expected prosodicity of the input text utterance 320. The first TTS system 210 can synthesize (e.g., using a speech synthesizer 150) the intermediate output audio signal 201 into an intermediate synthesized speech representation 202. As explained above, the intermediate synthesized speech representation 202 is configured to accurately capture the expected prosody and is permitted to include reduced audio quality and a lack of speaker characteristics for identifying the target speech. In other words, the intermediate synthesized speech representation 202 may not necessarily be suitable for human listening, but rather serves as a carrier for conveying the expected prosody for use by the second TTS model 222 to reproduce and incorporate prosodic information into the expressive speech of the target speech.

[0044] Extractor 214 can then sample a sequence of fixed-length reference frames 211 from intermediate synthesized speech representation 202, the prosodic features representing the expected prosodicity possessed by intermediate synthesized speech representation 202. Encoder portions 300, 300a of the VAE-based second TTS model 222a are configured to encode the sequence of fixed-length reference frames 211 into a speech embedding 204 specifying the expected prosodicity. The prosodic features representing the expected prosodicity possessed by intermediate synthesized speech representation 202 may include duration, pitch profile, energy profile, and / or Mel frequency spectrum profile.

[0045] Continue to refer to Figure 2A The decoder portions 400, 400a of the second TTS model 222a based on CHiVE are configured to process the input text utterance 320 and the utterance embedding 204 to generate the output audio signal 280 by decoding the corresponding utterance embedding 204 into a sequence of fixed-length prediction frames 280 that provide the prosodic representation 302 of the input text utterance 320.

[0046] Figure 3 and Figure 4A This illustrates a controllable model for providing prosodic transfer. Figure 2A The hierarchical language structure of the second TTS model 222a based on CHiVE. Model 222a can jointly predict the duration of each syllable of a given input text 320, as well as the pitch (F0) and energy (C0) profile of that syllable, without relying on any unique mapping from a given input text or other language norms to produce synthesized speech 152 with the expected prosody in the target speech. The second TTS model 222a includes: encoder part 300a ( Figure 2A and Figure 3 ), which encodes multiple fixed-length reference frames 211 sampled from the intermediate synthesized speech representation 202 (or from the intermediate output) into a fixed-length speech embedding 204; and a decoder section 400a ( Figure 2A and Figure 4A The second TTS model 222a learns how to decode a fixed-length utterance embedding 204. The decoder section 400a decodes the fixed-length utterance embedding 204 into an output audio signal 280 of expressive speech, which may include multiple fixed-length prediction frames 280 (e.g., to predict the pitch (F0), energy (C0), or spectral characteristics (M0) of the utterance embedding 204). As will become apparent, the second TTS model 222a is trained such that the number of prediction frames 280 output from the decoder section 400a is equal to the number of reference frames 211 input to the encoder section 300a. Furthermore, the second TTS model 222a is trained such that the prosodic information associated with the reference frames 211 and the prediction frames 280 substantially matches each other.

[0047] refer to Figure 2A and Figure 3The encoder section 300a receives a sequence of fixed-length reference frames 211 sampled from the intermediate synthesized speech representation 202 output from the first TTS system 210. The intermediate synthesized speech representation 202 captures the expected prosody of the input text utterance 320. Each reference frame 211 may include a duration of 5 milliseconds (ms) and represent one of the profiles of pitch (F0) or energy (C0) (and / or spectral characteristics (M0)) of the intermediate synthesized speech representation 202. In particular, the encoder section 300a may also receive a second sequence of reference frames 211, each including a duration of 5 ms and representing the other of the profiles of pitch (F0) or energy (C0) (and / or spectral characteristics (M0)) of the intermediate synthesized speech representation 202. Thus, the sequence of reference frames 211 sampled from the intermediate synthesized speech representation 202 provides duration, pitch profile, energy profile, and / or spectral characteristic profile to represent the expected prosody captured by the intermediate synthesized speech representation 202. The length or duration of the intermediate synthesized speech representation 202 is related to the sum of the total number of reference frames 211.

[0048] The encoder section 300a includes hierarchical levels of reference frames 211, phonemes 321, 321a, syllables 330, 330a, words 340, 340a, and sentences 350, 350a, which are timed relative to each other in the intermediate synthesized speech representation 202. For example, the level associated with the sequence of reference frames 211 times faster than the next level associated with the sequence of phonemes 321. Similarly, the level associated with the sequence of syllables 330 times slower than the level associated with the sequence of phonemes 321 and faster than the level associated with the sequence of words 340. Thus, the slower timing layer receives the output from the faster timing layer as input, such that the output after the last clock (i.e., state) of the faster layer is treated as the input to the corresponding slower layer to substantially provide the sequence to the sequence encoder. In the example shown, the hierarchical levels include Long Short-Term Memory (LSTM) levels.

[0049] In the example shown, the intermediate synthesized speech representation 202 includes a sentence 350, 350A with three words 340, 340A-C. The first word 340, 340A includes two syllables 330, 330Aa-Ab. The second word 340, 340B includes one syllable 330, 330Ba. The third word 340, 340a includes two syllables 330, 330Ca-Cb. The first syllable of the first word 340, 340A, 330, 330Aa, includes two phonemes 321, 321Aa1-Aa2. The second syllable of the first word 340, 340A, 330, 330Ab, includes one phoneme 321, 321Ab1. The first syllable of the second word 340, 340B, 330, 330Ba, includes three phonemes 321, 321Ba1-Ba3. The first syllables of the third word 340 and 340C, 330 and 330Ca, include one phoneme, 321 and 321Ca1. The second syllables of the third word 340 and 340C, 330 and 330Cb, include two phonemes, 321 and 321Cb1-Cb2.

[0050] In some embodiments, encoder portion 300a first encodes the sequence of reference frame 211 into frame-based syllable embeddings 332, 332Aa-Cb. Each frame-based syllable embedding 332 may indicate a reference prosodic feature represented as a numerical vector indicating the duration, pitch (F0), and / or energy (C0) associated with the corresponding syllable 330. In some embodiments, reference frame 211 defines a sequence of phonemes 321Aa1-321Cb2. Instead of encoding a subset of reference frame 211 into one or more phonemes 321, encoder portion 300a alternatively describes phonemes 321 by encoding phoneme-level linguistic features 322, 322Aa1-Cb2 into phoneme feature-based syllable embeddings 334, 334Aa-Cb. Each phoneme-level linguistic feature 322 can indicate the position of a phoneme; however, each phoneme-based syllable embedding 334 includes a vector indicating the position of each phoneme within the corresponding syllable 330 and the number of phonemes 321 within the corresponding syllable 330. For each syllable 330, the corresponding syllable embeddings 332 and 334 can be concatenated and encoded with the corresponding syllable-level linguistic features 336 and 336Aa-Cb of the corresponding syllable 330. Furthermore, each syllable embedding 332 and 334 indicates the corresponding state of the level of the syllable 330.

[0051] Continue to refer to Figure 3The blocks in the hierarchical layers that include diagonal shading patterns correspond to language features at specific levels of the hierarchy (except for word level 340). The shading patterns at word level 340 include word embeddings 342 extracted as language features from the input text discourse 320 or WP embeddings 342 output from the BERT model 270 based on word units 272 obtained from transcription 206. Since the recurrent neural network (RNN) part of the encoder 300a does not have the concept of words, the WP embedding 342 corresponding to the first word for each word can be selected to represent words that may contain one or more syllables 330. Using frame-based syllable embeddings 332 and phoneme-based syllable embeddings 334, the encoder part 300a concatenates and encodes these syllable embeddings 332, 334 with other language features 336, 352, 342 (or WP embeddings 342). For example, encoder section 300a encodes the concatenated syllable embeddings 332 and 334 using syllable-level language features 336, 336Aa-Cb, word-level language features (or WP embeddings 342, 342A-C output from BERT model 270) and / or sentence-level language features 352, 352A. By encoding the syllable embeddings 332 and 334 using language features 336, 352, 342 (or WP embedding 342), encoder section 300a generates utterance embedding 204 for intermediate synthesized speech representation 202. The utterance embedding 204 can be stored together with the input text utterance 320 (e.g., text representation) of intermediate synthesized speech representation 202 in data storage device 180. Figure 1 From the input text utterance 320, language features 322, 336, 342, and 352 can be extracted and stored for use during training to adjust the hierarchical language structure. Language features (e.g., language features 322, 336, 342, and 352) may include, but are not limited to, the individual sound of each phoneme and / or the position of each phoneme within a syllable, whether each syllable is stressed or unstressed, syntactic information for each word, and whether the utterance is a question or a phrase and / or the gender of the speaker. As used herein, any reference to word-level language features 342 of the encoder portion 300a and decoder portion 400a of the VAE-based second TTS model 222a can be replaced with WP embeddings from the BERT model 270.

[0052] exist Figure 3In the example, coding blocks 322, 322Aa-Cb are shown to depict the encoding between language features 336, 342, 352 and syllable embeddings 332, 334. Here, block 322 generates a sequence of utterance embeddings 204 at a syllable rate. As an illustration, the first block 322Aa is fed as input to the second block 322Ab. The second block 322Ab is fed as input to the third block 322Ba. The third block 322Ca is fed as input to the fourth block 322Ca. The fourth block 322Ca is fed to the fifth block 322Cb. In some configurations, the utterance embedding 204 includes the mean μ and standard deviation σ of intermediate synthesized speech representations 202, where the mean μ and standard deviation σ are training data relative to multiple intermediate synthesized speech representations 202.

[0053] In some implementations, each syllable 330 receives a corresponding encoding of a subset of reference frames 211 as input and includes a duration equal to the number of reference frames 211 in the encoded subset. In the example shown, the first seven fixed-length reference frames 211 are encoded as syllable 330Aa; the next four fixed-length reference frames 211 are encoded as syllable 330Ab; the next eleven fixed-length reference frames 211 are encoded as syllable 330Ba; the next three fixed-length reference frames 211 are encoded as syllable 330Ca; and the last six fixed-length reference frames 211 are encoded as syllable 330Cb. Thus, each syllable 330 in the sequence of syllables 330 may include a corresponding duration based on the number of reference frames 211 encoded as syllable 330 and a corresponding pitch and / or energy profile. For example, syllable 330Aa includes a duration of 35ms (i.e., seven reference frames 211, each with a fixed length of five milliseconds), and syllable 330Ab includes a duration of 20ms (i.e., four reference frames 211, each with a fixed length of five milliseconds). Therefore, the level of reference frames 211 at the level of syllable 330 counts a total of ten times for a single timing interval between syllable 330Aa and the next syllable 330Ab. The duration of syllable 330 can indicate the timing of syllable 330 and the pause between adjacent syllables 330.

[0054] In some examples, the utterance embedding 204 generated by encoder section 300a is a fixed-length utterance embedding 204 that includes a numerical vector representing the prosody of the intermediate synthesized speech representation 202. In some examples, the fixed-length utterance embedding 204 includes a numerical vector whose value is equal to "128" or "256".

[0055] Now for reference Figure 2A and Figure 4AThe decoder portion 400a of the second VAE-based TTS model 222a is configured to generate multiple fixed-length syllable embeddings 335 by initially decoding a fixed-length utterance embedding 204 of the expected prosody of a specified input text utterance 320. More specifically, the utterance embedding 204 represents the expected prosody possessed by the intermediate synthesized speech representation 202 of the input text utterance 320 output from the first TTS system 210. Therefore, the decoder portion 400a is configured to backpropagate the utterance embedding 204 to generate multiple fixed-length prediction frames 280 that closely match the multiple fixed-length reference frames 211. For example, fixed-length prediction frames 280 for both pitch (F0) and energy (C0) can be generated in parallel to represent expected prosody (e.g., predicted prosody) that substantially matches the expected prosody possessed by the training data. In some examples, the speech synthesizer 150 uses the fixed-length prediction frames 280 to generate synthesized speech 152 with expected prosody and target speech based on the fixed-length utterance embedding 204. For example, the unit selection module or WaveNet module of the speech synthesizer 150 can use frame 280 to generate synthesized speech 152 with the desired prosody.

[0056] In the example shown, decoder section 400a will receive from encoder section 300a ( Figure 2A and Figure 3 The received utterance embedding 204 (e.g., the numerical value of "256") is encoded into hierarchical levels of words 340, 340b, syllables 330, 330b, phonemes 321, 321b, and fixed-length prediction frames 280. Specifically, the fixed-length utterance embedding 204 corresponds to variable layers of hierarchical input data for decoder section 400a, and each of the stacked hierarchical levels includes a Long Short-Term Memory (LSTM) processing unit that is variablely timed to the length of the hierarchical input data. For example, syllable level 330 is timed faster than word level 340 and slower than phoneme level 321. The rectangular blocks in each level correspond to LSTM processing units for the corresponding word, syllable, phoneme, or frame. Advantageously, the second TTS model 222a based on VAE provides LSTM processing units at the word level 340 with memory for more than the last 100 words, LSTM units at the syllable level 330 with memory for more than the last 100 syllables, LSTM units at the phoneme level 321 with memory for more than the last 100 phonemes, and LSTM units at fixed-length pitch and / or energy frames 280 with memory for more than the last 100 fixed-length frames 280. When each of the fixed-length frames 280 comprises a duration of five milliseconds (e.g., frame rate), the corresponding LSTM processing unit provides memory for more than the last 500 milliseconds (e.g., half a second).

[0057] In the example shown, the decoder section 400a of the hierarchical language structure simply backpropagates the sequence of three words 340A-340C, five syllables 330Aa-330Cb, and nine phonemes 321Aa1-321Cb2 encoded by the encoder section 300a to generate a fixed-length utterance embedding 204 that predicts a fixed-length frame 280. The decoder section 400a is conditional on the linguistic features of the input text utterance 320. This is achieved by having the output from the faster timing layer received as input by the slower timing layer. Figure 3 In contrast to encoder section 300a, decoder section 400a includes the output from a slower timing layer that feeds a faster timing layer, such that the output of the slower timing layer is distributed to the input of the faster timing layer at each clock cycle with an attached timing signal.

[0058] refer to Figure 2A , Figure 3 and Figure 4A In some embodiments, the hierarchical language structure for the clock-device hierarchical variational autoencoder 222a is adapted to provide a controllable model for predicting Mel frequency spectrum information of the input text utterance 320, while effectively controlling the prosody implicitly represented in the Mel frequency information. Specifically, the second TTS model 222a can predict the Mel frequency spectrum 502 of the input text utterance (referred to simply as "input text" 320) and provide the Mel frequency spectrum 502 as input to the vocoder network 155 of the speech synthesizer for conversion into a time-domain audio waveform. The time-domain audio waveform includes an audio waveform that defines the amplitude of the audio signal over time. As will become apparent, the speech synthesizer 150 is capable of generating synthesized speech 152 from the input text 320 using the autoencoder 222a trained on sample input text and the corresponding Mel frequency spectrum 201 output solely from the first TTS model 212. In other words, the second TTS model 222a based on VAE does not accept complex linguistic and acoustic features that require considerable domain expertise to generate. Instead, it is able to use an end-to-end deep neural network to convert the input text 320 into a Mel frequency spectrum 502. The vocoder network 155, i.e., the neural vocoder, is trained separately and conditioned on the Mel frequency spectrum used for conversion into a time-domain audio waveform.

[0059] The Mel frequency spectrum comprises the frequency domain representation of sound. The Mel frequency spectrum emphasizes lower frequencies crucial for speech intelligibility, but does not emphasize higher frequencies dominated by fricatives and other noise bursts and generally do not require high-fidelity modeling. The vocoder network 155 can be any network configured to receive the Mel frequency spectrum and generate audio output samples based on it. For example, the vocoder network 155 can be or can be based on a parallel feedforward neural network as described in van den Oord, Parallel WaveNet: Fast High-Fidelity Speech Synthesis, available at https: / / arxiv.org / pdf / 1711.10433.pdf and incorporated herein by reference. Alternatively, the vocoder network 155 can be an autoregressive neural network.

[0060] As referenced above Figure 2A , Figure 3 and Figure 4A The described VAE-based second TTS model 222a includes an encoder section 300a and a decoder section 400a. The encoder section 300a is configured to encode multiple fixed-length reference Mel-frequency spectrum frames 211 sampled / extracted from the intermediate synthesized speech representation 202 into a speech embedding 204. The decoder section 400a is configured to learn how to decode the speech embedding into multiple fixed-length predicted Mel-frequency spectrum frames 280M0. The VAE-based TTS model 222a can be trained such that the number of predicted Mel-frequency spectrum frames 280 output from the decoder section 400a is equal to the number of reference Mel-frequency spectrum frames 211 input to the encoder section 300a. Furthermore, the VAE-based TTS model 222a is trained such that the prosodic information associated with the reference Mel-frequency spectrum frames 211 and the predicted Mel-frequency spectrum frames 280 substantially matches each other. The predicted Mel-frequency spectrum frames 280 can implicitly provide a prosodic representation of the intermediate synthesized speech representation 202. The reference Mel frequency spectrum frame 211 can be sampled from the intermediate output audio signal 201 output from the first TTS system 210 as a supplement to or replacement of the intermediate synthesized speech representation 202. Additional details of the second VAE-based TTS model 222a are described in U.S. Patent Application No. 16 / 867,427, filed May 5, 2020, the contents of which are incorporated herein by reference in their entirety.

[0061] Figure 2BA schematic view of prosodic transfer systems 200, 200b is provided, wherein the second TTS model 222 at the second TTS system 220 corresponds to the second TTS model 222b based on a sequence-to-sequence feature prediction network (hereinafter referred to as the S2S-based second TTS model 222b). In the illustrated example, the first TTS system 210 receives text utterance 320 and optional additional inputs 325 as inputs, which may include speaker characteristics of the target speech (e.g., speaker embedding Z). The additional inputs 325 may additionally or alternatively include one or more of language identifiers, text normalization, or prosodic vertical industry identifiers corresponding to the prosodic domain. Using the input text utterance and the optional additional inputs 325, the first TTS model 212 generates an intermediate output audio signal 201 that may include a Mel frequency spectrum having the expected prosodicity of the input text utterance 320. The first TTS system 210 may synthesize (e.g., using a speech synthesizer 150) the intermediate output audio signal 201 into an intermediate synthesized speech representation 202. As described above, the intermediate synthesized speech representation 202 is configured to accurately capture the expected prosody and is permitted to include reduced audio quality and a lack of speaker characteristics for identifying the target speech. In other words, the intermediate synthesized speech representation 202 is not suited for human listening; rather, it serves as a carrier for conveying the expected prosody to be used by the second TTS model 222 to reproduce and incorporate prosodic information into the expressive speech of the target speech.

[0062] The encoder portions 300, 300b of the second S2S-based TTS model 222b are configured to encode the intermediate synthesized speech representation 202 (or intermediate output audio signal 201) into a utterance embedding 204 specifying the expected prosody. The intermediate synthesized speech representation 202 (or intermediate output audio signal 201) fed to the encoder portion 300b may implicitly represent the expected prosody of the input text utterance 320. In some embodiments, the encoder portion 300b corresponds to a variational autoencoder that encodes the expected prosody as a latent factor into the utterance embedding 204. In these embodiments, the utterance embedding 204 may correspond to a latent embedding. These latent factors are typically not represented in the conditioning input of the decoder portion 400b, whereby the conditioning input may include the input text utterance 320 and other inputs 325, such as a speaker embedding 116 associated with the speaker characteristics of the target speech, a language embedding associated with the native language of the input text utterance 320, and a specific prosodic vertical industry 20 identifying the intended prosody. Figure 1 The rhythmic vertical industry identifier. Therefore, encoder part 300b passes utterance embedding 204 to decoder 400b.

[0063] Now for reference Figure 2B and Figure 4BThe decoder portion 400a of the second S2S-based TTS model 222b may include an architecture with a pre-net 410, a long short-term memory (LSTM) subnetwork 420, a linear projection 430, and a convolutional post-net 440. The pre-net 410, through which Mel frequency spectrum prediction for the previous time step is passed, may include two fully connected layers hiding ReLU. The pre-net 410 acts as an information bottleneck for learning attention to improve convergence speed and generalize the speech synthesis system during training. To introduce output variation during inference, dropout with a probability of 0.5 can be applied to each layer in the pre-net.

[0064] The LSTM subnetwork 420 may include two or more LSTM layers. At each time step, the LSTM subnetwork 420 receives a concatenation of the output of the pre-net 410, the utterance embedding 204, and a portion of the text utterance 320 for that time step. The LSTM layers may be regularized using zoneout with, for example, a probability of 0.1. The linear projection 430 receives the output of the LSTM subnetwork 420 as input and produces a prediction of the Mel frequency spectrum 118P.

[0065] A convolutional post-net 440 with one or more convolutional layers processes the predicted Mel frequency spectrum 118P at each time step to predict the residual 242, which is then added to the predicted Mel frequency spectrum 118P at adder 244. This improves the overall reconstruction. Each convolutional layer, except the last one, may be followed by batch normalization and hyperbolic tangent (TanH) activation. The convolutional layers are regularized using dropout with, for example, a probability of 0.5. The residual 242 is added to the predicted Mel frequency spectrum 118P generated by linear projection 430, and the result (i.e., the Mel frequency spectrum 118) can be provided to the speech synthesizer 150. In some implementations, in parallel with the decoder section 400b predicting the Mel frequency spectrum 118 at each time step, the output of the LSTM subnetwork 420, the utterance embedding 204, and portions of the text utterance 320 (e.g., character embeddings generated by a text encoder (not shown)) are projected onto a scalar and predicted by sigmoid activation to indicate the probability that the output sequence of the Mel frequency spectrum 118 has been completed. The output sequence Mel frequency spectrum 118 corresponds to the output audio signal 280 of the expressive speech of the input text utterance 320 and includes the expected prosody and speaker characteristics associated with the target speech.

[0066] This "stop lexicon" prediction is used during inference to allow model 222b to dynamically determine when to terminate generation, rather than generating for a fixed duration. When a stop lexicon indicates that generation has terminated, i.e., when the stop lexicon probability exceeds a threshold, decoder section 400b stops predicting the Mel frequency spectrum 118P and returns the Mel frequency spectrum predicted up to that point as the output audio signal 280 of the expressive speech. Alternatively, decoder section 400b may always generate the Mel frequency spectrum 118 of the same length (e.g., 10 seconds). In some embodiments, the speech synthesizer is a Griffin-Lim synthesizer. In some other embodiments, the speech synthesizer includes a vocoder 155. For example, speech synthesizer 150 may include a WaveRNN vocoder 155. Here, the WaveRNN vocoder 155 may generate a 16-bit signal sampled at 24 kHz conditioned on the spectrum 118 predicted by TTS model 222b. In some other embodiments, the waveform synthesizer is a trainable spectrum-to-waveform converter. After waveform synthesizer 150 generates a waveform, the audio output system can use the waveform to generate speech 152 and provide the generated speech 152 for, for example, playback on a user device, or provide the generated waveform to another system to allow that system to generate and play back speech 152. In some examples, WaveNet neural vocoder 155 replaces waveform synthesizer 150. Compared to the synthesized speech generated by waveform synthesizer 150, WaveNet neural vocoder can provide synthesized speech with a different audio fidelity. Thus, in some examples, first TTS system 210 can employ conventional waveform synthesizer 150 to generate intermediate synthesized speech representation 202 with intermediate speech of reduced quality but accurately possessing the intended prosody; however, second TTS system 220 can transfer the intended prosody from intermediate synthesized speech representation 202 to synthesized speech of the target speech generated by WaveNet neural vocoder 155.

[0067] In some implementations, decoder section 400b includes an attention-based sequence-to-sequence model configured to generate a sequence of output log-Mel spectrogram frames, such as output Mel spectrogram 118, based on input text utterance 320 and additional input such as speaker embedding Z providing speaker characteristics associated with the target utterance. For example, decoder section 400b may be based on a Tacotron 2 model (see, for example, J. Shen et al., “Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions”, https: / / arxiv.org / abs / 1712.05884, which is incorporated herein by reference). Therefore, the second S2S-based TTS model 222b provides an enhanced TTS model for transferring the expected prosody possessed in the intermediate synthesized speech representation 202 into the utterance embedding 204 and processing the utterance embedding 204 and the input text utterance 320 with additional input 325 (e.g., speaker embedding z) to produce an output audio signal 280 of expressive speech with the expected prosody in the target utterance. Additional input 325, such as speaker embedding z, language identifiers, and prosody vertical industry identifiers, helps permit the transfer of different utterances of different languages ​​across any expected prosody trained on the system 200 above.

[0068] Figure 5 This is a flowchart illustrating an example setup of a method 500 for synthesizing input text speech into expressive speech with a desired prosody in the target speech. Data processing hardware 122 ( Figure 1 The operation of method 500 can be performed by executing instructions stored on memory hardware 124. In operation 502, method 500 includes receiving input text utterance 320 to be synthesized into expressive speech 152 with a desired prosody and target speech. In operation 504, method 500 includes generating an intermediate synthesized speech representation 202 of the input text utterance 320 using a first text-to-speech (TTS) model 212. Here, the intermediate synthesized speech representation 202 possesses the desired prosody. The intermediate synthesized speech representation 202 may include an audio waveform or Mel frequency spectrum sequence capturing the desired prosody. Furthermore, the intermediate synthesized speech representation 202 may include an intermediate speech that lacks the speaker characteristics of the target speech and includes undesirable acoustic artifacts. Therefore, the intermediate synthesized speech representation 202 provides expressiveness but may lack intelligibility and naturalness.

[0069] In operation 506, method 500 includes providing an intermediate synthesized speech representation 202 to a second TTS model 222, the second TTS model 222 including an encoder portion 300 and a decoder portion 400. The encoder portion 300 is configured to encode the intermediate synthesized speech representation 202 into a utterance embedding 204 specifying an expected prosody. The decoder portion 400 is configured to process the input text utterance 320 and the utterance embedding 204 to generate an output audio signal 280 of expressive speech 152. Here, the output audio signal has the expected prosody and speaker characteristics of the target speech specified by the utterance embedding 204.

[0070] In some examples, method 500 also includes additional input 325 to obtain a speaker embedding Z representing speaker characteristics of the target speech. In these examples, decoder portion 400 is configured to process input text utterance 320, utterance embedding 204, and speaker embedding Z to generate an output audio signal of expressive speech. The first TTS model 212 and the second TTS model 222 may each include the same or different types of neural network architectures.

[0071] A software application (i.e., a software resource) can refer to computer software that instructs a computing device to perform a task. In some examples, a software application may be referred to as an "application," "App," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0072] Non-transitory memory can be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. Non-transitory memory can be volatile addressable semiconductor memory and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0073] Figure 6This is a schematic view of an example computing device 600 that can be used to implement the systems and methods described in this document. The computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0074] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 connected to the memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 connected to a low-speed bus 670 and the storage device 630. Each of components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and may be mounted on a common motherboard or otherwise, as appropriate. Processor 610 is capable of processing instructions that run within computing device 600, including instructions stored in memory 620 or on storage device 630 to display graphical information of a graphical user interface (GUI) on an external input / output device, such as a display 680 coupled to high-speed interface 640. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and various types of memory, may be used as appropriate. Additionally, multiple computing devices 600 may be connected, with each device providing a portion of the necessary operation (e.g., as a server group, blade server group, or multiprocessor system).

[0075] Memory 620 stores information non-transitorily within computing device 600. Memory 620 may be a computer-readable medium, volatile memory(s), or non-volatile memory(s). Non-transitory memory 620 may be a physical device for storing programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0076] Storage device 630 provides mass storage for computing device 600. In some embodiments, storage device 630 is a computer-readable medium. In various embodiments, storage device 630 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device or array of devices, including devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 620, storage device 630, or memory on processor 610.

[0077] High-speed controller 640 manages bandwidth-intensive operations of computing device 600, while low-speed controller 660 manages less bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 640 is coupled to memory 620, display 680 (e.g., via a graphics processor or accelerator), and to high-speed expansion port 650 which can accept various expansion cards (not shown). In some embodiments, low-speed controller 660 is coupled to storage device 630 and low-speed expansion port 690. Low-speed expansion port 690, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices such as switches or routers, for example, via a network adapter.

[0078] As shown in the figure, the computing device 600 can be implemented in many different forms. For example, it can be implemented as a standard server 600a or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0079] Various implementations of the systems and techniques described herein can be implemented as digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can be incorporated into one or more computer programs operable and / or interpretable on a programmable system comprising at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0080] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural languages ​​and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0081] The processes and logical flows described in this specification can be executed by one or more programmable processors, also known as data processing hardware, running one or more computer programs to perform functions by manipulating input data and generating output. The processes and logical flows can also be executed by special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). As an example, processors suitable for running computer programs include both general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. Essential components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from or send data to one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or both. However, a computer does not necessarily have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory can be supplemented by dedicated logic circuitry systems or incorporated into dedicated logic circuitry systems.

[0082] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer, which has a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0083] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A method comprising: The input text speech is received at the data processing hardware and is to be synthesized into expressive speech with the expected prosody and target speech. The data processing hardware uses a first text-to-speech model to generate an intermediate synthesized speech representation of the input text utterance, the intermediate synthesized speech representation having the expected prosody; as well as The data processing hardware provides the intermediate synthesized speech representation to a second text-to-speech model, the second text-to-speech model comprising: The encoder portion is configured to encode the intermediate synthesized speech representation into a utterance embedding specifying the expected prosody; as well as The decoder section is configured to process the input text utterance and the utterance embedding to generate an output audio signal of expressive speech, the output audio signal having the expected prosody and speaker characteristics of the target speech specified by the utterance embedding.

2. The method according to claim 1, further comprising: The data processing hardware samples a sequence of fixed-length reference frames from the intermediate synthesized speech representation, the prosodic features representing the expected prosody possessed by the intermediate synthesized speech representation. The provision of the intermediate synthesized speech representation to the second text-to-speech model includes providing a sequence of fixed-length reference frames sampled from the intermediate synthesized speech representation to the encoder portion, the encoder portion being configured to encode the sequence of fixed-length reference frames into the speech embedding.

3. The method according to claim 2, wherein, The prosodic features representing the expected prosodic features possessed by the intermediate synthesized speech representation include duration, pitch profile, energy profile, and / or Mel frequency spectrum profile.

4. The method according to claim 2, wherein, The encoder portion is configured to encode the sequence of the fixed-length reference frames into the speech embedding by performing the following operations for each syllable in the intermediate synthesized speech representation: The phoneme-level linguistic features associated with each phoneme in the syllable are encoded into phoneme-feature-based syllable embeddings; The fixed-length reference frame associated with the syllable is encoded into a frame-based syllable embedding, the frame-based syllable embedding indicating the duration, pitch and / or energy associated with the corresponding syllable; as well as The phoneme-based syllable embedding and the frame-based syllable embedding, having the following components, are encoded into a corresponding prosodic syllable embedding of the syllable: syllable-level language features associated with the syllable, sentence-level language features associated with the intermediate synthesized speech representation, and word-level language features associated with a word including the corresponding syllable.

5. The method according to claim 4, wherein, The word-level language features include word embeddings obtained from a sequence of word embeddings generated from the input text discourse by a bidirectional encoder representation (BERT) model from the transducer.

6. The method according to claim 2, wherein, The decoder portion is configured to process the input text utterance and the utterance embedding to generate the output audio signal by decoding the corresponding utterance embedding into a sequence of fixed-length prediction frames that provide a prosodic representation of the input text utterance, the prosodic representation representing the expected prosodicity specified by the utterance embedding.

7. The method according to claim 6, wherein, The second text-to-speech model is trained such that the number of fixed-length prediction frames decoded by the decoder is equal to the number of fixed-length reference frames sampled from the intermediate synthesized speech representation.

8. The method according to claim 1, wherein, The discourse embedding includes a fixed-length numerical vector.

9. The method according to claim 1, wherein: The intermediate synthesized speech representation includes capturing the audio waveform or Mel frequency spectrum sequence of the expected prosody; and Providing the intermediate synthesized speech representation to the second text-to-speech model includes providing the audio waveform or the Mel frequency spectrum sequence to the encoder portion, the encoder portion being configured to encode the audio waveform or the Mel frequency spectrum sequence into the speech embedding.

10. The method of claim 1, further comprising: The data processing hardware obtains a speaker embedding representing the speaker characteristics of the target speech; as well as The data processing hardware provides the speaker embedding to the decoder portion of the second text-to-speech model, the decoder portion being configured to process the input text utterance, the utterance embedding, and the speaker embedding to generate the output audio signal of expressive speech.

11. The method according to claim 1, wherein, The intermediate synthesized speech representation generated using the first text-to-speech model includes an intermediate voice that lacks the speaker characteristics of the target voice and includes unwanted acoustic artifacts.

12. The method of claim 1, further comprising: The data processing hardware receives training data comprising multiple training audio signals and corresponding transcriptions, each training audio signal comprising a speech of human speech having the expected prosody spoken by a corresponding speaker in a prosodic domain / vertical industry associated with the expected prosody, and each transcription comprising a text representation of the corresponding training audio signal. as well as For each corresponding transcription of the training data: The first text-to-speech model is trained by the data processing hardware to generate a corresponding reference audio signal, the corresponding reference audio signal including a trained synthesized speech representation that captures the expected prosody of the corresponding utterance of human speech; The encoder portion of the second text-to-speech model is trained by the data processing hardware by encoding the corresponding trained synthetic speech representation into a corresponding utterance embedding representing the expected prosody captured by the trained synthetic speech representation; The decoder portion of the second text-to-speech model is trained by the corresponding transcription of the training data by the data processing hardware through embedding the corresponding utterance encoded by the encoder portion into a predicted output audio signal of expressive speech with the expected prosody. Generate the gradient / loss between the predicted output audio signal and the corresponding reference audio signal; and The gradient / loss is backpropagated through the second text-to-speech model.

13. The method according to any one of claims 1-12, wherein, The first text-to-speech model and the second text-to-speech model were trained separately.

14. The method according to any one of claims 1-12, wherein, The first text-to-speech model includes a first neural network architecture, and the second text-to-speech model includes a second neural network architecture that is different from the first neural network architecture.

15. The method according to any one of claims 1-12, wherein, The first text-to-speech model and the second text-to-speech model include the same neural network architecture.

16. A system comprising: Data processing hardware; as well as Memory hardware that communicates with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including the following: Receive input text utterances that are to be synthesized into expressive speech with expected prosody and target speech; A first text-to-speech model is used to generate an intermediate synthesized speech representation of the input text utterance, the intermediate synthesized speech representation having the expected prosody; as well as The intermediate synthesized speech representation is provided to a second text-to-speech model, the second text-to-speech model comprising: The encoder portion is configured to encode the intermediate synthesized speech representation into a utterance embedding specifying the expected prosody; as well as The decoder section is configured to process the input text utterance and the utterance embedding to generate an output audio signal of expressive speech, the output audio signal having the expected prosody and speaker characteristics of the target speech specified by the utterance embedding.

17. The system according to claim 16, wherein, The operation further includes: A sequence of fixed-length reference frames providing prosodic features is sampled from the intermediate synthesized speech representation, the prosodic features representing the expected prosodicity possessed by the intermediate synthesized speech representation. The provision of the intermediate synthesized speech representation to the second text-to-speech model includes providing a sequence of fixed-length reference frames sampled from the intermediate synthesized speech representation to the encoder portion, the encoder portion being configured to encode the sequence of fixed-length reference frames into the speech embedding.

18. The system according to claim 17, wherein, The prosodic features representing the expected prosodic features possessed by the intermediate synthesized speech representation include duration, pitch profile, energy profile, and / or Mel frequency spectrum profile.

19. The system according to claim 17, wherein, The encoder portion is configured to encode the sequence of the fixed-length reference frames into the speech embedding by performing the following operations for each syllable in the intermediate synthesized speech representation: The phoneme-level linguistic features associated with each phoneme in the syllable are encoded into phoneme-feature-based syllable embeddings; The fixed-length reference frame associated with the syllable is encoded into a frame-based syllable embedding, the frame-based syllable embedding indicating the duration, pitch and / or energy associated with the corresponding syllable; as well as The phoneme-based syllable embedding and the frame-based syllable embedding having the following elements are encoded into a corresponding prosodic syllable embedding of the syllable: syllable-level language features associated with the syllable, sentence-level language features associated with the intermediate synthesized speech representation, and word-level language features associated with a word including the corresponding syllable.

20. The system according to claim 19, wherein, The word-level language features include word embeddings obtained from a sequence of word embeddings generated from the input text discourse by a bidirectional encoder representation (BERT) model from the transducer.

21. The system according to claim 16, wherein, The decoder portion is configured to process the input text utterance and the utterance embedding to generate the output audio signal by decoding the corresponding utterance embedding into a sequence of fixed-length prediction frames that provide a prosodic representation of the input text utterance, the prosodic representation representing the expected prosodicity specified by the utterance embedding.

22. The system according to claim 21, wherein, The second text-to-speech model is trained such that the number of fixed-length prediction frames decoded by the decoder is equal to the number of fixed-length reference frames sampled from the intermediate synthesized speech representation.

23. The system according to claim 16, wherein, The discourse embedding includes a fixed-length numerical vector.

24. The system according to claim 16, wherein: The intermediate synthesized speech representation includes capturing the audio waveform or Mel frequency spectrum sequence of the expected prosody; and Providing the intermediate synthesized speech representation to the second text-to-speech model includes providing the audio waveform or the Mel frequency spectrum sequence to the encoder portion, the encoder portion being configured to encode the audio waveform or the Mel frequency spectrum sequence into the speech embedding.

25. The system according to claim 16, wherein, The operation further includes: Obtain the speaker embedding representing the speaker characteristics of the target speech; and The speaker embedding is provided to the decoder portion of the second text-to-speech model, the decoder portion being configured to process the input text utterance, the utterance embedding, and the speaker embedding to generate the output audio signal of expressive speech.

26. The system according to claim 16, wherein, The intermediate synthesized speech representation generated using the first text-to-speech model includes an intermediate voice that lacks the speaker characteristics of the target voice and includes unwanted acoustic artifacts.

27. The system according to claim 16, wherein, The operation further includes: The system receives training data comprising multiple training audio signals and corresponding transcriptions, each training audio signal comprising utterances of human speech with the expected prosody spoken by a corresponding speaker in a prosodic domain / vertical industry associated with the expected prosody, and each transcription comprising a text representation of the corresponding training audio signal; and For each corresponding transcription of the training data: The first text-to-speech model is trained to generate a corresponding reference audio signal, the corresponding reference audio signal including a trained synthesized speech representation that captures the expected prosody of the corresponding utterance of human speech; The encoder portion of the second text-to-speech model is trained by encoding the corresponding trained synthetic speech representation into a corresponding utterance embedding representing the expected prosody captured by the trained synthetic speech representation; The corresponding transcription using the training data trains the decoder portion of the second text-to-speech model by embedding the corresponding utterance encoded by the encoder portion into a predicted output audio signal that decodes into expressive speech with the expected prosody. Generate the gradient / loss between the predicted output audio signal and the corresponding reference audio signal; and The gradient / loss is backpropagated through the second text-to-speech model.

28. The system according to any one of claims 16-27, wherein, The first text-to-speech model and the second text-to-speech model were trained separately.

29. The system according to any one of claims 16-27, wherein, The first text-to-speech model includes a first neural network architecture, and the second text-to-speech model includes a second neural network architecture that is different from the first neural network architecture.

30. The system according to any one of claims 16-27, wherein, The first text-to-speech model and the second text-to-speech model include the same neural network architecture.

Images