Extending multilingual speech synthesis with zero supervision of discovered data

Abstract

A method (600) includes receiving training data (301) including a plurality of sets of training utterances (310) each associated with a respective language. Each training utterance includes a corresponding reference speech representation (304) paired with a corresponding input text sequence (302). For each training utterance, the method includes generating a corresponding encoded text representation for a corresponding input text sequence (312, 313), generating a corresponding speech code for a corresponding reference speech representation (314), generating a shared encoder output (332, 334), and outputting the shared encoder output (332, 334). And determine a text-to-speech (TTS) loss based on the corresponding encoded text representation, the corresponding speech coding, and the shared encoder output (305). The method further comprises training the TTS model (501) based on the TTS loss determined for the training utterances in each set of training utterances to teach the TTS model how to learn to synthesize speech in each of the respective languages.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to extending multilingual speech synthesis with zero supervision on discovered data. BACKGROUND

[0002] Text-to-speech (TTS) systems vocalize digital text to users and are increasingly popular on mobile devices. Certain TTS models aim to synthesize various aspects of speech, such as speaking style and language, to produce natural speech that resembles humans. Some TTS models are multilingual, enabling the TTS model to output synthesized speech in multiple different languages. However, even these multilingual TTS models are compatible with only a relatively small fraction of all languages in the world. Specifically, the lack of sufficient training data for other languages, especially low-resource languages, hinders TTS models from learning to generate synthesized speech in these other languages. Thus, training a multilingual TTS model to generate synthesized speech in multiple different languages, even for low-resource languages, further increases the use of TTS models. SUMMARY

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for zero supervision on discovered data to extend multilingual speech synthesis. The operations include receiving training data that includes multiple sets of training utterances. Each set of training utterances is associated with a respective language and includes speech spoken in the respective language that is different from the respective language associated with each other set of training utterances. Each training utterance includes a corresponding reference speech representation paired with a corresponding input text sequence. For each training utterance in each set of training utterances of the received training data, the operations include generating, using a text encoder, a corresponding encoded text representation for the corresponding input text sequence; generating, using a speech encoder, a corresponding speech encoding for the corresponding reference speech representation; generating, using a shared encoder configured to receive the corresponding encoded text representation or the corresponding speech encoding, a shared encoder output; and determining, based on the corresponding encoded text representation, the corresponding speech encoding, and the shared encoder output, a text-to-speech (TTS) loss. The operations further include training, based on the TTS loss determined for the training utterances in each set of training utterances, a TTS model to teach the TTS model to learn how to synthesize speech in each of the respective languages.

[0004] Implementations of the disclosure can include one or more of the following optional features. In some implementations, for each training utterance of each group of training utterances of the received training data, the operations further include obtaining a corresponding speaker embedding that characterizes speaker characteristics of a corresponding speaker who spoke the training utterance in the respective language, and obtaining a corresponding language embedding that identifies the respective language of the utterance. Here, the text encoder is configured to receive a concatenation of the corresponding speaker embedding and the corresponding language embedding. In some examples, for each training utterance of each group of training utterances of the received training data, the operations further include generating a speech recognition hypothesis using an automatic speech recognition (ASR) decoder configured to receive the shared encoder output as input, the speech recognition hypothesis representing a candidate transcription for the corresponding training utterance, and determining an ASR loss based on the speech recognition hypothesis and the corresponding input text sequence. Here, the TTS loss includes the ASR loss. In these examples, the ASR decoder includes a recurrent neural network transducer (RNN-T) architecture.

[0005] In some implementations, for each training utterance of each group of training utterances of the received training data, the operations further include determining a feature loss between the encoded text representation generated using the text encoder for the corresponding input text sequence and speech encoding generated using the speech encoder for the corresponding reference speech representation. Here, the TTS loss includes the feature loss. In some examples, for each training utterance of each group of training utterances of the received training data, the operations further include obtaining a sequence representation of the corresponding input text sequence concatenated with the variational embedding; predicting, using a duration model, a duration of the input text sequence based on the sequence representation; upsampling, using the duration model, the sequence representation to an upscaled output of a specified number of frames; and determining a duration loss based on the predicted duration of the input text sequence and a ground truth duration. Here, the TTS loss includes the duration loss.

[0006] In some implementations, the training data further includes non-spoken text utterances associated with the respective plurality of different languages, where each non-spoken text utterance is not paired with any corresponding spoken utterance, and the operations further include, for each non-spoken text utterance, generating, using the text encoder, a corresponding non-spoken encoded text representation for the corresponding non-spoken text utterance, and determining a masked language modeling (MLM) loss for the aligned text of the corresponding non-spoken encoded text representation generated for the corresponding non-spoken text utterance. Here, the TTS loss includes the MLM loss for the aligned text. In these implementations, each non-spoken text utterance can be paired with a corresponding language identifier label, and the operations further include, for each non-spoken text utterance, generating, using a language identifier configured to receive as input the corresponding non-spoken encoded text representation generated for the corresponding non-spoken text utterance, a predicted language identifier, and determining a text language identifier loss based on the predicted language identifier and the language identifier label. Here, the TTS loss includes the text language identifier loss. In some examples, the training data further includes unpaired spoken utterances spoken in the respective plurality of different languages, where each unpaired spoken utterance is not paired with any corresponding text, and the operations further include, for each unpaired spoken utterance, generating, using the speech encoder, a corresponding unpaired speech encoding for the corresponding unpaired spoken utterance, and determining a masked language modeling (MLM) loss for the aligned speech of the corresponding unpaired speech encoding generated for the corresponding unpaired spoken utterance. Here, the TTS loss includes the MLM loss for the aligned speech. In these examples, for each unpaired spoken utterance, the operations can further include generating, using a shared encoder further configured to receive the corresponding unpaired speech encoding, an unpaired shared encoder output, and generating, using an automatic speech recognition (ASR) decoder configured to receive as input the unpaired shared encoder output, a pseudo-label representing a candidate transcription for the corresponding unpaired spoken utterance. Here, the training data further includes the non-spoken text utterance including the pseudo-label.

[0007] In these examples, each unpaired spoken utterance can be paired with a corresponding language identifier label, and the operations further include, for each unpaired spoken utterance, generating a predicted language identifier using a language recognizer configured to receive, as input, a corresponding unpaired speech encoding for the corresponding unpaired spoken utterance, and determining a speech language identifier loss based on the predicted language identifier and the language identifier label. Here, the TTS loss includes the speech language identifier loss. Each corresponding input text sequence can include a sequence of graphemes, wordpiece model units, phonemes, or bytes. In some examples, generating the speech encoding for the corresponding reference speech representation includes applying a random projection using a random projection quantizer to project the corresponding utterance, and mapping the corresponding projected utterance to a discrete label.

[0008] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving training data including multiple sets of training utterances. Each set of training utterances is associated with a respective language and includes speech spoken in the respective language that is different from the respective language associated with each other set of training utterances. Each training utterance includes a corresponding input text sequence paired with a corresponding reference speech representation. For each training utterance in each set of training utterances of the received training data, the operations include generating, using a text encoder, a corresponding encoded text representation for the corresponding input text sequence, generating, using a speech encoder, a corresponding speech encoding for the corresponding reference speech representation, generating, using a shared encoder configured to receive the corresponding encoded text representation or the corresponding speech encoding, a shared encoder output, and determining, based on the corresponding encoded text representation, the corresponding speech encoding, and the shared encoder output, a text-to-speech (TTS) loss. The operations further include training, based on the TTS loss determined for the training utterances in each set of training utterances, a TTS model to teach the TTS model to learn how to synthesize speech in each of the respective languages.

[0009] Implementations of the disclosure can include one or more of the following optional features. In some implementations, for each training utterance of each group of training utterances of the received training data, the operations further include: obtaining a corresponding speaker embedding that characterizes speaker characteristics of a corresponding speaker who spoke the training utterance in the respective language, and obtaining a corresponding language embedding that identifies the respective language of the utterance. Here, the text encoder is configured to receive a concatenation of the corresponding speaker embedding and the corresponding language embedding. In some examples, for each training utterance of each group of training utterances of the received training data, the operations further include: generating a speech recognition hypothesis using an automatic speech recognition (ASR) decoder configured to receive the shared encoder output as input, the speech recognition hypothesis representing a candidate transcription for the corresponding training utterance, and determining an ASR loss based on the speech recognition hypothesis and the corresponding input text sequence. Here, the TTS loss includes the ASR loss. In these examples, the ASR decoder includes a recurrent neural network transducer (RNN-T) architecture.

[0010] In some implementations, for each training utterance of each group of training utterances of the received training data, the operations further include: determining a feature loss between the encoded text representation generated using the text encoder for the corresponding input text sequence and speech encoding generated using the speech encoder for the corresponding reference speech representation. Here, the TTS loss includes the feature loss. In some examples, for each training utterance of each group of training utterances of the received training data, the operations further include: obtaining a sequence representation of the corresponding input text sequence concatenated with the variational embedding; predicting a duration of the input text sequence based on the sequence representation using a duration model; upsampling the sequence representation to an upscaled output of a specified number of frames using the duration model; and determining a duration loss based on the predicted duration of the input text sequence and a ground truth duration. Here, the TTS loss includes the duration loss.

[0011] In some implementations, the training data further includes non-spoken text utterances associated with the respective plurality of different languages, where each non-spoken text utterance is not paired with any corresponding spoken utterance, and the operations further include, for each non-spoken text utterance, generating, using the text encoder, a corresponding non-spoken encoded text representation for the corresponding non-spoken text utterance, and determining a masked language modeling (MLM) loss for the aligned text of the corresponding non-spoken encoded text representation generated for the corresponding non-spoken text utterance. Here, the TTS loss includes the MLM loss for the aligned text. In these implementations, each non-spoken text utterance can be paired with a corresponding language identifier label, and the operations further include, for each non-spoken text utterance, generating, using a language identifier configured to receive as input the corresponding non-spoken encoded text representation generated for the corresponding non-spoken text utterance, a predicted language identifier, and determining a text language identifier loss based on the predicted language identifier and the language identifier label. Here, the TTS loss includes the text language identifier loss. In some examples, the training data further includes unpaired spoken utterances spoken in the respective plurality of different languages, where each unpaired spoken utterance is not paired with any corresponding text, and the operations further include, for each unpaired spoken utterance, generating, using the speech encoder, a corresponding unpaired speech encoding for the corresponding unpaired spoken utterance, and determining a masked language modeling (MLM) loss for the aligned speech of the corresponding unpaired speech encoding generated for the corresponding unpaired spoken utterance. Here, the TTS loss includes the MLM loss for the aligned speech. In these examples, for each unpaired spoken utterance, the operations can further include generating, using a shared encoder further configured to receive the corresponding unpaired speech encoding, an unpaired shared encoder output, and generating, using an automatic speech recognition (ASR) decoder configured to receive as input the unpaired shared encoder output, a pseudo-label representing a candidate transcription for the corresponding unpaired spoken utterance. Here, the training data further includes the non-spoken text utterance including the pseudo-label.

[0012] In these examples, each unpaired spoken utterance can be paired with a corresponding language identifier label, and the operations further include, for each unpaired spoken utterance, generating a predicted language identifier using a language recognizer configured to receive as input a corresponding unpaired speech encoding for the corresponding unpaired spoken utterance, and determining a speech language identifier loss based on the predicted language identifier and the language identifier label. Here, the TTS loss includes the speech language identifier loss. Each corresponding input text sequence can include a sequence of graphemes, wordpiece model units, phonemes, or bytes. In some examples, generating a speech encoding for the corresponding reference speech representation includes applying a random projection using a random projection quantizer to project the corresponding utterance, and mapping the corresponding projected utterance to a discrete label.

[0013] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a schematic diagram of an example speech recognition system.

[0015] Figure 2 is a schematic diagram of an example automatic speech recognition model.

[0016] Figures 3A to 3E is a schematic diagram of an example training process for training a text-to-speech (TTS) model using groups of training utterances.

[0017] Figure 4 is a schematic diagram of an example alignment model used during an example training process.

[0018] Figure 5 is a schematic diagram of an example training process for an alignment model.

[0019] Figure 6 is a flowchart of an example arrangement of operations for a method of utilizing zero supervision on discovered data to expand multilingual speech synthesis.

[0020] Figure 7 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0021] In the various drawings, like reference numerals refer to like elements. DETAILED DESCRIPTION

[0022] Text-to-speech is the process of generating synthesized speech based on input textual data. In some cases, a TTS model is multilingual, whereby the TTS model can receive textual input and generate synthesized speech corresponding to the textual input in a plurality of different languages. Recently, TTS models have made significant progress in synthesizing high-quality speech that is similar to human speech in a plurality of languages. However, even multilingual TTS models are only able to generate synthesized speech in a few different languages. One major obstacle that prevents TTS models from scaling to hundreds or even thousands of different languages is the difficulty in collecting large amounts of high-quality paired training data for each different language needed to train a TTS model. In particular, low-resource languages have very sparse amounts (or even zero) of paired training data, further increasing the difficulty of scaling a TTS model to these low-resource languages.

[0023] Accordingly, implementations herein relate to methods and systems for training a large-scale multilingual TTS model. That is, a training process can receive training data that includes a plurality of sets of training utterances. Each set of training utterances is associated with a respective language that is different from the respective language associated with each other set of training utterances, and each set of training utterances includes speech spoken in the respective language. Each training utterance includes a corresponding reference speech representation paired with a corresponding input text sequence. For each training utterance in each set of training utterances, the training process generates a corresponding encoded text representation for the corresponding input text sequence, generates a corresponding speech encoding for the corresponding reference speech representation, generates a shared encoder output, and determines a text-to-speech (TTS) loss based on the corresponding encoded text representation, the corresponding speech encoding, and the shared encoder output. The training process further includes training a TTS model based on the TTS loss determined for the training utterances in each set of training utterances to teach the TTS model to learn how to synthesize speech in each of the respective languages. Notably, the training process can employ one or more components of an automatic speech recognition (ASR) model (e.g., a speech encoder and / or a text encoder) to train the multilingual TTS model. In some examples, the ASR model and the TTS model share the same text encoder. In other examples, the ASR model and the TTS model each include a respective text encoder.

[0024] Figure 1An example system 100 implementing an automatic speech recognition (ASR) model 200 and a text-to-speech (TTS) model 501 is shown, which resides on a user device 102 associated with a user 104 and / or a remote computing device 201 (e.g., one or more servers of a distributed system executing in a cloud computing environment) in communication with the user device 102. Although the user device 102 is depicted as a mobile computing device (e.g., a smartphone), the user device 102 can correspond to any type of computing device such as, but not limited to, a tablet device, a laptop / desktop computer, a wearable device, a digital assistant device, a smart speaker / display, a smart appliance, an automotive infotainment system, or an Internet of Things (IoT) device, and is equipped with data processing hardware 111 and memory hardware 113.

[0025] The user device 102 includes an audio subsystem 108 configured to receive an utterance 106 spoken by the user 104 (e.g., the user device 102 can include one or more microphones for recording the spoken utterance 106) and convert the utterance 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the ASR system 100. In the illustrated example, the user speaks a corresponding utterance 106 of the phrase “What is the weather in New York City?” in natural language English, and the audio subsystem 108 converts the utterance 106 into corresponding acoustic frames 110 for input to the ASR system 100. Thereafter, the ASR model 200 receives the acoustic frames 110 corresponding to the utterance 106 as input and generates / predicts a corresponding transcription 120 (e.g., a recognition result / hypothesis) of the utterance 106 as output. In the illustrated example, the user device 102 and / or the remote computing device 201 also execute a user interface generator 107 configured to present a representation of the transcription 120 of the utterance 106 to the user 104 of the user device 102. In some configurations, the transcription 120 output from the ASR system 100 is processed by a natural language understanding (NLU) module, e.g., executing on the user device 102 or the remote computing device 201, to execute a user command. Additionally or alternatively, the TTS model 501 (e.g., executing on any combination of the user device 102 or the remote computing device 201) can convert the transcription 120 into synthesized speech for audible output by the audio subsystem 108 or another device. For example, the original utterance 106 can correspond to a message that the user 104 is sending to a friend, where the transcription 120 is converted into synthesized speech for audible output to the friend to hear the message conveyed in the original utterance 106.

[0026] The TTS model 501 receives a text input 112 corresponding to a word or sequence of words as input and generates a corresponding speech representation 520 of the text input as output. Specifically, the TTS model 501 can generate a text encoding based on the text input 112 and decode the text encoding 520 to produce the speech representation 520. The user 104 can provide the text input 112 to the user device 102 via user input. In some examples, the user 104 directly provides the text input 112 by typing on a screen of the user device 102. In other examples, the user 104 can speak the utterance 106 such that the ASR model 200 generates the transcription 120 based on the utterance 106 that is used as the text input 112. Without departing from the scope of the present disclosure, the text input 112 can correspond to a response, notification, or other communication that the digital assistant is conveying to the user 104. The user 104 can also select a target embedding for the TTS model 501 to use in generating synthesized speech with the target speaker’s characteristics. Additionally or alternatively, the user 104 can further specify an intended prosody / style of the resulting synthesized speech. The audio subsystem 108, including a vocoder, can receive the speech representation 520 and generate an audible output of the text input 112 (e.g., via one or more speakers of the user device 102).

[0027] Reference Figure 2 In some examples, the ASR model 200 includes a recurrent neural network transducer (RNN-T) model architecture that adheres to latency constraints associated with interactive applications. The use of the RNN-T model architecture is exemplary, and the ASR model 200 can include other architectures such as transformer-transducer and conformer-transducer model architectures. The RNN-T model architecture provides a small computational footprint and uses less memory requirements than conventional ASR architectures, making the RNN-T model architecture suitable for performing speech entirely on the user device 102 (e.g., without needing to communicate with a remote server). The RNN-T model architecture of the ASR model 200 includes an encoder network 210, a prediction network 220, and a joint network 230. The encoder network 210, which is roughly analogous to an acoustic model (AM) in a traditional ASR system, includes a stack of self-attention layers (e.g., Conformer layers or Transformer layers) or a recurrent network of stacked long short-term memory (LSTM) layers. For example, the encoder network 210 reads in a sequence of acoustic feature vectors (e.g., acoustic frames 110 d Figure 1 x = (x 1 , x 2 , ···, x ​T ), where x t ∈ d , and produces a higher-order feature representation at each output step. This higher-order feature representation is denoted as .

[0028] Similarly, the prediction network 220 is also an LSTM network that, like a language model (LM), processes the sequence of non-blank symbols output so far by the final Softmax layer 240 y 0 ,..., y ui-1 into a dense representation . Finally, the representations produced by the encoder network 210 and the prediction / decoder network 220 are combined by the joint network 230, utilizing the RNN-T model architecture. The prediction network 220 can be replaced by an embedding lookup table to improve latency by outputting the looked-up sparse embedding instead of processing the dense representation. The joint network then predicts , which is a distribution over the next output symbol. In other words, the joint network 230 generates a probability distribution over possible speech recognition hypotheses at each output step (e.g., time step). Here, a “possible speech recognition hypothesis” corresponds to a set of output labels that each represent a symbol / character using a specified natural language. For example, when the natural language is English, the set of output labels can include twenty-seven (27) symbols, e.g., one label for each of the 24 letters in the English alphabet, and one label to designate a space. Thus, the joint network 230 can output a set of values that indicate the likelihood of each output label in a predetermined set of output labels to occur. The set of values can be a vector and can indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, as well as possible punctuation and other symbols), but the set of output labels is not limited as such. For example, the set of output labels can include word pieces, phonemes, and / or entire words in addition to or instead of graphemes. The output distribution of the joint network 230 can include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels that represent different graphemes or other symbols, the output y i of the joint network 230 can include 100 different probability values, one for each output label. The probability distribution can then be used (e.g., by the Softmax layer 240) to select and assign scores to candidate positive word elements (e.g., graphemes, word pieces, and / or words) in a beam search process for determining the transcription 120.

[0029] The softmax layer 240 can employ any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the ASR model 200 at the corresponding output step. In this way, the RNN-T model architecture of the ASR model 200 does not make the conditional independence assumption, instead the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The ASR model 200 does assume that the output symbol is independent of future acoustic frames 110, which allows the RNN-T model architecture of the ASR model 200 to be employed in a streaming fashion.

[0030] In some examples, the encoder network (i.e., audio encoder) 210 of the ASR model 200 includes a stack of self-attention layers / blocks, such as conformer blocks. Here, each conformer block includes a series of multi-head self-attention layers, depth-wise convolutional layers, and feed-forward layers. The prediction network 220 can have two 2,048-dimensional LSTM layers, where each layer is further followed by a 440-dimensional projection layer. Alternatively, the prediction network 220 can include a stack of transformer blocks or conformer blocks, or an embedding lookup table in place of the LSTM layers. Finally, the joint network 230 can also have 440 hidden units. The softmax layer 240 can be composed of a unified set of word pieces or graphemes that is generated using all unique word pieces or graphemes in multiple training datasets.

[0031] Figures 3A to 3E An example training process 300 for training the TTS model 501 using the sets of training utterances 310 is shown. In particular, the training process 300 can train the text encoder 202 of the TTS model 501. The TTS model 501 and the ASR model 200 can share the text encoder 202. As will be apparent, the training process 300 can train the TTS model 501 using the training data 301 including multiple sets of training utterances 310. More specifically, each set of training utterances 310 of the multiple sets of training utterances 310 includes a set of non-spoken text utterances X text ) 308, a set of transcribed speech utterances X sup ) 304, and / or untranscribed speech utterances X unsup) 306. The set of transcribed speech utterances 304 and the set of untranscribed speech utterances 306 can each include non-synthetic speech utterances spoken by a human and / or synthetic speech utterances generated by another TTS model. Each non-verbal text utterance 308 includes plain text data (i.e., unpaired data) such that each non-verbal text utterance 308 is not paired with any corresponding verbal audio representation (i.e., speech) of the utterance. The non-verbal text utterances 308 can include any sequence of text blocks, including words, word pieces (word piece model units), phonemes, bytes, and / or graphemes. Since the non-verbal text utterances 308 are non-verbal, each non-verbal text utterance 308 can be associated with a respective plurality of different languages. As will be apparent, the alignment model 400 can generate alignment outputs 402 for non-verbal text utterances 308 in one or more different languages. Each untranscribed speech utterance (i.e., unpaired verbal utterance) 306 includes plain audio data (i.e., unpaired data) such that the untranscribed speech utterances 306 are not paired with any corresponding transcription. On the other hand, each transcribed speech utterance 304 includes a corresponding transcription (i.e., input text sequence) 302 paired with a corresponding speech representation of the corresponding transcribed speech utterance 304.

[0032] Moreover, each set of training utterances 310 is associated with a respective language that is different from the respective language associated with each other set of training utterances 310, and each set of training utterances includes training utterances 310 of speech spoken in the respective language. For example, in the illustrated example, the training data 301 includes a first set of training utterances 310, 310a, including transcriptions 302, transcribed speech utterances 304, untranscribed speech utterances 306, and non-verbal text utterances 308, each associated with a first respective language (e.g., English). Continuing the illustrated example, the training data 301 also includes a second set of training utterances 310, 310b, including transcriptions 302, transcribed speech utterances 304, untranscribed speech utterances 306, and non-verbal text utterances 308, each associated with a second respective language (e.g., Chinese). For clarity only, the illustrated example includes two sets of training utterances 310 associated with two respective languages, as it will be appreciated that the training data 301 can include multiple sets of training utterances 310 associated with any number of languages.

[0033] For simplicity, the training process 300 includes a contrastive self-supervised loss portion 300a ( Figure 3A ), a supervised loss portion 300b ( Figure 3B ), and a consistency regularization portion 300c ( Figure 3C ). The training process 300 trains the TTS model 501 for a total loss (i.e., TTS loss 305) based on: using the contrastive self-supervised loss portion 300a from the non-verbal training text utterances (X text ) 308, transcribed speech utterances X sup ) 304, corpus of untranscribed speech utterances X unsup ) 306, contrastive loss derived L w2v ) 316; supervised loss portion 300b using non-spoken training text utterances X text ) 306, transcribed speech utterances X sup ) 304, supervised loss derived L aux ) 342, 344; consistency loss derived using consistency regularization portion 300c ) 352, and other losses determined by the training process discussed herein.

[0034] In some examples, the training process 300 employs an alignment model 400 configured to generate, at each of a plurality of output steps, an aligned output (i.e., a text representation) 402 for a respective one of the plurality of non-spoken training text utterances 308 and / or transcriptions 302. Thus, the alignment model 400 can generate a corresponding aligned output 402 for each of the non-spoken text utterances 308 and / or transcriptions 302. Thereafter, the training process 300 uses the generated aligned outputs 402 to train the TTS model 501.

[0035] Referring now to Figure 4 In some examples, the alignment model 400 includes an embedding extractor 410, a duration predictor 420, and an up-sampler 430. The embedding extractor 410 receives a respective one of the non-spoken text utterances 308 and / or transcriptions 302. Here, the non-spoken text utterances 308 and transcriptions 302 can each include a sequence of text blocks, including words, word pieces, phonemes, bytes, and / or graphemes. Thus, the embedding extractor 410 extracts a corresponding initial text representation 402 for the respective one of the non-spoken text utterances 308 and / or transcriptions 302. e t412. For example, embedding extractor 410 may receive a corresponding transcription 302 and extract an initial text representation (i.e., sequence representation) 412 from that transcription 302. The initial text representation 412 includes embedded lexical information from the text block sequence. In some examples, embedding extractor 410 concatenates the initial text representation 412 with a variational embedding 404 and provides this concatenation to a duration predictor (i.e., duration model) 420. Duration predictor 420 receives the initial text representation 412 (or concatenation) from embedding extractor 410 and predicts the corresponding text block duration (i.e., word, word fragment, phoneme, and / or character duration) 422. The text block duration 422 indicates the duration required to pronounce the corresponding text block when a human (or text-to-speech system) pronounces the corresponding transcription 302. For example, transcription 302 may include a sequence of phonemes, and duration predictor 420 predicts the phoneme duration 422 for each phoneme in the phoneme sequence. In this example, the duration predictor 420 predicts phoneme duration 422 by predicting the probability of a non-zero duration for each phoneme and the probability of predicting the consecutive phoneme durations for each phoneme. Since the phoneme sequence includes regular phonemes, pauses between word boundaries, and punctuation marks, only regular phonemes are associated with non-zero durations, while pauses and punctuation marks are typically associated with consecutive phoneme durations. Therefore, the duration predictor 420 can use sigmoid activation after the first of two independent activations to predict the probability of a non-zero duration, and use soft plus activation after the second of two independent projections to predict the consecutive text block duration 422 for each text block. The duration predictor 420 determines for each text block whether the probability of a non-zero duration is less than a threshold, and when the probability of a non-zero duration is less than the threshold, the multiplier can zero out the consecutive text block duration 422 predicted by the soft plus activation for the corresponding text block. Otherwise, when the probability of a non-zero duration is not less than the threshold, the predicted text block duration 422 can be set equal to the consecutive phoneme duration predicted by the soft plus activation.

[0036] Upsampler 430 receives each corresponding initial text representation 412 and corresponding predicted text block duration 422 output by embedding extractor 410, and generates an aligned output having a number of frames by upsampling the initial text representation 412 using the corresponding predicted text block duration 422. ) 402. In some examples, the alignment model 400 sends the alignment output 402 to the text encoder 202. In other examples (not shown), the alignment model 400 sends the alignment output 402 to the shared encoder 250 of the encoder 210 (e.g., bypassing the text encoder 202). In these other examples, the alignment output 402 is used as the encoded text representation 312, such that the shared encoder 250 can receive the alignment output 402 directly from the alignment model. In some additional examples, paired training data is available, and the up-sampler 430 generates the alignment output 402 as follows.

[0037]

[0038] Here, the up-sampler includes a resampler and a refiner layer that aligns the initial text embedding 412 to directly align with the corresponding encoded audio representation 314. In other examples, paired training data is not available, and the up-sampler 430 generates the alignment output 402 as follows.

[0039]

[0040] In particular, the number of frames of the alignment output 402 indicates a predicted speech duration of the respective one of the non-verbal text utterance 308 or the transcription 302. In other words, the number of frames of the alignment output 402 maps (i.e., aligns) a sequence of text blocks of the text input to speech frames. Here, the up-sampler 430 includes a resampler and a refiner layer that replicate the initial text embedding 412 to match the predicted text block duration 422 (i.e., speech duration). Thus, the alignment output 402 includes a text representation of the text input (e.g., the non-verbal text utterance 308 and / or the transcription 302) with a timing component that aligns with how a human would speak the text input.

[0041] Notably, in most cases, a TTS system (i.e., an auxiliary TTS system) generates an audible output to impart a timing component of human speech to a text input, such that the training process can use the audible output (i.e., synthesized speech) to train the encoder 210. Thus, because the alignment model 400 generates an alignment output 402 that directly maps a sequence of text blocks to speech frames, the training process 300 does not need speech synthesis of speech to generate the alignment output 402. That is, the alignment model 400 does not convert input text to synthesized speech.

[0042] Figure 5An example training process 500 for training the alignment model 400 using paired training data and unpaired training data is shown. That is, the training process 500 uses transcribed speech utterances 304 with corresponding transcriptions 302 (i.e., paired training data) and non-verbal textual utterances 308 (i.e., unpaired training data) to learn how to generate the alignment output 402. In the illustrated example, the speech encoder 204 receives as input each transcribed speech utterance 304 as a sequence of acoustic frames 110 of feature vectors (e.g., Figure 1 and generates as output, for each of a plurality of output steps, an encoded audio representation (i.e., speech encoding) 314 corresponding to the transcribed speech utterance 304 at the corresponding output step. When the speech encoder 204 generates the encoded audio representation 314 from the untranscribed speech 306, the encoded audio representation 314 represents unpaired speech encoding 314. In parallel, the alignment model 400 receives the transcription 302 corresponding to the same transcribed speech utterance 304 and generates the alignment output 402 corresponding to the same transcribed speech utterance. Additionally or alternatively, the alignment model 400 can receive the non-verbal textual utterance 308 and generate the corresponding alignment output 402. The text encoder 202 receives as input the alignment output 402 and generates as output, for each of a plurality of output steps, an encoded textual representation 312. When the text encoder 202 generates the encoded textual representation 312 from the non-verbal textual utterance 308, the encoded textual representation 312 represents non-verbal encoded textual representation 312.

[0043] The encoder 210 can include a shared encoder 250 that receives as input the encoded textual representation 312 and generates as output a first encoded shared representation 322. The shared encoder 250 can also receive as input the encoded audio representation 314 and generate as output a second encoded shared representation 324. The auxiliary decoder 390 receives as input the first encoded shared representation 322 and the second encoded shared representation 324 and generates as output corresponding first and second probability distributions 392, 294 over possible speech recognition hypotheses.

[0044] The alignment loss module 550 receives the first probability distribution 392 corresponding to the encoded textual representation 312 and the second probability distribution 394 corresponding to the encoded audio representation 314 and generates an alignment loss 552 by comparing the first probability distribution 392 to the second probability distribution 394. In some implementations, the alignment loss module 550 determines a duration loss 554. Here, the alignment loss module 550 can receive as input a number of frames (e.g., the number of frames 110) of the transcribed speech utterance 304 and determine the duration loss 554 based on the number of frames 110 of the transcribed speech utterance 304 and the number of frames 110 of the non-verbal textual utterance 308. Figure 4The alignment loss module 550 compares the input text sequence 412 with the aligned output 402 and the corresponding real-value duration 406 paired with the corresponding transcription 302 or non-spoken text utterance 308 from which the alignment output 402 was generated. That is, the real-value duration 406 can represent the number of frames that the upsampled output or aligned output 402 should have, such that the alignment loss module 550 compares the input text sequence 412 with the actual output 402. Figure 4 The predicted duration 422 is compared with the true duration 406 to determine the duration loss 554. The training process 500 can train any combination of the components of the alignment model 400 based on the alignment loss 552 and / or the duration loss 554 by updating the parameters of the alignment model 400.

[0045] Now for specific reference Figure 3A In some implementations, encoder 210 includes a reference Figure 3B and Figure 3C The speech encoder 204 and text encoder 202 are described in more detail. In the example shown, the speech encoder 204 processes audio input (e.g., transcribed speech 304 and untranscribed speech 306), and the text encoder 206 processes text input (e.g., non-spoken text 308). Each of the speech encoder 204 and text encoder 202 includes a Conformer encoder comprising a stack of Conformer blocks, each Conformer block comprising a series of multi-head self-attention layers, depthwise convolutional layers, and feedforward layers. Alternatively, the audio encoder 210 may comprise another type of encoder, such as a transformer encoder, with a stack of self-attention layers / blocks. Each of the speech encoder 204 and text encoder 202 can be naturally categorized as: a feature encoder comprising a convolutional subsampling block 212; and a context network comprising a stack of linear layers 214 and Conformer blocks 216. In some implementations, the convolutional subsampling block 212 has two two-dimensional convolutional layers, both with a stride of (2, 2), resulting in a reduction of the feature sequence length to one-quarter of its original length. The convolutional subsampling block 212 receives a sequence of input features / vectors (e.g., Mel spectrograms, etc.) associated with each transcribed speech utterance 304 and each untranscribed speech utterance 306. Figure 1 The acoustic frame 110 is taken as input, and for each of the multiple output steps, an encoded audio feature 211 corresponding to a corresponding one in the transcribed speech 304 or a corresponding one in the untranscribed speech 306 is generated as output. The convolutional sub-sampling block 212 can receive each aligned output 402 as input, and for each of the multiple output steps, an encoded text feature 213 corresponding to a corresponding one in the aligned output 402 is generated as output.

[0046] The encoded audio features 211 and encoded text features 213 (i.e., interchangeably referred to as "encoded features 211, 213") output from the convolutional sub-sampling block 212 can be fed into a masking module 218, in which some of the encoded features 211, 213 are randomly selected and replaced with a trained feature vector shared across all masked time steps to provide corresponding masked encoded audio features 211, 211m and masked encoded text features 213, 213m. In some examples, the masking module 218 performs masking by selecting a certain proportion of features from all time steps without replacement. p The random sampling of time steps is used as the starting index to mask the randomly selected encoded features 211, 213 to be masked, and then the subsequent M consecutive time steps caused by each sample are masked, where some segments may overlap. After applying the mask, the linear layer 214 and the Conformer block 216 of the context network receive the masked encoded features 211m (or the encoded features 211, 213 not selected by the masking module 218) and output the corresponding contrast context vector (i.e., the encoded representation) 215 from the masked encoded features 211m, 213m. In addition, the quantizer 217 receives the encoded features 211, 213 as input and generates a quantized vector (i.e., the target context vector) 219 as output. In some implementations, the quantizer 217 uses a random projection quantizer to apply random projection to project the corresponding utterance 310 (e.g., the encodings 211, 213). Here, quantizer 217 generates target context vector 219 by mapping the corresponding projected utterance to discrete labels. Subsequently, contrast loss module 315 derives the contrast loss between the contrast context vector 215 at the masked location and the target context vector 219 as follows: L w2v 316.

[0047]

[0048] in, c t It is the contrast field vector 215 centered on the masked time step t, and q t Indicates including q t and K A group of interference items K The target context vector 219 at time step t in the +1 candidate target context vector 219. The interference term can be uniformly sampled from other masked time steps of the same utterance.

[0049] The contrastive loss 316 is optimized between the contrastive context vectors 215 at the masked positions and the target context vectors 219. After the encoder 210 converges on the untranscribed speech utterance 306, the training process is repeated for both the aligned output 402 corresponding to the non-verbal text utterance 308 and the transcribed speech utterance 304. Thus, the contrastive loss L w2v ) is optimized for both the real / human and non-verbal text utterance 308 represented by the aligned output 402, with additional auxiliary losses on the transcribed speech utterance 304 and the aligned output 402, described in more detail below with reference to Figure 3B The contrastive portion 300a of the training process 300 trains the speech encoder 204 and the text encoder 202 for the derived contrastive loss 316, which is applied to the corresponding encoded features 211, 213 associated with each aligned output 402, each transcribed speech utterance 304, and each untranscribed speech utterance 306 provided as input to the encoder 210. Training the encoder 210 can include updating parameters of the encoder 210 based on the contrastive loss 316.

[0050] In some implementations, the contrastive loss module 315 determines a masked language modeling (MLM) loss 318 for the aligned text of the aligned output 402 (e.g., generated from the non-verbal text utterance 308) and a MLM loss 319 for the aligned speech of the speech input (e.g., the transcribed speech utterance 304 and the untranscribed speech utterance 306). That is, the contrastive loss module 315 determines the MLM loss 318 for the aligned text of the aligned output 402 by comparing the encodings generated for the masked encoded features and the unmasked encoded features for the aligned output 402. The contrastive loss module 315 determines the MLM loss 319 for the aligned speech of the speech input by comparing the contrastive context vectors 215 generated from the masked encoded features and the contrastive context vectors 219 generated from the corresponding unmasked encoded features. That is, the contrastive loss module 315 determines the MLM loss 319 for the aligned speech of the speech input by comparing the encodings generated for the masked encoded features and the unmasked encoded features for the speech input. The training process 300 can train the TTS model 501 based on the MLM loss 318 for the aligned text and / or the MLM loss 319 for the aligned speech.

[0051] Reference is now made to Figure 3BThe supervised loss portion 300b of the training process 300 is configured to inject lexical information into the text encoder 204 of the TTS model 501 during pre-training based on supervised loss terms 342, 344 derived from the transcribed speech utterance 304 and the alignment output 402 corresponding to the non-verbal text utterance 308 output by the alignment model 400. Notably, the supervised loss portion 300b utilizes one or more ASR decoders 390 to generate the supervised loss terms (i.e., ASR loss) 342, 344. The ASR decoders 390 can include a connectionist temporal classification (CTC) decoder, a listen-attend-spell (LAS) decoder, or an RNN-T decoder (e.g., RNN-T architecture). These ASR decoders 390 can include at least one of a phoneme decoder configured to decode a sequence of phonemes or a wordpiece decoder configured to decode a sequence of wordpieces. The ASR decoders 390 can also include a grapheme decoder configured to decode a sequence of graphemes.

[0052] During the supervised loss portion 300b, the text encoder 202 is configured to receive the alignment output 402 (i.e., text embedding) from the alignment model 400 and the speech encoder 204 is configured to receive the transcribed speech utterance 304. That is, the text encoder 202 generates an encoded text representation 312 for the alignment output 402 (e.g., corresponding to the non-verbal text utterance 308) and the speech encoder 204 of the encoder 210 generates an encoded audio representation 314 for the speech input (i.e., transcribed speech utterance 304). Here, neither the encoded text representation 312 nor the encoded audio representation 314 can be compatible with the ASR decoder 390. In some examples, the text encoder 202 obtains a corresponding speaker embedding 326 that characterizes speaker characteristics of a corresponding speaker who spoke the training utterance in the respective language and generates the corresponding encoded text representation 312 based on a concatenation of the corresponding alignment output 402 and the corresponding speaker embedding 326. When the training utterance 310 includes synthetic speech, the speaker embedding 326 can represent an embedding input to the TTS model that generated the training utterance 310 to produce the particular speech characteristics of the training utterance 310. Also, in addition to or instead of the speaker embedding 326, the text encoder 202 can obtain a corresponding language embedding 328 that identifies the respective language of the respective training utterance 310. The training process 300 can concatenate the speaker embedding 326 and the language embedding 328 and provide the concatenation as input to the text encoder 202 such that the text encoder generates the encoded text representation 312 based on the alignment output 402 and the concatenation of the speaker embedding 326 and the language embedding 328.

[0053] Accordingly, the supervised loss portion 300b can employ a shared encoder 250 that receives the encoded text representations 312 as input and generates first encoded shared representations 322 (e text ) as output. Similar to the text encoder 202, the TTS model 501 and the ASR model 200 can share this shared encoder 250. Moreover, the shared encoder 250 receives the encoded audio representations 314 as input and generates second encoded shared representations (e sup ) 324 as output. Accordingly, the shared encoder 250 generates the first encoded shared representations 322 and the second encoded shared representations 324 into a shared latent representation space that is compatible with the ASR decoder 390.

[0054] In particular, the shared encoder 250 receives each encoded text representation 312 corresponding to an aligned output 402 generated from a non-verbal text utterance 308 as input and, for each of the plurality of time steps, generates a first encoded shared representation (e text ) 322 corresponding to the aligned output 402 at the corresponding output step as output. The ASR decoder 390, which includes a phoneme decoder or a wordpiece decoder, receives each first encoded shared representation (i.e., shared encoder output) 332 output from the shared encoder 250 as input and, for the corresponding aligned output 402 at the corresponding output step, generates a first probability distribution 392 over possible speech recognition hypotheses as output. In some examples, the first probability distribution 392 over possible speech recognition hypotheses includes one of possible phoneme labels, possible wordpiece labels, or possible grapheme labels. Accordingly, the first probability distribution 392 over possible speech recognition hypotheses can represent a first speech recognition hypothesis that represents a candidate transcription for the corresponding training utterance 310. As such, the first probability distribution 392 can also be referred to herein as a first speech recognition hypothesis 392. Thereafter, the supervised loss module 340 can determine an aligned output loss term 342 based on the first probability distribution 392 over possible speech recognition hypotheses for the aligned output 402 corresponding to the non-verbal text utterance 308. Here, the corresponding non-verbal text utterance 308 from which the aligned output 402 is generated is also used as the ground truth transcription 302. Since the aligned output 402 can be masked, the aligned output loss term 342 also acts as a masked MLM loss. The supervised loss portion 300b can train the text encoder 202 and / or the speech encoder 204 with respect to the aligned output loss term 342 by updating the parameters of the text encoder 202 and / or the speech encoder 204 based on the aligned output loss term 342.

[0055] Similarly, during the supervised loss portion 300b, the shared encoder 250 receives as input each transcribed and encoded audio representation 314 corresponding to the transcribed speech utterance 304, and for each of the plurality of time steps, generates a second encoded shared representation (e) corresponding to the transcribed speech utterance 304 at the corresponding time step. sup The ASR decoder 390, including a phoneme decoder or a word-fragment decoder, receives each second encoded shared representation (i.e., shared encoder output) 334 from the shared encoder 250 as input and generates a second probability distribution 394 on possible speech recognition hypotheses as output for the corresponding transcribed speech utterance 304 at the corresponding time step. In some examples, the second probability distribution 394 on possible speech recognition hypotheses includes one of possible phoneme labels, possible word-fragment labels, or possible character labels. Therefore, the second probability distribution 394 on possible speech recognition hypotheses can represent a second speech recognition hypothesis that represents a candidate transcription for the corresponding training utterance 310. Thus, the second probability distribution 394 can also be referred to herein as the second speech recognition hypothesis 394. Subsequently, the supervised loss module 340 can determine a speech loss term 344 based on the second probability distribution 394 on possible speech recognition hypotheses and the corresponding transcription 302 paired with the transcribed speech utterance 304. Here, the corresponding transcription 302 is used as the ground truth transcription and may include sequences of target phonemes, target word fragments, and / or target character elements. The supervised loss part 300b can train the text encoder 202 and / or the speech encoder 204 for the speech loss term 344 by updating the parameters of the text encoder 202 and / or the speech encoder 204 based on the speech loss term 344.

[0056] Untranscribed spoken utterances 306 and non-spoken text utterances 308 each correspond to “unpaired” training data, thereby allowing the data to be transferred from non-spoken text utterances ( X text The contrast loss derived from 308 ( L w2v ) 316 and the supervision loss associated with the alignment output loss term 342 aux Combining to obtain a non-verbal text loss function text ,as follows.

[0057]

[0058] Similarly, untranscribed speech ( X unsup The contrast loss derived from 306 ( L w2v) 316 can be used to express the unsupervised speech loss function unsup_speech As follows.

[0059]

[0060] During training of the text encoder 202 and the speech encoder 204, the aligned outputs 402 and the untranscribed utterances 306 can be separated or mixed within each batch. To force the text encoder 202 to learn a representation that is valid for both non-synthetic (human / real) speech and aligned outputs 402 corresponding to non-spoken text utterances 308, the loss function combining equation 5 and equation 6 text is applied with a loss mask s to obtain the unpaired data loss function unpaired As follows.

[0061]

[0062] The transcribed speech utterances 304 correspond to “paired” and “supervised” training data, whereby the derived contrastive loss L w2v and the derived supervised loss associated with the speech loss term 344 aux are combined to obtain the paired data loss function paired As follows.

[0063]

[0064] Referring to Figure 3C , the consistency regularization part (i.e., the modality matching part) 300c of the training process 300 is configured to facilitate the text encoder 202 and the speech encoder 204 to learn consistent predictions between speech (e.g., real / human speech) and aligned outputs 402 corresponding to non-spoken text utterances 308 by generating a consistency loss term ) 352 between pairs of training utterances 303, each training utterance pair 303 comprising a transcribed speech utterance X sup) 304 and the paired alignment output 404 of the same speech utterance 304 as the corresponding transcribed speech utterance 304. Thus, the speech utterance 304 and the paired alignment output 404 in each training utterance pair 303 are associated with the same ground truth transcription. In short, the consistency loss term 352 between the transcribed speech utterance 304 and the paired alignment output 404 of the same training utterance 304 provides an unsupervised training aspect by encouraging the audio encoder 210 to perform consistently regardless of whether the training utterance belongs to speech (i.e., speech training data) or the alignment output (i.e., text training data), and provides this unsupervised training aspect independently of the supervised loss term between the ground truth transcription 302 and each of: the speech recognition hypothesis output by the auxiliary decoder 390; and the speech recognition hypothesis output by the auxiliary decoder 390.

[0065] Similarly to the alignment output 402 generated from the non-verbal text utterance 308 in Figure 3B The alignment model 400 can generate each paired alignment output 404 using the corresponding transcription 302 paired with the transcribed speech utterance 304. Here, the speech representation 304 is associated with the paired alignment output 404 generated by mapping the non-verbal text utterance 308 into speech frames by the alignment model 400.

[0066] During the consistency regularization portion 300c, the text encoder 202 receives each paired alignment output 404 as input and, for each of the plurality of time steps, generates as output an encoded text representation 313 corresponding to the paired alignment output 404 at the corresponding output step. In some examples, the text encoder 202 obtains a corresponding speaker embedding 326 that characterizes speaker characteristics of a corresponding speaker who spoke the training utterance in the respective language, and the text encoder generates the corresponding encoded text representation 312 based on a concatenation of the corresponding alignment output 402 and the corresponding speaker embedding 326. Also, in addition to or instead of the speaker embedding 326, the text encoder 202 can obtain a corresponding language embedding 328 that identifies the respective language of the respective training utterance 310. The training process 300 can concatenate the speaker embedding 326 and the language embedding 328 and provide the concatenation as input to the text encoder 202, such that the text encoder generates the encoded text representation 312 based on the alignment output 402 and the concatenation of the speaker embedding 326 and the language embedding 328.

[0067] The shared encoder 250 receives as input the encoded text representation 313 and generates a first encoded shared representation 315. sup) 323 is the output. An auxiliary decoder 390, including a phoneme decoder or a word-piece decoder, receives each first encoded shared representation 323 output from the shared encoder 250 as input and generates a first probability distribution 311 over possible speech recognition hypotheses as output for the corresponding paired aligned output 404 at the corresponding output step. In some examples, the first probability distribution 311 over possible speech recognition hypotheses includes either possible phoneme tags or possible word-piece tags.

[0068] Similarly, the speech encoder 204 receives as a sequence of features / vectors (e.g., Mel spectrograms, such as...). Figure 1 Each transcribed speech utterance 304 of the acoustic frame 110 is taken as input, and for each of a plurality of time steps, an encoded audio representation 314 corresponding to the transcribed speech utterance 304 at the corresponding output step is generated as output. The shared encoder 250 receives the encoded audio representation 314 as input and generates a second encoded shared representation (e sup The auxiliary decoder 390, including a phoneme decoder or a word-fragment decoder, receives each second encoded shared representation 324 output from the shared encoder 250 as input and generates a second probability distribution 394 on possible speech recognition hypotheses as output for the corresponding transcribed speech utterance 304 at the corresponding time step. In some examples, the second probability distribution 394 on possible speech recognition hypotheses includes either possible phoneme tags or possible word-fragment tags.

[0069] Continue to refer to Figure 3C The consistency regularization part 300c of the training process 300 further determines the consistency loss term of the corresponding training utterance pair 301 at each output step in the multiple output steps of each training utterance pair 301 based on the first probability distribution 311 and the second probability distribution 394 on the possible speech recognition assumptions. 352. For example, the training process 300 may employ a consistency loss term module 350, which is configured to receive the corresponding speech and speech recognition results 311, 394 output by the auxiliary decoder 390 at each time step, and determine the consistency loss term 352 of the corresponding training utterance pair 301 at the time step.

[0070] In some examples, the consistency regularization part 300c of the training process 300 is based on the Kullback-Leibler divergence between the first probability distribution 311 and the second probability distribution 394 on the possible speech recognition hypotheses. D KL To determine the consistent loss term 352. Based onD KL The consistency loss term 352 can be expressed with the following equation.

[0071]

[0072] Here, the consistency loss term 352 determined for each training utterance pair 301 at a time step provides an “unsupervised” loss term that is independent of the accuracy of the auxiliary decoder 390 (e.g., independent of the supervised loss terms 342, 344 of the alignment output) and, thus, can be used to update the parameters of the encoder 210 to promote consistency between the speech representation and the aligned output for the same utterance. In batch training, the consistency loss term 352 can correspond to an average loss term obtained for the batch. In other words, the consistency loss term 352 allows the text encoder 202 and the speech encoder 204 to learn to perform similarly, e.g., to make consistent encoded representation predictions for both the speech (e.g., real / human speech) and the aligned output for the same training utterance, regardless of whether the training utterance belongs to the speech or the aligned output. Figure 3B

[0073] In some implementations, the consistency loss module 350 receives the encoded text representation 313 generated by the text encoder 202 for the corresponding transcription 302 and the encoded audio representation (i.e., speech encoding) 314 generated by the speech encoder 204 for the corresponding reference speech representation (i.e., transcribed speech utterance) 304. Here, the training utterance pair 303 corresponds to the same training utterance 310, the consistency loss module 350 can determine a feature loss 354 between the encoded text representation 313 and the speech encoding 314 corresponding to the same training utterance 310. Thus, the consistency loss module 350 determines the feature loss 354 before decoding the encoded representations 313, 314 into speech recognition hypotheses. The training process 300 can train the TTS model 501 based on the feature loss 354 determined for each training utterance 310.

[0074] Finally, the training process 300 can combine the unpaired data loss function unpaired , the paired data loss function paired , and the consistency loss term to obtain an overall loss term that can be expressed as follows .

[0075]

[0076] where λ1may equal 1.0 and λ2equals 0.1. The training process 300 can use the overall loss term ​The audio encoder, speech encoder 204, and text encoder 202 are pre-trained by updating the parameters of the speech encoder 204 and the text encoder 202 to effectively teach the speech encoder 204 and the text encoder 202 to learn a shared representation between speech and text. After pre-training the speech encoder 204 and the text encoder 202, the training process 300 can fine-tune the pre-trained speech encoder 204 and the text encoder 202 on transcribed speech utterances, which can include supervised training samples of both aligned outputs and (e.g., human speech) corresponding to the non-spoken text utterances 308.

[0077] In some implementations, the training process 300 for pre-training the speech encoder 204 and the text encoder 202 applies encoder consistency regularization. Unlike decoder consistency regularization, which requires hypothesized labels (e.g., the transcripts 302 and the non-spoken text utterances 308) applied to the auxiliary decoder during the consistency regularization portion 300c, the encoder consistency regularization does not require hypothesized labels and thus has the advantage of allowing application to all training data 304, 306, 308. The encoder consistency regularization can be applied via a hierarchical contrastive consistency regularization (HCCR) technique, in which encoder activations e, e By auxiliary network projection to generate z and z Thereafter, the positive and negative pairs are constructive, and the contrastive loss l t,z,z is computed as follows.

[0078]

[0079] Specific to HCCR, a convolutional neural network (CNN) projection network can compute projections over length-increased segments (30, 50, 120 ms) of the encoder activations e to produce 3 views (V) and extract negative examples from the same utterance in short segments and from other utterances in the batch with 120 ms segments. Thus, the HCCR loss can be computed for the transcribed speech utterances 304 (paired speech), the untranscribed speech utterances 306 (unpaired speech), and the aligned outputs 402 generated from the non-spoken text utterances 308, as follows.

[0080]

[0081] The HCCR loss computed by Equation 11 can be added to Equation 9 with a coefficient of le-3 as part of the overall loss term for pre-training the speech encoder 204 and the text encoder 202.

[0082] Briefly, the training process 300 trains the TTS model 501 using the training utterances 310 set by training the speech decoder 204, the text encoder 202, and / or the shared encoder 250 based on any loss derived by the training process 300. Although the TTS model 501 can not employ the speech decoder 204 and the shared encoder 240 during inference, the training process 300 trains these components to learn a better shared representation between speech and text, thereby further training the TTS model 501 (e.g., the text encoder 202 of the TTS model 501) to generate encodings that accurately represent human speech.

[0083] Referring now to Figure 3D In some implementations, the training process 300 includes a training data generation process 300, 300d. Here, the speech encoder 204 receives the untranscribed speech utterances 306 and generates a corresponding unpaired speech encoding 314 for each respective untranscribed speech utterance (i.e., unpaired speech utterance) 306. The shared encoder 250 receives the unpaired speech encodings 314 and generates a corresponding unpaired shared encoder output 324 for each respective unpaired speech encoding 314. The auxiliary decoder 390 generates a pseudo-label 394 representing a candidate transcription for the corresponding unpaired spoken utterance 306. That is, the probability distribution 394 over possible speech recognition hypotheses can represent a single transcription such that the candidate transcription serves as a self-supervised label for the unpaired speech utterance 306. From this, the training process 300 can pair the pseudo-label 394 with the corresponding unpaired speech utterance 306 such that the pairings now represent transcribed speech utterances 304 that are added to the training data 301.

[0084] Referring now to Figure 3EIn some implementations, the training process 300 includes a language loss portion 300e. During the language loss portion 300e, the text encoder 202 is configured to receive the alignment output 402 (i.e., the text embedding) from the alignment model 400, and the speech encoder 204 is configured to receive the transcribed speech utterance 304. That is, the text encoder 202 generates an encoded text representation 312 for the alignment output 402 (e.g., corresponding to the non-verbal text utterance 308), and the speech encoder 204 of the encoder 210 generates an encoded audio representation 314 for the speech input (i.e., the transcribed speech utterance 304). In some examples, the text encoder 202 obtains a corresponding speaker embedding 326 that characterizes speaker characteristics of a corresponding speaker who spoke the training utterance in the respective language, and the text encoder generates the corresponding encoded text representation 312 based on a concatenation of the corresponding alignment output 402 and the corresponding speaker embedding 326. When the training utterance 310 includes synthetic speech, the speaker embedding 326 can represent an embedding input to a TTS model that generated the training utterance 310 to produce particular speech characteristics of the training utterance 310. Also, in addition to or instead of the speaker embedding 326, the text encoder 202 can obtain a corresponding language embedding 328 that identifies the respective language of the respective training utterance 310. The training process 300 can concatenate the speaker embedding 326 and the language embedding 328 and provide the concatenation as input to the text encoder 202, such that the text encoder generates the encoded text representation 312 based on the alignment output 402 and the concatenation of the speaker embedding 326 and the language embedding 328.

[0085] Language loss portion 300e can include a language identifier 360. Language identifier 360 can be integrated into any component of TTS model 501. For example, language identifier 360 can be integrated into an encoder or a decoder of TTS model 501. Language identifier 360 is configured to generate or predict a predicted language identifier 362 for a corresponding training utterance 310. That is, language identifier can generate predicted language identifier 362 based on encoded text representation 312 or generate predicted language identifier 362 based on encoded audio representation 314. Thereafter, language loss module 370 can receive predicted language identifier 362 predicted for each training utterance 310 and determine a text language identifier loss 372 or a speech language identifier loss 374. That is, predicted language identifier 362 generated from encoded text representation 312 can be compared to a corresponding language embedding 328 paired with training utterance 310 such that language loss module 370 determines text language identifier loss 372. Similarly, predicted language identifier 362 generated from encoded audio representation 314 can be compared to a corresponding language embedding 328 paired with training utterance 310 such that language loss module 370 determines speech language identifier loss 374. Training process 300 can update parameters of language identifier 360 and / or any other component of TTS model 501 based on language identifier losses 372, 374. Also, training process 300 can determine a TTS loss (i.e., overall loss) 305 based on any combination of losses determined during training process 300. The illustrated example shows language loss module 370 determining TTS loss 305 as an example only, as any loss module can determine TTS loss 305 and / or training process 300 can combine each loss from a loss module to determine TTS loss 305. Thus, TTS loss 305 can include any combination of losses determined during training process 300 or training process 500 Figure 5 ) such that training process 300 can train TTS model 501 by updating parameters of TTS model 501 based on TTS loss 305.

[0086] Figure 6 is a flowchart of example operational arrangements of a computer- implemented method 600 for large-scale multilingual speech-text joint semi-supervised learning of text-to-speech. Method 600 can be executed on data processing hardware 710 Figure 7 ) using instructions stored on memory hardware 720 Figure 7 ) can reside within a computing device 700 Figure 1 Figure 7 ​corresponding user device 102 and / or remote computing device 201.

[0087] At operation 602, the method 600 includes receiving training data 301 comprising a plurality of sets of training utterances 310. Each set of training utterances 310 is associated with a respective language that is different from the respective language associated with each other set of training utterances 310, and each set of training utterances comprises speech spoken in the respective language. Each training utterance 310 comprises a corresponding reference speech representation 304 paired with a corresponding input text sequence 302. For each training utterance 310 in each set of training utterances 310 of the received training data 301, the method 600 performs operations 604-610. At operation 604, the method 600 includes generating, using the text encoder 202, a corresponding encoded text representation 312, 313 for the corresponding input text sequence 302. At operation 606, the method 600 includes generating, using the speech encoder 204, a corresponding speech encoding 314 for the corresponding reference speech representation 304. At operation 608, the method 600 includes generating, using the shared encoder 250 configured to receive the corresponding encoded text representation 312, 313 or the corresponding speech encoding 314, a shared encoder output 332, 334. At operation 610, the method 600 includes determining a text-to-speech (TTS) loss 305 based on the corresponding encoded text representation 312, 313, the corresponding speech encoding 314, and the shared encoder output 332, 334. At operation 612, the method 600 includes training the TTS model 501 based on the TTS loss 305 determined for the training utterances 310 in each set of training utterances 310 to teach the TTS model to learn how to synthesize speech in each of the respective languages.

[0088] Figure 7 is a schematic diagram of an example computing device 700 that can be used to implement the systems and methods described in this document. The computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions, are meant to be examples for the purposes of this document only. The disclosure is not limited to implementations that include all of the components shown and / or to implementations that include all of the components described and / or illustrated herein.

[0089] The computing device 700 includes a processor 710, a memory 720, a storage device 730, a high-speed interface / controller 740 connecting the memory 720 and the high-speed expansion ports 750, and a low speed interface / controller 760 connecting the low speed bus 770 and the storage device 730. Each of the components 710, 720, 730, 740, 750, and 760 are interconnected using various busses, and can be mounted on a common motherboard or in other manners as appropriate. The processor 710 can process instructions for execution within the computing device 700, including instructions stored in the memory 720 or on the storage device 730 to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 780 coupled to the high-speed interface 740. In other implementations, multiple processors and / or multiple buses can be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices 700 can be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

[0090] The memory 720 stores information non-transitorily within the computing device 700. The memory 720 can be a computer-readable medium, a volatile memory unit(s), or non-volatile memory unit(s). The non-transitory memory 720 can be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by or

[0091] The storage device 730 is capable of providing mass storage for the computing device 700. In some implementations, the storage device 730 is a computer- readable medium. In various different implementations, the storage device 730 can be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 720, the storage device 730, or memory on processor 710.

[0092] The high-speed controller 740 manages bandwidth-intensive operations for the computing device 700, while the low speed controller 740 manages lower bandwidth-intensive operations. Such allocation of functions is exemplary only. In some implementations, the high-speed controller 740 is coupled to memory 720, display 780 (e.g., through a graphics processor or accelerator), and to high-speed expansion ports 750, which can accept various expansion cards (not shown). In some implementations, the low-speed controller 740 is coupled to storage device 730 and low-speed expansion port 790. The low-speed expansion port 790, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

[0093] The computing device 700 can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a standard server 700a or multiple times in a group of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

[0094] Various implementations of the systems and techniques described here can be realized in digital electronic and / or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0095] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer readable medium, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0096] The processes and logic flows described in this specification can be performed by one or more programmable processors, also referred to as data processing hardware, that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical, or optical disks, or

[0097] To provide for interaction with a user, one or more aspects of the disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display), or touch screen, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device of the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0098] A variety of implementations have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Accordingly, other implementations are within the scope of the following claims.

Claims

1. A computer-implemented method (600) executed on data processing hardware (710) to cause the data processing hardware (710) to perform operations, the operations including: Receive training data (301) comprising multiple sets of training utterances (310), each set of training utterances (310) being associated with a corresponding language and including speech spoken in the corresponding language, the corresponding language being different from the corresponding language associated with each other set of training utterances (310), each training utterance (310) including a corresponding reference speech representation (304) paired with a corresponding input text sequence (302); For each training utterance (310) in each group of training utterances (310) of the received training data (301): The text encoder (202) is used to generate the corresponding encoded text representation (312) for the corresponding input text sequence (302); The speech encoder (204) generates the corresponding speech code (314) for the corresponding reference speech representation (304); Using a shared encoder (250) configured to receive the corresponding encoded text representation (312, 313) or the corresponding speech code (314), a shared encoder output (332, 334) is generated; and The text-to-speech (TTS) loss (305) is determined based on the corresponding encoded text representations (312, 313), the corresponding speech encoding (314), and the shared encoder output (332, 334); and The TTS model (501) is trained based on the TTS loss (305) determined for each training utterance (310) in each set of training utterances (310) to teach the TTS model (501) to learn how to synthesize speech in each of the corresponding languages.

2. The computer-implemented method (600) as described in claim 1, wherein, The operation further includes, for each training utterance (310) in each group of the received training data (301): Obtain the corresponding speaker embedding (326), which represents the speaker characteristics of the speaker who spoke the training utterance (310) in the corresponding language; as well as Obtain the corresponding language embedding (328) of the language that identifies the utterance. The text encoder (202) is configured to receive a link between the corresponding speaker embedding (326) and the corresponding language embedding (328).

3. The computer-implemented method (600) as described in claim 1 or 2, wherein, The operation further includes, for each training utterance (310) in each group of the received training data (301): Speech recognition hypotheses (392, 394) are generated using an automatic speech recognition decoder (390) configured to receive the outputs (332, 334) of the shared encoder as input, the speech recognition hypotheses (392, 394) representing candidate transcriptions for the corresponding training utterance (310); as well as The ASR loss (342, 344) is determined based on the speech recognition hypotheses (392, 394) and the corresponding input text sequence (302). The TTS loss (305) includes the ASR loss (342, 344).

4. The computer-implemented method (600) as described in claim 3, wherein, The ASR decoder (390) includes a recurrent neural network transducer (RNN-T) architecture.

5. The computer-implemented method (600) as described in any one of claims 1 to 4, wherein, The operation further includes, for each training utterance (310) in each group of the received training data (301): The feature loss (354) is determined between the encoded text representation (313) generated by the text encoder (302) for the corresponding input text sequence (302) and the speech encoding (314) generated by the speech encoder (204) for the corresponding reference speech representation (304). The TTS loss (305) includes the feature loss (354).

6. The computer-implemented method (600) as described in any one of claims 1 to 5, wherein, The operation further includes, for each training utterance (310) in each group of the received training data (301): Obtain the sequence representation (412) of the corresponding input text sequence (302) connected to the variational embedding (404); Using the duration model (420): The duration (422) of the input text sequence (302) is predicted based on the sequence representation (412); and Based on the duration (422) of the input text sequence (302), the sequence representation (412) is upsampled to an upsampled output (402) of a specified number of frames; and The duration loss (554) is determined based on the predicted duration (422) and the true duration (406) of the input text sequence (302). The TTS loss (305) includes the duration loss (554).

7. The computer-implemented method (600) as described in any one of claims 1 to 6, wherein: The training data (301) further includes non-verbal text utterances (308) associated with multiple different languages, each non-verbal text utterance (308) not paired with any corresponding verbal utterance; and The operation further includes, for each non-verbal text utterance (308): The text encoder (202) is used to generate corresponding non-verbal encoded text representations (312, 313) for the corresponding non-verbal text utterances (308); and Determine the masked language modeling MLM loss (318) for the aligned text generated for the corresponding non-verbal encoded text representation (312, 313) for the corresponding non-verbal text utterance (308). The TTS loss (305) includes the MLM loss (318) of the aligned text.

8. The computer-implemented method (600) as described in claim 7, wherein: Each non-verbal text utterance (308) is paired with a corresponding language identifier tag (328); The operation further includes, for each non-verbal text utterance (308): A language recognizer (360) configured to receive a corresponding non-verbal encoded text representation (312, 313) generated for the corresponding non-verbal text utterance (308) as input is used to generate a predicted language identifier (362). as well as The text language identifier loss (372) is determined based on the predicted language identifier (362) and the language identifier label (362). The TTS loss (305) includes the text language identifier loss (372).

9. The computer-implemented method (600) as described in any one of claims 1 to 8, wherein: The training data (301) further includes unpaired spoken utterances (306) spoken in a plurality of different languages, each unpaired spoken utterance (306) not paired with any corresponding text; and The operation further includes, for each unpaired spoken utterance (306): The speech encoder (204) is used to generate a corresponding unpaired speech code (314) for the corresponding unpaired spoken utterance (306); and Determine the masked language modeling MLM loss (319) for the aligned speech generated for the corresponding unpaired speech code (314) for the corresponding unpaired spoken utterance (306). The TTS loss (305) includes the MLM loss (319) of the aligned speech.

10. The computer-implemented method (600) as described in claim 9, wherein, The operation further includes, for each unpaired spoken utterance (306): The shared encoder (250), which is further configured to receive the corresponding unpaired speech code (314), is used to generate the unpaired shared encoder output (332, 334); as well as An automatic speech recognition (ASR) decoder (390) configured to receive the unpaired shared encoder output (332, 334) as input is used to generate pseudo-labels (394) representing candidate transcriptions for the corresponding unpaired spoken utterance (306). The training data (301) further includes non-verbal text discourse (308), which includes the pseudo-labels (394).

11. The computer-implemented method (600) as described in claim 9 or 10, wherein: Each unpaired spoken word (306) is paired with a corresponding language identifier tag (328); The operation further includes, for each unpaired spoken utterance (306): A language recognizer (362) configured to receive the corresponding unpaired speech code (314) for the corresponding unpaired spoken utterance (306) as input is used to generate a predicted language identifier (362). as well as The speech language identifier loss (374) is determined based on the predicted language identifier (362) and the language identifier label (328). The TTS loss (305) includes the speech language identifier loss (374).

12. The computer-implemented method (600) as described in any one of claims 1 to 11, wherein, Each corresponding input text sequence (302) includes a sequence of characters, word slice model units, phonemes, or bytes.

13. The computer-implemented method (600) as described in claim 1, wherein, Generating the speech code (314) for the corresponding reference speech representation (304) includes: The corresponding utterance is projected using a random projection quantizer (217); and Map the corresponding projected utterance (219) to discrete labels.

14. A system (100) comprising: Data processing hardware (710); as well as A memory hardware (720) communicating with the data processing hardware (710), the memory hardware (720) storing instructions that, when executed on the data processing hardware (710), cause the data processing hardware (710) to perform operations, the operations including: Receive training data (301) comprising multiple sets of training utterances (310), each set of training utterances (310) being associated with a corresponding language and including speech spoken in the corresponding language, the corresponding language being different from the corresponding language associated with each other set of training utterances (310), each training utterance (310) including a corresponding reference speech representation (304) paired with a corresponding input text sequence (302); For each training utterance (310) in each group of training utterances (310) of the received training data (301): The text encoder (202) is used to generate the corresponding encoded text representation (312) for the corresponding input text sequence (302); The speech encoder (204) generates the corresponding speech code (314) for the corresponding reference speech representation (304); Using a shared encoder (250) configured to receive the corresponding encoded text representation (312, 313) or the corresponding speech code (314), a shared encoder output (332, 334) is generated; and The text-to-speech (TTS) loss (305) is determined based on the corresponding encoded text representations (312, 313), the corresponding speech encoding (314), and the shared encoder output (332, 334); and The TTS model (501) is trained based on the TTS loss (305) determined for each training utterance (310) in each set of training utterances (310) to teach the TTS model (501) to learn how to synthesize speech in each of the corresponding languages.

15. The system (100) as claimed in claim 14, wherein, The operation further includes, for each training utterance (310) in each group of the received training data (301): Obtain the corresponding speaker embedding (326), which represents the speaker characteristics of the speaker who spoke the training utterance (310) in the corresponding language; as well as Obtain the corresponding language embedding (328) of the language that identifies the utterance. The text encoder (202) is configured to receive a link between the corresponding speaker embedding (326) and the corresponding language embedding (328).

16. The system (100) as claimed in claim 14 or 15, wherein, The operation further includes, for each training utterance (310) in each group of the received training data (301): Speech recognition hypotheses (392, 394) are generated using an automatic speech recognition decoder (390) configured to receive the outputs (332, 334) of the shared encoder as input, the speech recognition hypotheses (392, 394) representing candidate transcriptions for the corresponding training utterance (310); as well as The ASR loss (342, 344) is determined based on the speech recognition hypotheses (392, 394) and the corresponding input text sequence (302). The TTS loss (305) includes the ASR loss (342, 344).

17. The system (100) as claimed in claim 16, wherein, The ASR decoder (390) includes a recurrent neural network transducer (RNN-T) architecture.

18. The system (100) as claimed in any one of claims 14 to 17, wherein, The operation further includes, for each training utterance (310) in each group of training utterances (310) of the received training data (301): The feature loss (354) is determined between the encoded text representation (313) generated by the text encoder (302) for the corresponding input text sequence (302) and the speech encoding (314) generated by the speech encoder (204) for the corresponding reference speech representation (304). The TTS loss (305) includes the feature loss (354).

19. The system (100) as claimed in any one of claims 14 to 18, wherein, The operation further includes, for each training utterance (310) in each group of the received training data (301): Obtain the sequence representation (412) of the corresponding input text sequence (302) connected to the variational embedding (404); Using the duration model (420): The duration (422) of the input text sequence (302) is predicted based on the sequence representation (412); and Based on the duration (422) of the input text sequence (302), the sequence representation (412) is upsampled to an upsampled output (402) of a specified number of frames; and The duration loss (554) is determined based on the predicted duration (422) and the true duration (406) of the input text sequence (302). The TTS loss (305) includes the duration loss (554).

20. The system (100) as claimed in any one of claims 14 to 19, wherein: The training data (301) further includes non-verbal text utterances (308) associated with multiple different languages, each non-verbal text utterance (308) not paired with any corresponding verbal utterance; and The operation further includes, for each non-verbal text utterance (308): The text encoder (202) is used to generate corresponding non-verbal encoded text representations (312, 313) for the corresponding non-verbal text utterances (308); and Determine the masked language modeling MLM loss (318) for the aligned text generated for the corresponding non-verbal encoded text representation (312, 313) for the corresponding non-verbal text utterance (308). The TTS loss (305) includes the MLM loss (318) of the aligned text.

21. The system (100) of claim 20, wherein: Each non-verbal text utterance (308) is paired with a corresponding language identifier tag (328); The operation further includes, for each non-verbal text utterance (308): A language recognizer (360) configured to receive a corresponding non-verbal encoded text representation (312, 313) generated for the corresponding non-verbal text utterance (308) as input is used to generate a predicted language identifier (362). as well as The text language identifier loss (372) is determined based on the predicted language identifier (362) and the language identifier label (362). The TTS loss (305) includes the text language identifier loss (372).

22. The system (100) as claimed in any one of claims 14 to 21, wherein: The training data (301) further includes unpaired spoken utterances (306) spoken in a plurality of different languages, each unpaired spoken utterance (306) not paired with any corresponding text; and The operation further includes, for each unpaired spoken utterance (306): The speech encoder (204) is used to generate a corresponding unpaired speech code (314) for the corresponding unpaired spoken utterance (306); and Determine the masked language modeling MLM loss (319) for the aligned speech generated for the corresponding unpaired speech code (314) for the corresponding unpaired spoken utterance (306). The TTS loss (305) includes the MLM loss (319) of the aligned speech.

23. The system (100) as claimed in claim 22, wherein, The operation further includes, for each unpaired spoken utterance (306): The shared encoder (250), which is further configured to receive the corresponding unpaired speech code (314), is used to generate the unpaired shared encoder output (332, 334); as well as An automatic speech recognition (ASR) decoder (390) configured to receive the unpaired shared encoder output (332, 334) as input is used to generate pseudo-labels (394) representing candidate transcriptions for the corresponding unpaired spoken utterance (306). The training data (301) further includes non-verbal text discourse (308), which includes the pseudo-labels (394).

24. The system (100) as claimed in claim 22 or 23, wherein: Each unpaired spoken word (306) is paired with a corresponding language identifier tag (328); The operation further includes, for each unpaired spoken utterance (306): A language recognizer (362) configured to receive the corresponding unpaired speech code (314) for the corresponding unpaired spoken utterance (306) as input is used to generate a predicted language identifier (362). as well as The speech language identifier loss (374) is determined based on the predicted language identifier (362) and the language identifier label (328). The TTS loss (305) includes the speech language identifier loss (374).

25. The system (100) as claimed in any one of claims 14 to 24, wherein, Each corresponding input text sequence (302) includes a sequence of characters, word slice model units, phonemes, or bytes.

26. The system (100) as claimed in claim 14, wherein, Generating the speech code (314) for the corresponding reference speech representation (304) includes: The corresponding utterance is projected using a random projection quantizer (217); and Map the corresponding projected utterance (219) to discrete labels.

Images