Phonemes and Graphemes for Neural Text-to-Speech

Abstract

The method (400) includes receiving a text input (152) including a sequence of words represented as an input encoder embedding (210). The input encoder embedding includes a plurality of tokens (212), the plurality of tokens including a first set of grapheme tokens (212G) representing the text input as respective graphemes and a second set of phoneme tokens (212P) representing the text input as respective phonemes. The method also includes, for each phoneme token in the second set of phoneme tokens, identifying a respective word in the sequence of words corresponding to each phoneme token, and determining a respective grapheme token representing each word in the sequence of words corresponding to each phoneme token. The method also includes generating an output encoder embedding (Vc) based on a relationship between each phoneme token and the corresponding grapheme token determined to represent the same respective word as the respective phoneme token.

Description

[Technical field]

[0001] This disclosure relates to phonemes and graphemes for neural text-to-speech (TTS). [Background technology]

[0002] Speech synthesis systems use text-to-speech (TTS) models to generate speech from text input. The generated / synthesized speech must sound like a human speech (naturalness) with the intended prosody (expressiveness) while simultaneously conveying the message accurately (intelligibility). Speech synthesis systems have evolved from concatenative and parametric synthesis models to models with neural networks. Although this evolution has significantly improved the naturalness of synthetic speech, current TTS models are unable to fully preserve the input text fed into the TTS model to generate synthetic speech. That is, most TTS models, even neural network TTS models, convert the input text into a phonemic representation without preserving the corresponding graphemes. Due to the lack of graphemes, the TTS model cannot exploit the relationship between phonemes and graphemes to further improve the capabilities of the TTS model. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Devlin, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," available at https: / / arxiv.org / pdf / 1810.04805v2.pdf Summary of the Invention [Means for solving the problem]

[0004] One aspect of the disclosure provides a computer-implemented method for implementing phoneme and grapheme tokens for neural text-to-speech synthesis. The computer-implemented method, when executed on data processing hardware, causes the data processing hardware to perform operations including receiving a text input including a sequence of words represented as an input encoder embedding at an encoder of a speech synthesis model. The input encoder embedding includes a plurality of tokens, the plurality of tokens including a first set of grapheme tokens representing the text input as respective graphemes and a second set of phoneme tokens representing the text input as respective phonemes. The operations also include identifying, for each phoneme token of the second set of phoneme tokens, a respective word of the sequence of words corresponding to each phoneme token, and determining a respective grapheme token representing each word of the sequence of words corresponding to each phoneme token. The operations also include generating an output encoder embedding based on a relationship between each phoneme token and the corresponding grapheme token determined by the encoder to represent the same respective word as the respective phoneme token.

[0005] Implementations of this disclosure may include one or more of any of the following features: In some implementations, identifying, by the encoder, each word of the sequence of words corresponding to each phoneme token includes identifying a position within each word corresponding to each phoneme token, and determining, by the encoder, each grapheme token representing each word of the sequence of words corresponding to each phoneme token includes determining, at a position within each word corresponding to each phoneme token, each grapheme token representing each word of the sequence of words corresponding to each phoneme token.

[0006] In some examples, each token of the multiple tokens of the input encoder embedding represents one of a grapheme token embedding or a phoneme token embedding, a combination of segment embeddings, word position embeddings, and / or position embeddings. In these examples, identifying each word of the sequence of words corresponding to each phoneme token may include identifying each word of the sequence of words corresponding to each phoneme token based on a respective word position embedding associated with the respective phoneme token. Here, determining each grapheme token representing each word of the sequence of words corresponding to each phoneme token may include determining that each grapheme token includes a corresponding word position embedding that matches a respective word position embedding of the respective phoneme token.

[0007] In some implementations, the speech synthesis model includes an attention mechanism in communication with the encoder. The speech synthesis model may include a duration-based upsampler in communication with the encoder. Additionally, the tokens of the input encoder embedding may include a special token that identifies the language of the input text.

[0008] The operations may also include pre-training the encoder of the speech synthesis model by providing a plurality of training examples to the encoder, each training example being represented as a sequence of training grapheme tokens corresponding to a training sequence of words and a sequence of training phoneme tokens corresponding to the same training sequence of words, masking training phoneme tokens from the sequence of training phoneme tokens for each word from the training sequence of words, and masking training grapheme tokens from the sequence of training phoneme tokens for each word from the training sequence of words.

[0009] In some implementations, the speech synthesis model includes a multilingual speech synthesis model and the operations further include pre-training an encoder of the speech synthesis model using the classification goal to predict classification tokens for a plurality of tokens of the input encoder embedding, the classification tokens including a language identifier. In other implementations, the speech synthesis model includes a multilingual speech synthesis model and the output encoder embedding includes a sequence of encoder tokens, each encoder token including linguistic information for the input text.

[0010] In another implementation, the speech synthesis model includes a multi-accent speech synthesis model, and the operations further include pre-training an encoder of the speech synthesis model using the classification target to predict classification tokens for a plurality of tokens of an input encoder embedding, where the classification tokens include an accent identifier.

[0011] Another aspect of the disclosure provides a system for implementing phoneme and grapheme tokens for neural text-to-speech synthesis. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving a text input including a sequence of words represented as an input encoder embedding at an encoder of a speech synthesis model. The input encoder embedding includes a plurality of tokens, the plurality of tokens including a first set of grapheme tokens representing the text input as respective graphemes and a second set of phoneme tokens representing the text input as respective phonemes. The operations also include, for each phoneme token of the second set of phoneme tokens, identifying a respective word of the sequence of words corresponding to each phoneme token, and determining a respective grapheme token representing a respective word of the sequence of words corresponding to each phoneme token. The operations also include generating an output encoder embedding based on a relationship between each phoneme token and the corresponding grapheme token determined by the encoder to represent the same respective word as the respective phoneme token.

[0012] This aspect may include one or more of any of the following features: In some implementations, identifying, by the encoder, each word of the sequence of words corresponding to each phoneme token includes identifying a position within each word corresponding to each phoneme token, and determining, by the encoder, each grapheme token representing each word of the sequence of words corresponding to each phoneme token includes determining each grapheme token representing each word of the sequence of words corresponding to each phoneme token at a position within each word corresponding to each phoneme token.

[0013] In some examples, each token of the multiple tokens of the input encoder embedding represents one of a grapheme token embedding or a phoneme token embedding, a combination of segment embeddings, word position embeddings, and / or position embeddings. In these examples, identifying each word of the sequence of words corresponding to each phoneme token may include identifying each word of the sequence of words corresponding to each phoneme token based on a respective word position embedding associated with the respective phoneme token. Here, determining each grapheme token representing each word of the sequence of words corresponding to each phoneme token may include determining that each grapheme token includes a corresponding word position embedding that matches a respective word position embedding of the respective phoneme token.

[0014] In some implementations, the speech synthesis model includes an attention mechanism in communication with the encoder. The speech synthesis model may include a duration-based upsampler in communication with the encoder. Additionally, the tokens of the input encoder embedding may include a special token that identifies the language of the input text.

[0015] The operations may also include pre-training the encoder of the speech synthesis model by providing a plurality of training examples to the encoder, each training example being represented as a sequence of training grapheme tokens corresponding to a training sequence of words and a sequence of training phoneme tokens corresponding to the same training sequence of words, masking training phoneme tokens from the sequence of training phoneme tokens for each word from the training sequence of words, and masking training grapheme tokens from the sequence of training phoneme tokens for each word from the training sequence of words.

[0016] In some implementations, the speech synthesis model includes a multilingual speech synthesis model and the operations further include pre-training an encoder of the speech synthesis model using the classification goal to predict classification tokens for a plurality of tokens of the input encoder embedding, the classification tokens including a language identifier. In other implementations, the speech synthesis model includes a multilingual speech synthesis model and the output encoder embedding includes a sequence of encoder tokens, each encoder token including linguistic information for the input text.

[0017] In another implementation, the speech synthesis model includes a multi-accent speech synthesis model, and the operations further include pre-training an encoder of the speech synthesis model using the classification target to predict classification tokens for a plurality of tokens of an input encoder embedding, where the classification tokens include an accent identifier.

[0018] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a schematic diagram of an exemplary audio environment. [Figure 2A] 1 is a schematic diagram of an example extension encoder for a text-to-speech model. [Figure 2B] 1 is a schematic diagram of an example extension encoder for a text-to-speech model. [Figure 2C] 1 is a schematic diagram of an example extension encoder for a text-to-speech model. [Diagram 3] FIG. 2 is a schematic diagram of an example training process for an extended encoder. [Figure 4] 1 is a flowchart of an exemplary configuration of method operations for generating an output encoder embedding using both phonemes and graphemes. [Diagram 5]FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Like reference symbols in the various drawings indicate like elements.

[0021] The rapid development of neural end-to-end text-to-speech (TTS) synthesis has enabled speech production to approach levels of human naturalness. The neural networks of these TTS systems typically include an encoder that encodes the input text representation into a hidden state, and a decoder that decodes spectrogram frames or waveform samples from the hidden state. These TTS systems may use attention or duration-based upsamplers to connect the encoder to the decoder.

[0022] During the development of these TTS systems, the form of the input to the neural network representing the text evolved. For example, early TTS systems would take the pure characters of the input text as input to the neural network. However, over time, it was discovered that the stability of the TTS system could be improved if the input to the neural network was phonemes (i.e., the way the text representation is pronounced) instead of pure characters. Unfortunately, phoneme-based TTS models are not without their drawbacks. For example, one obvious problem with phoneme-based models occurs when two words have the same pronunciation, yet these two words actually have different meanings; i.e., homonyms. For example, the sentence "To cancel payment, press 1. To continue, press" is an example of a homonym and may be frequently used by conversational AI agents in call centers. In the phoneme representation of this sentence, the ending "..., two" may be easily confused with "..., too" because "too" is a more frequently occurring word in regular English. That is, the predictive system is more likely to output the word "too" even if "two" was intended. In this example, different prosody in natural speech is expected at the comma position in the two patterns: a moderate pause is expected at the comma in "..., two", but there is often no pause in the pattern "..., too". If the TTS model is phoneme-based, the phoneme input to a neural network of a text representation of this example sentence lacks the context of the written sentence to ensure a consistently accurate output, i.e., an output with a pause after the comma in the synthetic speech to make the synthetic speech sound like the input sentence.

[0023] Therefore, to overcome such problems with phoneme-based models, it is advantageous for the input to the neural network of the TTS model to include both phonemes and graphemes. That is, the input is a representation of the pronunciation (i.e., phonemes) of the input text as well as a representation of the writing (i.e., graphemes) of the text. Ideally, the inclusion of graphemes reduces or eliminates problems due to pronunciation ambiguity (e.g., homonyms). However, generating input to a neural network that includes both phonemes and graphemes is not as easy as it may seem. One complication is that phonemes and graphemes may represent the same content, yet may be of different lengths. For example, a word in a sentence may have two subwords or graphemes and only one phoneme, or in the opposite example, a word in a sentence may have one subword or grapheme (e.g., a subword is the same as a word) and two phonemes. Thus, a first sequence of phonemes representing a sentence has an inherent alignment problem with a second sequence of graphemes representing the same sentence.

[0024] Although this problem exists, several approaches attempt to combine phoneme and grapheme representations for the input. Instead of proposing a solution to this alignment problem, these approaches generally tend to avoid the problem. For example, one approach is to combine phoneme and grapheme representations at the word level by concatenating grapheme-based embeddings with phoneme embeddings. These approaches do not address the challenge of alignment between phonemes and grapheme-based tokens (i.e., at the subword level), and therefore cannot fully exploit the phoneme-grapheme relationship. That is, these approaches limit the potential (e.g., in terms of accuracy) that grapheme incorporation can provide to a TTS model.

[0025] To address the problems of previous approaches, the current technique described herein considers alignment (or misalignment) between phonemes and graphemes. More specifically, the approach represents alignment relationships (i.e., at the subword or tokenization level) between phoneme tokens and grapheme tokens across the text represented as input to the neural network (i.e., input to the neural network's encoder). The approach is similar to the encoder of the Bidirectional Encoder Representations from Transformers (BERT) model (e.g., as described in "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" by Devlin, available at https: / / arxiv.org / pdf / 1810.04805v2.pdf, incorporated herein by reference). In the traditional BERT model, the encoder received inputs corresponding to multiple sentences (i.e., segments) identified by segment identifiers (IDs). The input to BERT represented multiple sentences as a sum of phoneme-based token embeddings, segment embeddings, and positional embeddings. Although similar in some respects, the present technique may be considered as Extended BERT or PnG BERT since the technique includes phonemes and graphemes for token embeddings. To consider graphemes at the token level, Extended BERT not only includes the token embeddings, segment embeddings, and positional embeddings of traditional BERT, but also includes word position embeddings that provide word-level alignment between phonemes and graphemes. Thus, the input to Extended BERT is a representation of the four types of embeddings, whereas traditional BERT had no representation of graphemes and only included three types of embeddings.

[0026] Because the extended BERT method represents phoneme-grapheme relationships as input to the encoder of a neural network, this extended BERT encoder (also called the extended encoder) can be used in various types of TTS models. That is, because neural networks are usually encoder-decoder structures with attention or duration-based upsampling, the extended encoder can replace the encoder in other encoder-decoder structures of the TTS model. This means that the extended encoder is compatible with both attention-based neural networks and duration-based neural networks without significant modification. Therefore, functionally speaking, the extended encoder can be used in monolingual TTS models, locale TTS models, multilingual TTS models, and / or multi-accent TTS models.

[0027] Another reason why current techniques are built on the BERT model is that BERT is a model architecture that uses pre-training to improve natural language processing. Generally speaking, the pre-training is self-supervised pre-training on a large text corpus using objectives such as language models (LMs) or masked language models (MLMs). For traditional BERT, pre-training was only performed at the grapheme (subword level) or sentence level. Traditionally, pre-training was not done on phonemes. Since the extended BERT can model the relationship between phonemes and graphemes, this relationship can be translated into pre-training so that the extended BERT can pre-train on both phonemes and graphemes.

[0028] Referring to FIG. 1, in some implementations, a voice environment 100 includes a user 10 who communicates a voice utterance 12 to a voice-enabled device 110 (also referred to as a device 110 or a user device 110). The user 10 (i.e., the speaker of the utterance 12) can speak the utterance 12 as a query or command to solicit a response from the device 110 or to cause the device 110 to perform a task specified by the query. The device 110 is configured to capture sounds from one or more users 10 in the voice environment 100. Here, audio sound may refer to an utterance 12 by the user 10 that serves as an audible query, a command to the device 110, or an audible communication captured by the device 110. A voice-enabled system (e.g., a digital assistant interface) of or associated with the device 110 may field the query to a command by responding to the query and / or causing the command to be performed.

[0029] Here, the device 110 captures audio data 14 corresponding to utterances 12 by the user 10. The device 110 may correspond to any computing device associated with the user 10 and capable of receiving the audio data 14. Some examples of the user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, e-book readers, etc.), computers, wearable devices (e.g., smart watches), music players, casting devices, smart appliances (e.g., smart TVs), and Internet of Things (IoT) devices, remote controls, smart speakers, etc. The device 110 includes data processing hardware 112 and memory hardware 114 that communicates with the data processing hardware 112 and stores instructions that, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations related to voice and / or text processing. In some examples, device 110 includes one or more applications (i.e., software applications), each of which may utilize one or more voice processing systems 140, 150, 200 associated with device 110 to perform various functions within the application. For example, device 110 includes an assistant application configured to communicate synthetic playback audio 154 (also referred to as synthetic speech 154) to user 10 to assist user 10 with various tasks.

[0030] Device 110 further includes an audio subsystem with an audio capture device (e.g., a microphone) 116 for capturing and converting audio data 14 in audio environment 100 into an electrical signal, and an audio output device (e.g., a speaker) 118 for communicating an audible audio signal (e.g., a composite playback signal 154 from device 110). In the illustrated example, device 110 implements a single audio capture device 116, but device 110 may implement an array of audio capture devices 116 without departing from the scope of this disclosure, whereby one or more audio capture devices 116 in the array may not be physically resident on device 110, but may be in communication with the audio subsystem (e.g., peripherals of device 110). For example, device 110 may correspond to a vehicle infotainment system that utilizes an array of microphones located throughout the vehicle.

[0031] Additionally, the device 110 is configured to communicate with a remote system 130 over the network 120. The remote system 130 may include remote resources 132, such as remote data processing hardware 134 (e.g., a remote server or CPU) and / or remote memory hardware 136 (e.g., a remote database or other storage hardware). The device 110 may utilize the remote resources 132 to perform various functions related to voice processing and / or synthesis playback communications. For example, the device 110 is configured to perform voice recognition using a voice recognition system 140 and / or text-to-speech conversion using a TTS system 150 (e.g., using a TTS model 200). These systems 140, 150, 200 may reside on the device 110 (referred to as on-device systems) or may reside remotely (e.g., reside on the remote system 130), but are in communication with the device 110. In some examples, some of these systems 140, 150, 200 reside locally or on the device, while other systems reside remotely. In other words, any of these systems 140, 150, 200 may be local or remote in any combination. For example, if the size or processing requirements of the systems 140, 150, 200 are significant, the systems 140, 150, 200 may reside in a remote system 130. Additionally, if the device 110 can support the size or processing requirements of one or more of the systems 140, 150, 200, one or more of the systems 140, 150, 200 may reside on the device 110 using the data processing hardware 112 and / or memory hardware 114. Optionally, one or more of the systems 140, 150, 200 may reside both locally / on the device and remotely.For example, one or more of systems 140, 150, 200 may run by default on remote system 130 when a connection to network 120 between device 110 and remote system 130 is available, but if the connection is lost or network 120 is unavailable, systems 140, 150, 200 will run locally on device 110 instead.

[0032] The speech recognition system 140 receives audio data 14 as input and transcribes the audio signal into a transcript 142 as output. Generally speaking, by converting the audio data 14 into a transcript 142, the speech recognition system 140 enables the device 110 to recognize when an utterance 12 from the user 10 corresponds to a query, a command, or some other form of audio communication. The transcript 142 refers to a sequence of text that the device 110 may use to generate a response to the query or command. For example, when the user 10 asks the device 110 the question "What will the weather be like today?", the device 110 passes audio data 14 corresponding to the question "What will the weather be like today?" to the speech recognition system 140. The speech recognition system 140 converts the audio data 14 into a transcript that includes the text "What will the weather be like today?". The device 110 may then use the text or a portion of the text to determine a response to the query. For example, to determine the weather for the current day (i.e., today), device 110 passes text (e.g., "What's the weather going to be like today?"), or identifying portions of the text (e.g., "weather" and "today"), to a search engine. The search engine may then return one or more search results that device 110 interprets to generate a response to user 10.

[0033] In some implementations, the device 110 or a system associated with the device 110 identifies text 152 (also referred to as a sequence of text 152 or input text 152) that the device 110 communicates to the user 10 in response to the query of the utterance 12. The device 110 may then use the TTS system 150 to convert the text 152 into a corresponding synthetic playback audio 154 for the device 110 to communicate to the user 10 in response to the query of the utterance 12 (e.g., audibly communicate to the user 10). In other words, the TTS system 150 receives the text 152 as input and converts the text 152 (e.g., through a series of neural networks) into an output of the synthetic playback audio 154, which is an audio signal that defines an audible representation of the text 152. For example, the playback audio 154 is a verbalization or narration of the input text 152. In some examples, the input text 152 refers to a sequence of text or characters in a particular natural language (e.g., English, Spanish, or French). The sequence of characters can include letters, numbers, punctuation marks, and / or other special characters. When the TTS system 150 generates the playback audio 154, the playback audio 154 is synthetic speech in that it approximates how a human would verbalize the sequence of characters that defines the input text 152.

[0034] The TTS system 150 (or other speech synthesis system) includes a TTS model 200 (e.g., the TTS model 200 of FIG. 2) that utilizes a deep neural network to generate the synthetic playback audio 154. The TTS model 200 processes the embedding, which is a coded representation of speech features (e.g., features of the input text 152), to generate an audio waveform (e.g., a time-domain audio waveform that defines the amplitude of the audio signal over time). Once generated, the TTS system 150 communicates the synthetic playback audio 154 to the device 110, enabling the device 110 to output the synthetic playback audio 154. For example, the device 110 outputs the synthetic playback audio 154 of "It's sunny today" from the speaker 118 of the device 110. Here, the TTS model 200 of the TTS system 150 is configured to control speech-related attributes of the synthetic speech 154. In other words, the TTS model 200 is configured to naturally simulate the voice of a human speaker while simultaneously generating diverse synthetic speech by modeling fine-grained latent features. Although FIG. 1 shows an example of a TTS system 150 in the context of an assistant application, the TTS system 150 (e.g., using the TTS model 200) is also applicable to other text-to-speech scenarios, such as, for example, navigation or document reading.

[0035] 2A-2C, the TTS model 200 includes an enhanced encoder 202, an adapter 204, and a decoder 206. The enhanced encoder 202 receives the text 152 as input, which is converted into an input embedding 210 for the converter 220 to encode into a context vector Vc. Generally speaking, the encoder-decoder architecture uses the adapter 204 because the input and output sequences of the decoder 206 are of different lengths. Thus, the adapter 204 serves as a mechanism to handle how the input sequence corresponds to the output sequence. For example, the number of audio frames that correspond to the tokens of the context vector Vc. As shown in FIG. 2A, there are two options for the adapter 204, 204a-b. In the first option, the first adapter 204a uses attention (commonly referred to as an attention-based TTS model) to provide the context vector Vc in a compatible format for the decoder 206 to decode into the synthetic speech 154 (e.g., spectrogram or waveform). In a second option, the second adapter 204b performs duration-based upsampling (referred to as a duration-based TTS model) instead of focusing on the context vector Vc to provide the context vector Vc in a format compatible for the decoder 206 to decode into the synthetic speech 154.

[0036] To form the input embeddings 210, the TTS model 200 works similarly to the BERT model described above in that the input embeddings 210 are a combination of embeddings E that represent the input text 152. In some implementations, the input embeddings 210 correspond to a sequence of words (e.g., a sentence or sentences) represented as a plurality of tokens 212, 212a-n or a sequence of tokens 212. The plurality of tokens 212 includes a first set 212, 212P of phoneme tokens that represent the text input 152. 1~n and a second set 212G, 212G of grapheme tokens also representing the text input 152. 1~nThat is, both the first and second sets of tokens 212 represent the same text input 152. Thus, if the input text 152 is the sentence "my dog ​​is cute", the first set of phoneme tokens 212P represents the sentence "my dog ​​is cute" as the phoneme tokens 212P, and the second set of grapheme tokens 212G represents the same sentence "my dog ​​is cute" as the grapheme tokens 212G. Here, the phoneme tokens 212P may refer to International Phonetic Alphabet (IPA) phonemes, and the grapheme tokens 212G may correspond to sub-word units. Similar to the original BERT model, the plurality of tokens 212 may also include special tokens, denoted as CLS tokens 212, 212CLS and SEP tokens 212, 212SEP. The CLS token 212CLS is a special token that may be added before the first segment (i.e., sentence) or leading segment of the token 212, and the SEP token 212SEP acts as a separator added to each segment to indicate where one segment ends and another segment begins. For example, if the input text 152 contains two sentences represented as two segments, the sequence of tokens 212 will include a SEP token 212SEP separating the two segments (e.g., as shown in Figures 2A-2C). In some configurations, all tokens representing the input text 152 share the same identifier (ID) space for purposes of embedding lookup and masked language modeling (MLM) classification.

[0037] In some examples, unlike the original BERT, which uses a combination of three types of embeddings (e.g., token embeddings, segment embeddings, and positional embeddings) to form the input embeddings, the extended encoder 202 forms the input embeddings 210 from four types of embeddings E. For example, the extended encoder 202 uses four types of embeddings E instead of three types of embeddings E to provide the extended encoder 202 with an alignment context to incorporate graphemes in addition to phonemes. With reference to FIG. 2B, the input embeddings 210 represent the sum of positional embeddings E,Ep, word position embeddings E,Ewp, segment embeddings E,Es, and token embeddings E,Et. Here, the word position embeddings Ewp differ from the positional embeddings Ep in that the positional embeddings are an index of the overall position of multiple tokens 212. Meanwhile, the word position embeddings Ewp represent the position at the sub-word level (i.e., where the token occurs within the word of the segment). By embedding word positions in Ewp, the input embedding 210 can represent, for example, that a second phoneme in a sentence corresponds to a first word in a second word fragment (i.e., word position) of a first word, and also that a second grapheme in the same sentence corresponds to a second word in the sentence. That is, the alignment between phonemes and graphemes is represented in the input embedding 210.

[0038] FIG. 2B shows vertical dashed boxes around each token 212 in the input embedding 210 (i.e., the tokenization steps of the input embedding 210) to indicate that each token 212 in the input embedding 210 is a combination of a position embedding E,Ep, a word position embedding E,Ewp, a segment embedding E,Es, and a token embedding E,Et. Here, the position embedding Ep refers to the overall or global position of the token step relative to the input embedding 210 (e.g., the entire current input embedding 210). In FIG. 2B, there are nine positions or tokenization steps starting from index 0. The word position embedding Ewp indicates which word the token 212 corresponds to in a sequence of words from the input text 152. FIG. 2B has an input text 152 consisting of three words. The segment embedded Es identifies which phrase or sentence the token 212 corresponds to. In an extended encoder 202 having inputs corresponding to graphemes and phonemes, the input text 152 includes at least two segments for each single segment of the input text 152. In other words, each segment of the input text 152 is repeated twice, once as a phoneme segment (shown as segment A in FIG. 2B) and once as a grapheme segment (shown as segment B in FIG. 2B). Thus, if the input text 152 is two or four sentences, which means two or four segments in the case of conventional BERT, the input embeddings 210 of the extended encoder 202 will be four or eight segments, respectively (i.e., each sentence is represented once as a grapheme and once as a phoneme). Furthermore, the token embeddings Et forming the input embeddings 210 correspond to subunit representations of words of a segment (or sentence). As mentioned above, the phoneme tokens can be individual IPA phonemes, and the grapheme tokens are subwords of a word piece.

[0039] 2B, each token 212 of the input embedding 210 is shown within a vertical dashed box to indicate that each token 212 is a combination of multiple embeddings E. A first token 212a of the input embedding 210 is a combination of a first token embedding Et, Et1, a first segment embedding Es, EsA, a first word position embedding Ewp, Ewp0, and a first position embedding Ep, Ep0 of a CLS token. A second token 212b corresponding to a first phoneme token 212P1 in the set of phoneme tokens 212P of the input embedding 210 is a combination of a second token embedding Et, Et2, a first segment embedding Es, EsA, a second word position embedding Ewp, Ewp1, and a second position embedding Ep, Ep1 of a phoneme token P1. The third token 212c, which corresponds to the second phoneme token 212P2 in the set of phoneme tokens 212P of the input embedding 210, is a combination of the third token embedding Et, Et3, the first segment embedding Es, EsA, the second word position embedding Ewp, Ewp1, and the third position embedding Ep, Ep2 of the phoneme token P2. The fourth token 212d, which corresponds to the nth phoneme token 212Pn in the set of phoneme tokens 212P of the input embedding 210, is a combination of the nth token embedding Et, Et4, the first segment embedding Es, EsA, the third word position embedding Ewp, Ewp2, and the fourth position embedding Ep, Ep3 of the phoneme token Pn. The fifth token 212e, which corresponds to the SEP token 212SEP of the input embedding 210, is a combination of the fifth token embeddings Et, Et5, the first segment embeddings Es, EsA, the third word position embeddings Ewp, Ewp3, and the fifth position embeddings Ep, Ep4 of the SEP token SEP. The sixth token 212f, which corresponds to the first grapheme token 212G1 of the second set of grapheme tokens 212G of the input embedding 210, is a combination of the sixth token embeddings Et, Et6, the second segment embeddings Es, EsB, the first word position embeddings Ewp, Ewp1, and the sixth word position embeddings Ep, Ep4 of the grapheme token G1.The seventh token 212g, which corresponds to the second grapheme token 212G2 of the second set of grapheme tokens 212G of the input embeddings 210, is a combination of the seventh token embeddings Et, Et7, the second segment embeddings Es, EsB, the second word position embeddings Ewp, Ewp2, and the seventh position embeddings Ep, Ep7 of grapheme token G2. The eighth token 212h, which corresponds to the nth grapheme token 212Gn of the second set of grapheme tokens 212G of the input embeddings 210, is a combination of the eighth token embeddings Et, Et7, the second segment embeddings Es, EsB, the second word position embeddings Ewp, Ewp2, and the eighth word position embeddings Ep, Ep8 of grapheme token Gn. The ninth token 212i, which corresponds to the second SEP token 212SEP (e.g., indicating the end of the second segment) in the input embedding 210, is a combination of the ninth token embeddings Et, Et9, the second segment embeddings Es, EsB, the third word position embeddings Ewp, Ewp3, and the ninth position embeddings Ep, Ep9 of the CEP token. Here, because segments A and B are the same sentence, the embedding E representing the first word position (e.g., first word position embedding Ewp1) of the phoneme token 212P in the input embedding 210 is the same as the embedding E representing the first word position (e.g., first word position embedding Ewp1) of the grapheme token 212G, indicating that the grapheme and phoneme tokens 212 occur at the same word fragment level.

[0040] In some examples, the converter 220 of the extended encoder 202 receives the input encoder embedding 210 and generates a context vector Vc (also referred to as an output encoder embedding (Vc)) as an output of the extended encoder 202. Similar to the input encoder embedding 210, the output encoder embedding or context vector Vc may also be a sequence of output tokens (e.g., shown as V1-V9) based on the input tokens 212. Referring to FIG. 2C, in some implementations, the converter 220 is configured to receive the input encoder embedding 210 and generate the output encoder embedding Vc based on the relationship between the phoneme tokens 212P and the grapheme tokens 212G of the input encoder embedding 210. For example, for each phoneme token 212P of the set of phoneme tokens 212P from the input encoder embedding 210, the converter 220 identifies a respective word of the text input 152 that corresponds to the respective phoneme token 212P. The converter 220 may identify a respective word that corresponds to the respective phoneme token 212P based on the word position embedding Ewp for the respective phoneme. For example, for the first phoneme 212p of the input encoder embedding 210, the word position embedding Ewp is an embedding that identifies the first phoneme token 212P as being in the first word of segment A (i.e., the phoneme segment). Once the converter 220 identifies each word corresponding to each phoneme token 212P, the converter 220 determines which grapheme token 212G also belongs to that respective word. In this example, the first phoneme token 212P corresponds to the first word of the input text 152, and the first grapheme token 212G also corresponds to the first word of the input text 151 (e.g., shown as the first word (WP=1) in segment B (S=B)). As with the phoneme token 212P, the converter 220 may use the word position embedding E, Ewp for the set of grapheme tokens 212G to determine that the grapheme token 212G corresponds to the word position of the first phoneme token 212P. For example, the word position embedding E, Ewp1 corresponding to the first phoneme token 212P1 matches or is the same as the word position embedding E, Ewp1 corresponding to the first grapheme token 212G1.By identifying this relationship between the phoneme tokens 212P and the grapheme tokens 212G, the transducer 220 may represent this relationship as part of the context vector V. The transducer 220 may then repeat this process for each input token 212.

[0041] In some configurations, the converter 220 generates each token V in the context vector V by representing a particular input token 212 as a relationship to all other input tokens 212. For example, FIG. 2C shows that for a current token step of an input token 212P1, an output token V2 of the converter 220 describes the relationship between the first phoneme token and all other input tokens 212. In some implementations, the converter 220 may be configured to weight or qualitatively represent the strength of the current token step (e.g., the first phoneme token 212P) relative to each of the other input tokens 212 such that each of the other input tokens 212 has some influence on the output token V2. Here, because the first phoneme token 212P1 has a strong relationship (e.g., a shared embedding E at the word position level) with the first grapheme token 212G1, the converter 220 may generate an output token V2 that has some representation of this relationship.

[0042] Generating each token V in the context vector Vc as a representation of the relationship of a particular input token to all other input tokens 212 can be beneficial for different types of TTS models. That is, there are several TTS situations that have historically required specialized TTS models, or TTS models with specific architectures, to accommodate specific TTS tasks. This is typically the case for multilingual or multi-accent TTS models. A multilingual TTS model refers to a TTS model that can accommodate input text 152 from various languages ​​and thus generate synthetic speech 154 in these various languages. This is in contrast to a monolingual (or single language) TTS model. Traditionally, these TTS models have employed phoneme-based TTS models, which can lead to problems. Phonemes pose a challenge for multilingual models, as phonemes can overlap significantly between different languages. This overlap, which leads to language ambiguity, therefore requires traditional multilingual models to incorporate additional language inputs to make the model effective. However, using additional language inputs can lead to some problems when code-switching occurs. Code switching refers to the case where a sentence contains multiple languages. In a code switching situation, the combination of language input and text input may not be able to identify the multiple languages ​​or which specific token corresponds to which language. In other words, it is difficult to obtain an accurate language identifier for each token in a sentence. However, the extended encoder 202 may solve these problems because each token V in the context vector V represents the relationship of the specific input token to all other input tokens 212. Thus, if one of the input tokens 212 of the input encoder embedding 210 contains a language identifier, that language information is conveyed to the respective token V in the output encoder embedding V. For example, FIG. 2C shows that one of the special tokens 212CLS contains a language identifier 214 that is then translated into a second token V2 in the context vector Vc.2C shows a language identifier 214 associated with the special token 212CLS, the input encoder embedding 210 may be configured such that other input tokens 212 include a language identifier 214. Additionally or alternatively, in the case of a multi-accent TTS model, the identifier 214 may instead be a locale or accent identifier 214.

[0043] Besides multilingual TTS models, the extended encoder 202 may be incorporated into multi-accent TTS models. Multi-accent TTS models have historically also had some challenges. For example, like multilingual TTS models, multi-accent TTS models tend to use phoneme-based TTS models. This may be attributed to the fact that the alignment problem between the phonemes and graphemes of the encoder input has not been solved. However, using the current techniques of the extended encoder 202, multi-accent TTS models can leverage the use of graphemes to identify the locale or accent of the input text 152. With these alignment improvements to represent the relationship between phonemes and graphemes, the extended encoder 202 may be used with little, if any, significant modification in various TTS models such as monolingual TTS models, monolocale TTS models, multilingual TTS models, multi-accent TTS models, attention-based TTS, and duration-based TTS models. Thus, the extended encoder 202 may replace encoders used in TTS systems without disrupting other architectures of these TTS systems.

[0044] Referring to FIG. 3, similar to conventional BERT, the extended encoder 202 undergoes a pre-training process 300. Only here, the pre-training process 300 of the extended encoder 202 uses both phonemes and graphemes, unlike conventional BERT, where the pre-training process uses only graphemes. During the pre-training process, the extended encoder 202 pre-trains on a plaintext corpus 302. In some implementations, when training on a plaintext corpus 302, the pre-training process 300 uses a grapheme-to-phoneme (G2P) conversion system to obtain phonemes and a subword text tokenizer to obtain graphemes. In some examples, the pre-training process 300 may vary depending on the type of TTS model 200 incorporating the extended encoder 202. For example, when pre-training the extended encoder 202 for a monolingual or single-locale TTS model, the pre-training process 300 uses only a single target 320, or a masked language modeling (MLM) target 320, 320a. Generally speaking, when training with the MLM target 320, 320a, a certain percentage of the input tokens 312, 312a-n are randomly masked to predict the masked input token 312M. This masked prediction allows for both right and left contexts (i.e., bidirectional context). Furthermore, when using the extended encoder 202, the input content 304 of each training example 310, which corresponds to a sequence of words (i.e., a sentence or sentences), is represented twice, once as a sequence of training phoneme tokens 312, 312b-d and once as a sequence of training grapheme tokens 312, 312f-h. Now, if the MLM target 320a randomly applies masking to these training tokens 312, the training tokens 312 that are masked in the training phoneme tokens 312b-d may have counterparts (i.e., unmasked) in the training grapheme tokens 312f-312h, biasing the pre-training token prediction process.For this reason, the pre-training process 300 with MLM goal 320a instead applies random masking at the word level, but makes sure that it is consistent across phoneme and grapheme segments. That is, when a phoneme of a word is masked, the corresponding grapheme is also masked. For example, Figure 3 shows two masked phonemes 312, 312M for the first word of segment A. p1,2 and one masked grapheme 312, 312M for the corresponding first word (i.e., the complementary grapheme) of segment B. G Alternatively, the MLM goal 320a may be implemented using other masking strategies during the pre-training process 300. For example, the original BERT masking may be applied, but the masking rate is increased, or the pre-training process 300 applies masking in a P2G and G2P-like manner, where all tokens in one segment (e.g., grapheme segment—segment B) are masked out and all tokens in the other segment (e.g., phoneme segment—segment A) are kept.

[0045] For other TTS models 200, such as multilingual and multi-accent TTS models, the pre-training process 300 further uses classification targets 320, 320b. Here, the pre-training process 300 using the classification target 320b trains the output of the extension encoder 202 to predict a locale or language. For example, as described above, a special token, such as a CLS token, may carry a language or locale identifier 214. Here, the extension encoder 202 learns during pre-training to predict a language or locale and indicate the language or locale as an identifier associated with the token, such as a CLS token. In some examples, language classification may be easier than accent classification. In these examples, the classification loss of the classification target 320b may use a lower weight (e.g., 0.1).

[0046] The pre-training process 300 pre-trains the enhanced encoder 202 of the TTS model 200 to learn how to predict the input encoder embedding 210, and then the fine-tuning process trains the enhanced encoder 202 incorporated into the complete TTS model 200 for a specific task. The fine-tuning process initializes weights from the pre-trained model, which are then further adjusted during the training of the TTS model. In some examples, the fine-tuning freezes the embedding and lower layers of the converter 220 while fine-tuning the upper layers of the converter 220 to prevent performance degradation and promote generalization of the trained TTS model 200. Otherwise, performance degradation may occur due to the smaller training dataset used in the fine-tuning process. The fine-tuning process may pass only the hidden states from the final conversion layer on the downstream phoneme tokens 212P to the TTS components (e.g., the adapter 204 and the decoder 206). Although these hidden states only refer to the position of the phonemes, these hidden states still carry information from the graphemes as well as the language (or locale) based on how the enhanced encoder 202 was trained. In some configurations, the fine-tuning process turns off the MLM goal 320a but leaves the classification goal 320b active for a multilingual or multi-accent TTS model. For example, by leaving the classification goal 320b active during the fine-tuning process, linguistic information may be maintained in the encoded representation.

[0047] 4 is a flow chart of an exemplary configuration of operations of a method 400 for generating an output encoder embedding Vc using both phonemes and graphemes. In operation 402, the method 400 receives a text input 152 including a sequence of words represented as an input encoder embedding 210 in an encoder 202 of a speech synthesis model 200. The input encoder embedding 210 includes a plurality of tokens 212 including a first set of grapheme tokens 212, 212G that represent the text input 152 as respective graphemes and a second set of phoneme tokens 212, 212P that represent the text input 152 as respective phonemes. The method 400 performs operations 404 and 406 for each phoneme token 212P of the second set of phoneme tokens 212P. In operation 404, the method 400 identifies each word of the sequence of words that corresponds to each phoneme token 212P. At operation 406, the method 400 determines a respective grapheme token 212G that represents a respective word of the sequence of words that corresponds to each phoneme token 212P. At operation 408, the method 400 generates an output encoder embedding V based on the relationship between each phoneme token 212P and the corresponding grapheme token 212G that is determined to represent the same respective word as each phoneme token 212P.

[0048] 5 is a schematic diagram of an example computing device 500 that may be used to implement the systems (e.g., TTS system 150, TTS model 200, and / or enhanced encoder 202) and methods (e.g., method 400) described herein. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are for illustrative purposes only and are not intended to limit the implementation of the invention(s) described and / or claimed herein.

[0049] Computing device 500 includes a processor 510 (e.g., data processing hardware), a memory 520 (e.g., memory hardware), a storage device 530, a high-speed interface / controller 540 that connects to memory 520, and a high-speed expansion port 550, and a low-speed bus 570 and a low-speed interface / controller 560 that connects to storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 may be interconnected using various buses and mounted on a common motherboard or otherwise as desired. Processor 510 can process instructions for execution within computing device 500, including instructions stored in memory 520 and storage device 530 to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 coupled to high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used as desired, along with multiple memories and types of memories. Also, multiple computing devices 500 may be connected, each providing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0050] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) used by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disks or tapes.

[0051] The storage device 530 can provide mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. In additional implementations, the computer program product is embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.

[0052] The high-speed controller 540 manages the bandwidth-intensive operations of the computing device 500, while the low-speed controller 560 manages the less bandwidth-intensive operations. Such assignment of duties is merely an example. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., through a graphics processor or accelerator), and to a high-speed expansion port 550 that may accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is connected to a storage device 530 and a low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be coupled, for example, through a network adapter, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router.

[0053] The computing device 500 may be implemented in many different forms, as shown in the drawing. For example, the computing device 500 may be implemented as a standard server 500a, or multiple times within a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0054] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0055] These computer programs (also known as programs, software, software applications, or codes) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0056] The processes and logic flows described herein can be implemented by one or more programmable processors executing one or more computer programs to perform functions by operating on input data to generate output. The processes and logic flows can also be implemented by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, and any one or more processors of any type of digital computer. In general, a processor receives instructions and data from a read-only memory, a random access memory, or both. The essential elements of a computer are a processor for carrying out instructions and one or more memory devices for storing instructions and data. In general, a computer also includes one or more mass storage devices, e.g., magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to them, or both. However, a computer need not be equipped with such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks, e.g. internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0057] To provide for user interaction, one or more aspects of the present disclosure can be implemented on a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction, for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, tactile feedback, etc., and input from the user can be received in any form, including acoustic, speech, and tactile input. Additionally, the computer can interact with the user by sending and receiving documents to and from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0058] A number of implementations have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0059] 10 users 12 Voice utterances 14 Audio Data 100 Audio Environment 110 Voice-enabled devices 112 Data Processing Hardware 114 Memory Hardware 116 Audio Capture Device 118 Audio Output Devices 118 Speaker 120 Network 130 Remote Systems 132 Remote Resources 134 Remote Data Processing Hardware 136 Remote Memory Hardware 140 Voice Processing System 140 Voice Recognition System 142 Transcripts 150 Audio Processing System 150 TTS System 152 Text Input 152 Input Text 154 Synthetic playback audio 154 Synthetic Voice 154 Regenerative Signal 200 Audio Processing System 200 speech synthesis models 200 TTS model 202 Extended Encoder 204 Adapter 204a First adapter 204b Second Adapter 206 Decoder 210 Input padding 210 Input Encoder Embedding 212 Grapheme Tokens 212 Phoneme Tokens 212 CLS Tokens 212 SEP tokens 212, 212P 1~n The first set of phoneme tokens 212a~n Token 212a First Token 212b Second Token 212G Grapheme Token 212G, 212G 1~n A second set of grapheme tokens 212G1 First grapheme token 212P Phoneme Token 212P1 First phoneme token 212P1 Input Token 212P2 Second phoneme token 212c Third Token 212d Fourth Token 212e Fifth Token 212f 6th Token 212g 7th Token 212h 8th Token 212i 9th Token 212p First phoneme 214 Language Identifier 214 Language or Locale Identifier 220 Converter 300 Pre-training Process 302 Plaintext Corpus 304 Input Content 310 Training Examples 312, 312a-n Input token 312, 312b-d Training phoneme tokens 312, 312f~h Training grapheme tokens 312M Masked Input Token 320 goals 320, 320a Masked Language Modeling (MLM) Objectives 320, 320b classification objective 400 ways 402 operation 404 method 406 operation 500 computing devices 500a Standard Server 500b Laptop Computer 500c Rack Server System 510 Processor 520 Memory 530 Storage Devices 540 High Speed ​​Interface / Controller 550 High Speed ​​Expansion Port 560 Low Speed ​​Interface / Controller 570 Slow Bus 580 Display 590 Low Speed ​​Expansion Port

Claims

1. When executed by the data processing hardware (112), the data processing hardware (112) receiving, in an encoder (202) of a speech synthesis model (150), a text input (152) comprising a sequence of words represented as an input encoder embedding (210), the input encoder embedding (210) comprising a plurality of tokens (212), the plurality of tokens (212) comprising a first set of grapheme token (212G) embeddings representing the text input (152) as respective graphemes and a second set of phoneme tokens (212P) representing the text input (152) as respective phonemes; For each phoneme token (212P) of said second set of phoneme tokens (212P): identifying, by said encoder (202), each word of said sequence of words corresponding to said respective phoneme token (212P); determining, by the encoder (202), an embedding of each grapheme token (212G) representing each word of the sequence of words corresponding to each phoneme token (212P); generating an output encoder embedding (Vc) based on a relationship between each phoneme token (212P) and the corresponding grapheme token (212G) embedding determined by the encoder (202) to represent the same respective word as each phoneme token (212P); A computer-implemented method (400) performing operations comprising:

2. identifying, by the encoder (202), each word of the sequence of words corresponding to each phoneme token (212P) comprises identifying a position within each word that corresponds to each phoneme token (212P); 2. The method of claim 1, wherein the step of determining, by the encoder, an embedding of each grapheme token representing each word of the sequence of words corresponding to each phoneme token comprises determining, at the position within the respective word corresponding to the respective phoneme token, an embedding of each grapheme token representing each word of the sequence of words corresponding to the respective phoneme token.

3. Each token of the plurality of tokens (212) of the input encoder embedding (210) Either grapheme token embedding (212G) or phoneme token embedding (212P), Segment embedding, Word position embeddings, and Positional embedding The method (400) of claim 1 or 2, wherein the combination

4. 4. The method of claim 3, wherein identifying the respective words of the sequence of words corresponding to the respective phoneme tokens comprises identifying the respective words of the sequence of words corresponding to the respective phoneme tokens based on a respective word position embedding associated with the respective phoneme tokens.

5. 5. The method of claim 4, wherein determining a respective grapheme token (212G) embedding representing the respective word of a sequence of words corresponding to the respective phoneme token (212P) comprises determining that the respective grapheme token (212G) embedding comprises a corresponding word position embedding that matches the respective word position embedding of the respective phoneme token (212P).

6. The method (400) of any one of claims 1 to 5, wherein the speech synthesis model (150) comprises an attention mechanism in communication with the encoder (202).

7. The method (400) of any one of claims 1 to 6, wherein the speech synthesis model (150) comprises a duration-based upsampler in communication with the encoder (202).

8. 8. The method (400) of any one of claims 1 to 7, wherein the plurality of tokens (212) of the input encoder embedding (210) comprises a special token that identifies a language of the text input.

9. The operation, providing a plurality of training examples (310) to the encoder (202), each training example (310) represented as a sequence of training grapheme token (212G) embeddings corresponding to a training sequence of words, and a sequence of training phoneme tokens (212P) corresponding to the same training sequence of words; masking training phoneme tokens (212P) from said sequence of training phoneme tokens (212P) for each word from said training sequence of words; masking training grapheme token (212G) embeddings from said sequence of training phoneme tokens (212P) for each of said words from said training sequence of words; The method (400) of any one of claims 1 to 8, further comprising the step of pre-training the encoder (202) of the speech synthesis model (150) by:

10. the speech synthesis model (150) comprises a multilingual speech synthesis model (150); 10. The method (400) of any one of claims 1 to 9, wherein the operations further comprise pre-training the encoder (202) of the speech synthesis model (150) using a classification target (320) to predict a classification token for the plurality of tokens (212) of the input encoder embedding (210), the classification token comprising a language identifier (214).

11. the speech synthesis model (150) comprises a multilingual speech synthesis model (150); 11. The method (400) of any one of claims 1 to 10, wherein the output encoder embedding (Vc) comprises a sequence of encoder tokens, each encoder token comprising linguistic information regarding the textual input.

12. the speech synthesis model (150) comprises a multi-accent speech synthesis model (150); 12. The method (400) of any one of claims 1 to 11, wherein the operations further comprise pre-training the encoder (202) of the speech synthesis model (150) using a classification target (320) to predict a classification token for the plurality of tokens (212) of the input encoder embedding (210), the classification token comprising an accent identifier (214).

13. Data processing hardware (112); memory hardware (114) in communication with said data processing hardware (112); the memory hardware (114), when executed on the data processing hardware (112), causes the data processing hardware (112) to receiving, in an encoder (202) of a speech synthesis model (150), a text input (152) comprising a sequence of words represented as an input encoder embedding (210), the input encoder embedding (210) comprising a plurality of tokens (212), the plurality of tokens (212) comprising a first set of grapheme token (212G) embeddings representing the text input (152) as respective graphemes and a second set of phoneme tokens (212P) representing the text input (152) as respective phonemes; For each phoneme token (212P) of said second set of phoneme tokens (212P): identifying, by said encoder (202), each word of said sequence of words corresponding to said respective phoneme token (212P); determining, by the encoder (202), an embedding of each grapheme token (212G) representing each of the words of the sequence of words corresponding to each of the phoneme tokens (212P); generating an output encoder embedding (Vc) based on a relationship between each phoneme token (212P) and the corresponding grapheme token (212G) embedding determined by the encoder (202) to represent the same respective word as each phoneme token (212P); A system (200) storing instructions for performing operations comprising:

14. identifying, by the encoder (202), the respective words of the sequence of words corresponding to the respective phoneme tokens (212P) comprises identifying a position within the respective word corresponding to the respective phoneme tokens (212P); 14. The system of claim 13, wherein determining, by the encoder, an embedding of each grapheme token representing each word of the sequence of words corresponding to each phoneme token comprises determining, at the position within the respective word corresponding to the respective phoneme token, an embedding of each grapheme token representing each word of the sequence of words corresponding to the respective phoneme token.

15. Each token of the plurality of tokens (212) of the input encoder embedding (210) Either grapheme token embedding (212G) or phoneme token embedding (212P), Segment embedding, Word position embeddings, and Positional embedding 15. The system (200) of claim 13 or 14, wherein the combination

16. 16. The system (200) of claim 15, wherein identifying the respective words of the sequence of words corresponding to the respective phoneme tokens (212P) comprises identifying the respective words of the sequence of words corresponding to the respective phoneme tokens (212P) based on a respective word position embedding associated with the respective phoneme tokens (212P).

17. 17. The system (200) of claim 16, wherein determining the respective grapheme token (212G) embedding representing the respective word of a sequence of words corresponding to the respective phoneme token (212P) comprises determining that the respective grapheme token (212G) embedding has a corresponding word position embedding that matches the respective word position embedding of the respective phoneme token (212P).

18. 18. The system (200) of any one of claims 13 to 17, wherein the speech synthesis model (150) comprises an attention mechanism in communication with the encoder (202).

19. 19. The system (200) of any one of claims 13 to 18, wherein the speech synthesis model (150) comprises a duration-based upsampler in communication with the encoder (202).

20. 20. The system (200) of any one of claims 13 to 19, wherein the plurality of tokens (212) of the input encoder embedding (210) comprises a special token that identifies a language of the text input.

21. The operation, providing a plurality of training examples (310) to the encoder (202), each training example (310) being represented as a sequence of training grapheme token (212G) embeddings corresponding to a training sequence of words, and a sequence of training phoneme tokens (212P) corresponding to the same training sequence of words; masking training phoneme tokens (212P) from said sequence of training phoneme tokens (212P) for each word from said training sequence of words; masking training grapheme token (212G) embeddings from said sequence of training phoneme tokens (212P) for each of said words from said training sequence of words; 21. The system (200) of claim 13, further comprising pre-training the encoder (202) of the speech synthesis model (150) by:

22. the speech synthesis model (150) comprises a multilingual speech synthesis model (150); 22. The system (200) of any one of claims 13 to 21, wherein the operations further comprise pre-training the encoder (202) of the speech synthesis model (150) using a classification target (320) to predict a classification token for the plurality of tokens (212) of the input encoder embedding (210), the classification token comprising a language identifier (214).

23. the speech synthesis model (150) comprises a multilingual speech synthesis model (150); 23. The system (200) of any one of claims 13 to 22, wherein the output encoder embedding (Vc) comprises a sequence of encoder tokens, each encoder token comprising linguistic information related to the textual input.

24. the speech synthesis model (150) comprises a multi-accent speech synthesis model (150); 24. The system (200) of any one of claims 13 to 23, wherein the operations further comprise pre-training the encoder (202) of the speech synthesis model (150) using a classification target (320) to predict a classification token for the plurality of tokens (212) of the input encoder embedding (210), the classification token comprising an accent identifier (214).

Images