Training a speech recognition model based on cross-lingual speech synthesis

Abstract

A method (800) for training a speech recognition model (200) includes obtaining a multilingual text-to-speech (TTS) model (310). The method also includes generating a native-synthetic speech representation (306) for an input text sequence (302) in a first language, the native-synthetic speech representation conditioned on speaker characteristics (304) of a native speaker of the first language. The method also includes generating a cross-lingual-synthetic speech representation for the input text sequence in the first language, the cross-lingual-synthetic speech representation conditioned on speaker characteristics of a native speaker of a different, second language. The method also includes generating first and second speech recognition results (312) for the native-synthetic speech representation and the cross-lingual-synthetic speech representation. The method also includes determining a consistency loss term (352) based on the first and second speech recognition results, and updating parameters of the speech recognition model based on the consistency loss term.

Description

Technical Field

[0001] This disclosure relates to using speech recognition to improve cross-language speech synthesis. Background Technology

[0002] Automatic speech recognition (ASR) attempts to provide accurate transcription of spoken content by taking audio input and transcribing it into text. Languages ​​that are rarely used today or have limited spoken and textual resources pose a challenge to training ASR systems because only a limited amount of labeled training data exists. Using self-supervised training of ASR models can reduce the amount of labeled training data required to train an ASR model. Typically, even if an ASR model has sufficient labeled training data, each language requires a unique ASR model. Storing separate ASR models for each language requires a large amount of memory. Summary of the Invention

[0003] One aspect of this disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for training a speech recognition model. These operations include obtaining a multilingual text-to-speech (TTS) model. These operations also include generating a native-language synthesized speech representation for an input text sequence in a first language using the multilingual TTS model, the native-language synthesized speech representation being conditioned on speaker characteristics of a native speaker of the first language. These operations further include generating a first speech recognition result of the native-language synthesized speech representation and a second speech recognition result of the cross-lingual synthesized speech representation using the speech recognition model. These operations also include determining a consistency loss term based on the first speech recognition result and the second speech result, and updating the parameters of the speech recognition model based on the consistency loss term.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, these operations further include: generating a first cross-entropy loss term based on a first language recognition result and an input text sequence in the first language; determining a second cross-entropy loss term based on a second speech recognition result and an input text sequence in the first language; and updating the parameters of the speech recognition model based on the first and second cross-entropy loss terms. In some examples, the parameters of the speech recognition model are updated based on a consistency loss term, independent of the first and second cross-entropy loss terms. These operations may further include backpropagating the first and second cross-entropy loss terms through a multilingual TTS model. Optionally, these operations may also include applying data augmentation to at least one of the native language synthesized speech representation or the cross-language synthesized speech representation.

[0005] In some implementations, the multilingual TTS model includes an encoder part and a decoder part. The encoder part shares language embeddings across the first and second languages, and the decoder part shares language embeddings across both the first and second languages, as well as speaker embeddings from both the native speaker of the first language and the native speaker of the second language. In these implementations, the number of speaker embeddings from the native speaker of the first language may be less than the number of speaker embeddings from the native speaker of the second language. The decoder part may further be conditioned on prosodic information extracted from the synthesized speech representation using a variational autoencoder. Here, prosodic information extracted from the synthesized speech representation using a variational autoencoder is separated from speaker information by applying an adversarial loss to speaker classification.

[0006] In some examples, prior to generating native and cross-lingual synthesized speech representations, these operations further include: transcribing the first-language input text sequence into a native-language script; lexicalizing the native-language script into a phoneme sequence; encoding the phoneme sequence using an encoder of a multilingual TTS model; and decoding the encoded phoneme sequence using a decoder of a multilingual TTS model to generate either a native-language synthesized speech representation or a cross-lingual synthesized speech representation. In some implementations, these operations further include: generating a native-language audio encoder embedding for the native-language synthesized speech representation using a variational autoencoder; generating a cross-lingual audio encoder embedding for the cross-lingual synthesized speech representation using a variational autoencoder; determining an adversarial loss term conditioned on the first language based on the native-language and cross-lingual audio encoder embeddings; and updating the parameters of the multilingual TTS model based on the adversarial loss term.

[0007] Another aspect of this disclosure provides a system for training a speech recognition model, the system including data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. These operations include obtaining a multilingual text-to-speech (TTS) model. These operations also include generating a native-language synthesized speech representation for an input text sequence in a first language using the multilingual TTS model, the native-language synthesized speech representation being conditioned on speaker characteristics of a native speaker of the first language. These operations further include generating a first speech recognition result of the native-language synthesized speech representation and a second speech recognition result of the cross-lingual synthesized speech representation using the speech recognition model. These operations also include: determining a consistency loss term based on the first and second speech recognition results; and updating the parameters of the speech recognition model based on the consistency loss term.

[0008] Implementations of this disclosure may include one or more of the following optional features. In some implementations, these operations further include: generating a first cross-entropy loss term based on a first language recognition result and an input text sequence in the first language; determining a second cross-entropy loss term based on a second speech recognition result and an input text sequence in the first language; and updating the parameters of the speech recognition model based on the first and second cross-entropy loss terms. In some examples, the parameters of the speech recognition model are updated based on a consistency loss term, independent of the first and second cross-entropy loss terms. These operations may further include backpropagating the first and second cross-entropy loss terms through a multilingual TTS model. Optionally, these operations may also include applying data augmentation to at least one of the native language synthesized speech representation or the cross-language synthesized speech representation.

[0009] In some implementations, the multilingual TTS model includes an encoder part and a decoder part. The encoder part shares language embeddings across the first and second languages, and the decoder part shares both the shared language embeddings and the speaker embeddings of both the native speaker of the first language and the native speaker of the second language. In these implementations, the number of speaker embeddings of the native speaker of the first language may be less than the number of speaker embeddings of the native speaker of the second language. The decoder part may further be conditioned on prosodic information extracted from the synthesized speech representation using a variational autoencoder. Here, the prosodic information extracted from the synthesized speech representation using a variational autoencoder is separated from the speaker information by applying an adversarial loss to speaker classification.

[0010] In some examples, prior to generating native and cross-lingual synthesized speech representations, these operations further include: transcribing the first-language input text sequence into a native-language script; lexicalizing the native-language script into a phoneme sequence; encoding the phoneme sequence using an encoder of a multilingual TTS model; and decoding the encoded phoneme sequence using a decoder of a multilingual TTS model to generate either a native-language synthesized speech representation or a cross-lingual synthesized speech representation. In some implementations, these operations further include: generating a native-language audio encoder embedding for the native-language synthesized speech representation using a variational autoencoder; generating a cross-lingual audio encoder embedding for the cross-lingual synthesized speech representation using a variational autoencoder; determining an adversarial loss term conditioned on the first language based on the native-language and cross-lingual audio encoder embeddings; and updating the parameters of the multilingual TTS model based on the adversarial loss term.

[0011] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description, the drawings, and the claims. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of an example speech recognition system that includes a speech recognition model.

[0013] Figure 2 This is a schematic diagram of the Recurrent Neural Network-Transformer (RNN-T) model architecture.

[0014] Figure 3 This is a schematic diagram of an example training process for training speech recognition models and / or multilingual text-to-speech models.

[0015] Figure 4 This is a schematic diagram of an example training process for training a multilingual text-to-speech model.

[0016] Figure 5 This is a schematic diagram illustrating the training of multiple speech recognition models using a multilingual text-to-speech model.

[0017] Figure 6 This is a schematic diagram of an example speech recognition system.

[0018] Figure 7 This is a flowchart illustrating an example of the operational setup for training an automatic speech recognition model.

[0019] Figure 8 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described in this paper.

[0020] The same reference numerals in the various figures indicate the same elements. Detailed Implementation

[0021] Training an automatic speech recognition (ASR) model requires a large amount of transcriptional data. That is, an ASR model needs training data pairs, which consist of audio data and corresponding audio transcriptions. The audio data and corresponding transcriptions (e.g., training data pairs) are used together to train the ASR model. The ASR model receives the audio data, predicts the transcription of the audio data, and compares the predicted transcription with the corresponding transcription (i.e., the ground truth label). However, the amount of training data pairs required to train an ASR model is difficult to collect. In some cases, user permission is required to access the training data pairs. In other cases, low-resource languages ​​include only a limited number of available speakers, resulting in little acoustic diversity to train the ASR model, which only makes a small improvement to the ASR model's performance. For example, for low-resource languages ​​such as Hindi (e.g., Kannada, Telugu, Tamil, and Bengali), only a limited number of speakers with formal, restricted speaking styles are available to train a Hindi ASR model. In contrast, for high-resource languages ​​such as English, there may be thousands of speakers with diverse speaking styles available to train an English ASR model.

[0022] The implementation methods described in this paper pertain to systems and methods for training ASR models. Specifically, for an input text sequence fed into a multilingual TTS model, the TTS model generates a native-language synthesized speech representation in the first language. This native-language synthesized speech representation is conditioned on the speaker characteristics of the native speaker of the first language. The TTS model also generates a cross-language synthesized speech representation in the first language for the same input text sequence. Here, the cross-language synthesized speech representation is conditioned on the speaker characteristics of the native speaker of the second language. That is, the cross-language synthesized speech representation is conditioned on the non-native speaker of the first language. The ASR model receives the native-language and cross-language synthesized representations and generates a first speech recognition result for the native-language synthesized speech representation and a second speech recognition result for the cross-language synthesized speech representation. A consistency loss term module determines a consistency loss term based on a comparison of the first and second speech recognition results, and the ASR model updates its parameters based on the consistency loss term.

[0023] The embodiments of this paper also relate to systems and methods for separating speaker embeddings, language embeddings, and prosodic embeddings from synthesized speech representations generated by a multilingual TTS model for training the model. A variational autoencoder (VAE) can receive native and cross-lingual synthesized speech representations and generate native-language audio encoder embeddings and cross-lingual audio encoder embeddings, respectively. A classifier can then determine an adversarial loss term conditioned on the first language based on the native-language and cross-lingual audio encoder embeddings. The multilingual TTS model can use the adversarial loss term to update parameters to prevent the generation of accented synthesized speech representations. That is, the adversarial loss term prevents the synthesized speech representation from being closely similar to the speaker embeddings, language embeddings, and / or prosodic embeddings of the speaker characteristics conditioned on the synthesized speech representation.

[0024] Figure 1 An Automatic Speech Recognition (ASR) system is illustrated, implementing an ASR model 200 on a user device 102 residing in user 104 and / or a remote computing device 201 (e.g., one or more servers in a distributed system running in a cloud computing environment) communicating with user device 102. While user device 102 is depicted as a mobile computing device (e.g., a smartphone), user device 102 can correspond to any type of computing device, such as, but not limited to, tablet devices, laptops / desktops, wearable devices, digital assistant devices, smart speakers / displays, smart appliances, automotive infotainment systems, or Internet of Things (IoT) devices.

[0025] User equipment 102 includes an audio subsystem 108 configured to receive spoken words 106 from user 104 (e.g., user equipment 102 may include one or more microphones for recording spoken words 106) and convert the spoken words 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by ASR system 100. In the example shown, user 104 speaks the corresponding spoken words 106 in natural language in English for the phrase “What is the weather in New York City?”, and audio subsystem 108 converts the spoken words 106 into the corresponding acoustic frame 110 for input to ASR system 100. ASR model 200 then receives the acoustic frame 110 corresponding to the spoken words 106 as input and generates / predicts a corresponding transcription (e.g., recognition result / hypothesis) 120 of the spoken words 106 as output. In the illustrated example, user device 102 and / or remote computing device 201 also execute user interface generator 107, which is configured to present a representation of the transcription 120 of utterance 106 to user 104 of user device 102. In some configurations, the transcription 120 output from ASR system 100 is processed, for example, by a natural language understanding (NLU) module executed on user device 102 or remote computing device 201 to execute user commands. Additionally or alternatively, a text-to-speech system (e.g., executed on any combination of user device 102 or remote computing device 201) can convert the transcription 120 into synthesized speech for audible output to another device. For example, the original utterance 106 may correspond to a message that user 104 is sending to a friend, where the transcription 120 is converted into synthesized speech for audible output to the friend to listen to the message conveyed in the original utterance 106.

[0026] refer to Figure 2 The ASR model 200 may include an end-to-end (E2E) sequence-to-sequence model. The E2E sequence-to-sequence model may include a recurrent neural network converter (RNN-T) model architecture that adheres to latency constraints associated with interactive applications. The RNN-T model 200 offers a small computational footprint and uses less memory than conventional ASR architectures, making it suitable for performing speech recognition entirely on user device 102 (e.g., without needing to communicate with a remote server). The RNN-T model 200 includes an encoder network 210, a prediction network 220, and a joint network 230. The encoder network 210, broadly similar to the acoustic model (AM) in a conventional ASR system, includes a recurrent network with stacked long short-term memory (LSTM) layers. For example, the encoder reads a series of d-dimensional feature vectors (e.g., acoustic frames 110...). Figure 1 x = (x1, x2, ..., x T ),in, And at each time step, a higher-order feature representation is generated. This higher-order feature representation is represented as...

[0027] Similarly, prediction network 220 is also an LSTM network, which, like a language model (LM), takes the non-blank symbol sequence y0,...,y0 output so far by the final Softmax layer 240. ui-1 Process into dense representation Finally, utilizing the RNN-T model architecture, the representations generated by the encoder and prediction networks 210 and 220 are combined by the joint network 230. The joint network then predicts... This is the distribution over the next output symbol. In other words, the joint network 230 generates a probability distribution over possible speech recognition hypotheses at each output step (e.g., time step). Here, “possible speech recognition hypotheses” correspond to the set of output labels, each output label representing a symbol / character of a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, such as one label for each of the 26 letters of the English alphabet and one label for a space. Thus, the joint network 230 can output a set of values ​​that indicate the probability of each output label appearing in the predetermined set of output labels. This set of values ​​may be a vector and may indicate a probability distribution over the set of output labels. In some cases, the output labels are morphemes (e.g., a single character, possibly along with punctuation marks and other symbols), but the set of output labels is not limited to this. For example, in place of or in place of morphemes, the set of output labels may include wordpieces and / or entire words. The output distribution of the joint network 230 may include the posterior probability value for each different output label. Thus, if there are 100 different output labels representing different morphemes or other symbols, the output y of the joint network 230 may be... i It can include 100 different probability values, one probability value for each output label. The probability distribution can then be used to select and assign scores to candidate spelling elements (e.g., graphemes, chunks, and / or words) during beam search (e.g., via a Softmax layer 240) to determine transcription 120.

[0028] The Softmax layer 240 can employ any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200 in the corresponding output step. In this way, the RNN-T model 200 does not make a conditional independence assumption; instead, the prediction of each symbol is conditional not only on the acoustics but also on the sequence of output labels so far. The RNN-T model 200 does assume that the output symbol is independent of future acoustic frames 110, which allows the RNN-T model 200 to be used in a streaming manner.

[0029] In some examples, the encoder network 210 of the RNN-T model 200 consists of eight 2,048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. The prediction network 220 can have two 2,048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. Finally, the joint network 230 can also have 640 hidden units. The softmax layer 240 can consist of a unified set of word pieces or characters generated using all unique word pieces or characters from multiple training text utterances.

[0030] Figure 3 The illustration shows an example training process 300 for training the ASR model 200 and / or the multilingual TTS model 310 (also referred to as the TTS model 310). The TTS model 310 is configured to generate a synthetic speech representation 306 for each of a plurality of unspoken training text utterances 302 (also referred to as input text sequences 302) at each of multiple time steps. The input text sequences 302 consist of unspoken text as plain text data, i.e., unpaired data, such that each input text sequence 302 is not paired with any synthetic speech non-synthetic speech representation, i.e., the input text sequence 302 is not paired with a corresponding utterance of human speech. Therefore, the TTS model 310 generates a corresponding synthetic speech representation 306 for each input text sequence 302. In other words, the TTS model 310 creates paired data through self-supervision by predicting synthetic speech representations 306 for unpaired input text sequences 302. It is worth noting that the synthesized speech representation 306 may include Mel spectrogram frames for training the ASR model 200, thereby eliminating the need for the TTS model 310 to include a vocoder and / or synthesizer to synthesize the Mel spectrogram frames into synthesized speech.

[0031] In some examples, the TTS model 310 generates multiple synthesized speech representations 306 for each input text sequence 302. Each synthesized speech representation 306 may be conditioned on speaker features 304 of native speakers of different languages. In some examples, the speaker features 304 of each native speaker of the corresponding language include speaker embeddings 305. Figure 6 Language Embedding 303 Figure 6 ) and / or local embeddings representing accent / dialect information 307 ( Figure 6 (See below for reference) Figure 6 A more detailed discussion is needed.

[0032] In the example shown, the TTS model 310 receives an input text sequence 302 in a first language. For example, the input text sequence 302 may represent input text in a low-resource language, Kannada (i.e., the first language). The TTS model 310 then generates native-language synthesized speech representations 306 and 306a for the first-language input text sequence 302, conditional on speaker characteristics 304 and 304a of the native speaker of the first language. The speaker characteristics 304a of the native speaker of the first language can be referred to as the first conditional input 304a. Continuing the example, the TTS model 310 generates a native-language synthesized speech representation 306a in Kannada, conditional on speaker characteristics 304a of the native speaker of Kannada. Therefore, the native-language synthesized speech representation 306a is conditional on speaker characteristics 304a of the native speaker of the corresponding language of the input text sequence 302.

[0033] In some implementations, the training process 300 generates a coarse-to-fine (C2F) loss 315 by comparing the native-speaker synthesized speech representation 306a with the ground truth audio of the native speaker of the corresponding input text sequence 302. Therefore, the C2F loss 315 represents the difference between the non-synthesized speech (e.g., ground truth audio) from the native speaker and the native-speaker synthesized speech representation 306a. The training process 300 feeds the C2F loss 315 to the TTS model 310 and updates the parameters of the TTS model 310 based on the C2F loss 315.

[0034] However, in some cases, the acoustic diversity of the synthesized speech representation 306a is limited because some low-resource languages ​​have only a limited number of first-language native speakers. That is, Hindi languages ​​such as Kannada may only have one or two Kannada native speakers, and the TTS model 310 can condition the synthesized speech representation 306 with these speaker characteristics (e.g., adding conditional input) 304. This limited acoustic diversity of the synthesized speech representation 306 provides only incremental improvements when training the ASR model. In contrast, high-resource languages ​​such as English have thousands of native speakers with diverse speaking styles, thus providing significant improvements to the ASR model during training.

[0035] In some implementations, the TTS model 310 generates cross-language synthesized speech representations 306 and 306b for the same input text sequence 302 in a first language, conditioned on speaker characteristics 304 and 304b of a native speaker of a second language different from the first language. That is, the TTS model 310 generates a cross-language synthesized speech representation 306b for the input text sequence 302 in the first language, which conveys speaker characteristics 304b of a native speaker of the second language. The speaker characteristics 304b of the native speaker of the second language can be referred to as a second conditional input 304a. For example, the TTS model 310 generates a cross-language synthesized speech representation 306b for the same input text sequence 302 in Kannada, conditioned on speaker characteristics 304b of a native English speaker. In other words, the cross-language synthesized speech representation 306b represents Kannada speech spoken by a native English speaker. Therefore, the cross-language synthesized speech representation 306b is conditional on the speaker characteristics 304b of a native speaker speaking a language different from the language of the input text sequence 302.

[0036] By conditioned the synthesized speech representations 306 on speaker characteristics of native speakers of different languages, the TTS model 310 can generate multiple synthesized speech representations 306 for each input text sequence 302 in the first language, thereby increasing the acoustic diversity among the synthesized speech representations 306. That is, the TTS model 310 can utilize one or more speakers from a second language (e.g., English) to synthesize an input text sequence 302 in a first language (e.g., Kannada), even if the second language speaker does not speak the first language. Therefore, the TTS model 310 can leverage high-resource languages ​​such as English to obtain speaker characteristics of native English speakers, increasing the acoustic diversity among the synthesized speech representations generated for low-resource languages ​​to train the ASR model. Furthermore, the TTS model 310 can generate synthesized speech representations 306 for unspoken input text sequences 302, thereby increasing the lexical diversity of the training data for the ASR model 200.

[0037] In some implementations, the training process 300 generates a spectrum. Figure 1 Consistency loss (not shown). That is, the training process can extract latent variables from the ground-value audio of the first-language native speaker to perform teacher coercion on the cross-lingual synthesized speech representation 306b conditioned on the speaker characteristics 304b of the second-language native speaker. Subsequently, for the corresponding input text sequence 302, the training process 300 calculates the mean squared error (MSE) between the cross-lingual synthesized speech representation 306b (e.g., based on teacher coercion) and the ground-value audio of the first-language native speaker to determine the spectrum. Figure 1 Consistent loss. Therefore, the spectrum. Figure 1 Consistency loss improves the consistency between the cross-lingual synthesized speech representation 306b and the real-value audio from the first-language native speaker. The spectrum can be... Figure 1 Consistency loss is provided as feedback to TTS model 310 to update the parameters of TTS model 310.

[0038] In the example shown, for clarity only, the TTS model 310 generates a synthesized speech representation 306 conditioned only on the speaker characteristics 304a, 304b of the two speakers 304a, 304b. That is, the TTS model 310 can generate any number of synthesized speech representations 306 conditioned on the speaker characteristics 304 of any number of speakers. For example, the TTS model 310 can generate a third synthesized speech representation 306 in Kannada, conditioned on the speaker characteristics of the native speaker of the third language (e.g., a native speaker of Spanish spoken in Spanish). Alternatively, the TTS model 310 can generate a fourth synthesized speech representation 306 in Kannada, conditioned on the speaker characteristics of the second native speaker of the first language.

[0039] In some implementations, the training process 300 includes a data augmentation module 360 ​​that applies data augmentation to at least one of a native-language synthesized speech representation 306a and / or a cross-lingual synthesized speech representation 306b. The data augmentation of the synthesized speech representation 306 is configured to promote acoustic diversity of the training samples used to train the ASR model 200. In some examples, the data augmentation module 360 ​​applies data augmentation techniques including at least one of adding / injecting noise, adding reverberation, or manipulating the temporal sequence of the synthesized speech representation 306. Another data augmentation technique includes injecting various ambient noises into the synthesized speech representation using multi-style training (MTR). A further data augmentation technique that the data augmentation module 360 ​​may use to complement or replace the MTR application includes using spectral augmentation (SpecAugment) to make the synthesized speech representation acoustically more similar. Combined, MTR and SpecAugment can inject noise into the synthesized speech representation 306, tile random external noise sources along time and insert and overlap the representation before it is trained, and filter the synthesized speech representation 306 with injected noise before training the ASR model 200.

[0040] Example training procedure 300 uses multiple synthesized speech representations 306 generated by multilingual TTS model 310 to train ASR model 200. In the example shown, ASR model 200 is trained to recognize speech spoken in a first language (e.g., Kannada). For each synthesized speech representation 306, ASR model 200 generates a corresponding speech recognition result 312. Speech recognition result 312 may represent a probability distribution over possible speech recognition hypotheses. ASR model 200 generates first speech recognition results 312, 312a of native language synthesized speech representation 306a and second speech recognition results 312, 312b of cross-lingual synthesized speech representation 306b. Continuing the example above, ASR model 200 receives a Mel spectrogram frame of native language synthesized speech representation 306a conditioned on a first conditional input 306a (e.g., speaker characteristics 306a of a native Kannada speaker) and generates a first speech recognition result 312a. The ASR model also receives a Mel spectrogram frame of a cross-lingual synthesized speech representation 306b conditioned on a second conditional input 304b (e.g., speaker characteristics 304b of a native English speaker) and generates a second speech recognition result 312b.

[0041] In some examples, the training process 300 determines a consistency loss term 352 based on the first and second speech recognition results 312a, 312b. For example, the training process 300 may employ a consistency loss term module 350, which is configured to receive the corresponding speech recognition results 312a, 312b output by the ASR model 200 at each of multiple time steps and determine a consistency loss term 352 between the corresponding speech recognition results 312a, 312b at each of the multiple time steps. The consistency loss term module 350 may be based on the Kullback-Leibler divergence (D) between a first probability distribution on a possible first synthesized speech result hypothesis and a second probability distribution on a possible second synthesized speech result hypothesis. KL To determine the consistent loss term 352.

[0042] The consistency loss term 352 provides an "unsupervised" loss term independent of the accuracy of the ASR model 200 and can be used to update the parameters of the ASR model 200 to promote consistency between the first speech recognition result 312a recognized from the native language synthesized speech representation 306a and the second speech recognition result 312b recognized from the cross-lingual synthesized speech representation 306b. Notably, the synthesized speech representations 306a and 306b are generated from the same input text sequence 302 used as the true values ​​for the ASR model 200. In other words, the consistency loss term 352 allows the ASR model 200 to learn to perform consistently, for example, to make consistent predictions for the native language synthesized speech representation 306a conditioned on the speaker characteristics 304a of the first language native speaker and the cross-lingual synthesized speech representation 306b conditioned on the speaker characteristics 304b of the second language native speaker for the same input text sequence 302. During training, the consistency loss module 350 can provide the consistency loss term 352 back to the ASR model 200 for updating the parameters of the ASR model 200 based on the consistency loss term 352.

[0043] In some implementations, the training process 300 executes a supervised loss term module 340, which is configured to receive a speech recognition result 312 as input and generate a supervised loss term 342 as output based on the input text sequence 302 used as the ground truth. In the illustrated example, the supervised loss term module 340 receives the input text sequence 302 (i.e., the ground truth transcription) and a first speech recognition result 312a and outputs a first supervised loss term 342, 342a (also referred to as a first cross-entropy loss term 342a). Therefore, the first supervised loss term 342a is based on a comparison between the first speech recognition result 312a and the corresponding input text sequence 302 (e.g., the target speech recognition result).

[0044] Furthermore, the supervised loss term module 340 receives the input text sequence 302 and the second speech recognition result 312b and outputs a second supervised loss term 342, 342b (also known as a second cross-entropy loss term 342b). The second supervised loss term 342b is based on a comparison between the second speech recognition result 312b and the corresponding input text sequence 302 (e.g., the target speech recognition result). Therefore, the second supervised loss term represents the accuracy of the second speech recognition result 312b based on the cross-language synthesized speech representation 306b.

[0045] The supervised loss module 340 can provide a first supervised loss term 342a and a second supervised loss term 342b back to the ASR model 200, and the ASR model 200 updates its parameters based on the first supervised loss term 342a and the second supervised loss term 342b. In some examples, the training process 300 updates the parameters of the ASR model 200 based on the consistent loss term 352, without relying on the first supervised loss term 342a and the second supervised loss term 342b. Optionally, the training process 300 can backpropagate the first supervised loss term 342a and the second supervised loss term 342b to the TTS model 310 to update the parameters of the TTS model 310. Here, the ASR model 200 is fixed, such that the parameters of the ASR model 200 are static (e.g., not updated), while the parameters of the TTS model 310 are updated based on the first and second supervised loss terms 342a, 342b.

[0046] Figure 4 An example training process 400 for training a multilingual TTS model 310 is illustrated. In some implementations, the TTS model 310 generates a synthesized speech representation 306 that closely corresponds to the speaker characteristics of a particular speaker. For example, a finite number of first-language native speakers 304a may cause the TTS model 310 to generate a native-language synthesized speech representation 306a that is very similar to the speaker characteristics of the finite number of speakers. Therefore, the ASR model 200 ( Figure 3 Training is performed only on the synthesized speech representations 306, which are analogous to the speaker characteristics of a finite number of speakers. Therefore, the training process 400 includes a hierarchical variational autoencoder (VAE) 410, which is configured to separate speaker, language, and / or prosodic information from the synthesized speech representations 306.

[0047] In some examples, the TTS model 310 generates multiple synthesized speech representations 306 for each input text sequence 302. Each synthesized speech representation 306 may be conditional on a conditional input 304 representing different speakers with different speaker features 304. In some examples, the speaker features include speaker features 304 representing the speaking style of a particular speaker. That is, the speaker features 304 of a particular speaker may include speaker embeddings 305. Figure 6 Language Embedding 303 Figure 6 ) and / or local embeddings representing accent / dialect information 307 ( Figure 6 (See reference) Figure 6 To be discussed in more detail.

[0048] In the example shown, the TTS model 310 receives an input text sequence 302 in a first language. For example, the input text sequence 302 may represent input text in a low-resource language, Kannada (i.e., the first language). The TTS model 310 then generates native-language synthesized speech representations 306 and 306a for the first-language input text sequence 302, conditioned on the speaker characteristics 304a of the native speaker of the first language 304 and 304a. Continuing the example, the TTS model 310 generates a native-language synthesized speech representation 306a in Kannada, conditioned on the speaker characteristics 304a of the native speaker of Kannada 304a. Therefore, the native-language synthesized speech representation 306a is conditioned on the speaker characteristics 304a of the native speaker of the language corresponding to the input text sequence 302.

[0049] In some implementations, as described above... Figure 3 The training process is as described in 300. Figure 4 The training process 400 generates a coarse-to-fine (C2F) loss 315 by comparing the native-language synthesized speech representation 306a with the ground truth audio from a first-language native speaker for the corresponding input text sequence 302. Therefore, the C2F loss 315 represents the difference between the non-synthesized speech (e.g., ground truth audio) from the native speaker and the native-language synthesized speech representation 306a. The training process 400 feeds the C2F loss 315 to the TTS model 310 and updates the parameters of the TTS model 310 based on the C2F loss 315.

[0050] In some implementations, the TTS model 310 also generates cross-language synthesized speech representations 306 and 306b of the same input text sequence 302 in the first language, conditioned on speaker characteristics 304 and 304b of a native speaker of a second language different from the first language. That is, the TTS model 310 generates a cross-language synthesized speech representation 306b for the input text sequence 302 as if the native speaker of the second language were speaking it in the first language. For example, the TTS model 310 generates a cross-language synthesized speech representation 306b for the same Kannada input text sequence 302, conditioned on speaker characteristics 304b of a native speaker of English. In other words, the cross-language synthesized speech representation 306b represents Kannada speech as if the native speaker of English were speaking it. Therefore, the cross-language synthesized speech representation 306b is conditioned on speaker characteristics 304b of a native speaker of a language different from the language of the input text sequence 302.

[0051] By conditioned the synthesized speech representation 306 on multiple speakers 304, the TTS model 310 can generate multiple synthesized speech representations 306 for each input text sequence 302 in the first language, thereby increasing the acoustic diversity of the synthesized speech representations 306. That is, the TTS model 310 can utilize one or more speakers from a second language (e.g., English) to synthesize an input text sequence 302 from a first language (e.g., Kannada), even if the second language speaker does not speak the first language. Therefore, the TTS model 310 can use a high-resource language such as English to generate synthesized speech representations for a low-resource language to train the ASR model. Furthermore, the TTS model 310 can generate synthesized speech representations 306 for unspoken input text sequences 302, thereby increasing the lexical diversity of the training data for the ASR model 200.

[0052] In some implementations, the training process 400 generates a spectrum. Figure 1 Consistency loss (not shown). That is, the training process can extract latent variables from the ground-value audio of the first-language native speaker to enforce teacher coercion on the second-language native speaker 304b. Subsequently, for the corresponding input text sequence 302, the training process 400 calculates the mean squared error (MSE) between the cross-language synthesized speech representation 306b (e.g., based on teacher coercion) and the ground-value audio of the first-language native speaker to determine the spectrum. Figure 1 Consistent loss. Therefore, the spectrum. Figure 1 Consistency loss improves the consistency between the cross-lingual synthesized speech representation 306b and the real-value audio from the first-language native speaker. The spectrum can be... Figure 1 Consistency loss is provided as feedback to TTS model 310 to update the parameters of TTS model 310.

[0053] In the example shown, for clarity only, the TTS model 310 generates a synthesized speech representation 306 conditioned on only two speakers 304a and 304b. That is, the TTS model 310 can generate any number of synthesized speech representations 306 conditioned on speaker characteristics 304 of any number of speakers 304. For example, the TTS model 310 can generate a third synthesized speech representation 306 in Kannada conditioned on speaker characteristics of a native speaker of a third language (e.g., a native speaker of Spanish speaking Spanish). Alternatively, the TTS model 310 can generate a fourth synthesized speech representation 306 in Kannada conditioned on speaker characteristics of a second native speaker of a first language.

[0054] In some implementations, the training process 400 includes a data augmentation module 360 ​​that applies data augmentation to at least one of a native-language synthesized speech representation 306a and / or a cross-lingual synthesized speech representation 306b. The data augmentation of the synthesized speech representation 306 is configured to promote acoustic diversity of the training samples used to train the ASR model 200. In some examples, the data augmentation module 360 ​​applies data augmentation techniques including at least one of adding / injecting noise, adding reverberation, or manipulating the temporal sequence of the synthesized speech representation 306. Another data augmentation technique includes injecting various ambient noises into the synthesized speech representation using multi-style training (MTR). A further data augmentation technique that the data augmentation module 360 ​​may use to complement or replace the MTR application includes using spectral augmentation (SpecAugment) to make the synthesized speech representation acoustically more similar. Combined, MTR and SpecAugment can inject noise into the synthesized speech representation 306, tile random external noise sources along time and insert and overlap the representation before it is trained, and filter the synthesized speech representation 306 with injected noise before training the ASR model 200.

[0055] Example training procedure 400 uses multiple synthesized speech representations 306 generated by multilingual TTS model 310 to train ASR model 200. In the example shown, ASR model 200 is trained to recognize speech in a first language (e.g., Kannada). For each synthesized speech representation 306, ASR model 200 generates a corresponding speech recognition result 312. Speech recognition result 312 can represent a probability distribution over possible speech recognition hypotheses. ASR model 200 generates first speech recognition results 312, 312a of native language synthesized speech representation 306a and second speech recognition results 312, 312b of cross-lingual synthesized speech representation 306b. Continuing the example above, ASR model 200 receives a Mel spectrogram frame of native language synthesized speech representation 306a conditioned on a first conditional input 304a (e.g., speaker characteristics of a native Kannada speaker) and generates a first speech recognition result 312a. The ASR model also receives a Mel spectrogram frame of a cross-lingual synthesized speech representation 306b conditioned on a second conditional input 304b (e.g., speaker characteristics of a native English speaker) and generates a second speech recognition result 312b.

[0056] The training process 400 also includes a hierarchical VAE (interchangeably referred to as VAE) 410, which is configured to generate encoder embeddings 412 for the synthesized speech representation 306. VAE 410 includes: a local encoder configured to encode fixed two-second blocks of each synthesized speech representation 306 with one-second overlap; and a global encoder configured to encode the entire synthesized speech representation 306. In the example shown, VAE 410 receives a native-speaker synthesized speech representation 306a and generates native-speaker audio encoder embeddings 412, 412a. The native-speaker audio encoder embedding 412a represents latent variables extracted from the native-speaker synthesized speech representation 306a. For example, the native-speaker audio encoder embedding 412 can represent prosodic / accent information extracted from the native-speaker synthesized speech representation 306a. Furthermore, VAE 410 generates cross-lingual audio encoder embeddings 412, 412b for the cross-lingual synthesized speech representation 306b. The cross-language audio encoder embedding 412b represents latent variables (e.g., prosody / accent information) extracted from the cross-language synthesized speech representation 306b.

[0057] The training process 400 also executes a classifier 420, which receives a native language audio encoder embedding 412a and a cross-language audio encoder embedding 412b. The classifier 420 can be a language classifier. The classifier 420 determines an adversarial loss term 422 based on the native language audio encoder embedding 412a and the cross-language audio encoder embedding 412b. The TTS model 310 receives the adversarial loss term 422 from the classifier 420 and updates its parameters based on the adversarial loss term 422. That is, the adversarial loss term 422 prevents the TTS model 310 from generating a synthesized speech representation 306 that only resembles the prosody / accent information of the speaker 304. In other words, by applying the adversarial loss term 422 to the speaker classification, the encoder embedding 412 extracted from the synthesized speech representation 306 using the VAE 410 is used to separate the speaker information.

[0058] Now for reference Figure 5In some implementations, the TTS model 310 generates synthesized speech representations 306 in different languages ​​to train multiple monolingual ASR models 200 respectively. In the illustrated example, first ASR models 200, 200a are trained on synthesized speech representations 306, 306A in the first language generated by the TTS model 310 to recognize speech in the first language; second ASR models 200, 200b are trained on synthesized speech representations 306, 306A generated by the TTS model 310 in the second language to recognize speech in the second language; and third ASR models 200, 200c are trained on synthesized speech representations 306, 306C generated by the TTS model 310 in the third language to recognize speech in the third language. In other examples, the TTS model 310 generates synthesized speech representations 306 in multiple languages ​​to train a single multilingual ASR model 200 to recognize speech in multiple different languages. Therefore, the multilingual TTS model 310 generates synthesized speech representations from input text sequences in multiple different languages ​​to be used as training audio data for training one or more monolingual ASR models 200 and / or multilingual ASR models 200. The synthesized speech representation for each language may include a native-language synthesized speech representation 306a and / or a cross-lingual speech representation 306b of the corresponding language. Figure 3 Both, the native-language synthesized speech representation 306a is conditioned on the speaker characteristics 304a of the native speaker of the corresponding language, and the cross-lingual speech representation 306b is conditioned on the speaker characteristics 304b of the native speakers of different languages. While the illustrated example depicts training the ASR model 200 on a synthesized speech representation that may include a Mel spectrogram, the ASR model 200 can similarly be trained on the time-domain audio waveform of synthesized speech converted from the synthesized speech representation, for example, via a vocoder (not shown) or other synthesizer device (not shown).

[0059] In the example shown, the TTS model 310 receives an input text sequence 302 and one or more conditional inputs 304 as input, and generates a synthesized speech representation 306 in the corresponding language as output to train an ASR model 200 in the corresponding language. Here, the conditional inputs 304 received by the TTS model 310 may include at least one of a language embedding 303 associated with the corresponding language, a speaker embedding 305 specifying the speech characteristics of the corresponding speaker, or a local embedding 307 specifying accent / dialect information. These conditional inputs 304 are used to conditionally add to the synthesized speech representation 306 generated from the input text sequence 302 in the corresponding language. Thus, the resulting synthesized speech representation 306 can convey a speaking style having the accent / dialect specified by the local embedding and the speech of the target speaker specified by the speaker embedding 305.

[0060] Figure 6A schematic diagram of an example speech recognition system 600 is shown. Here, a TTS model 310 generates synthesized speech representations 306, each conditional on the speaker characteristics 304 of the corresponding speaker of the input text sequence 302. Speaker characteristics 304 may include language embeddings 303, speaker embeddings 305, and / or local embeddings 307. That is, language embeddings 303 may specify language information associated with the language of the synthesized speech representation 306 to be generated, speaker embeddings 305 may represent the speech characteristics of the target speaker, and local embeddings 307 may specify the accent / dialect associated with the synthesized speech representation 306 to be generated.

[0061] The phoneme tokenizer 610 receives the input text sequence 302 and transliterates the language of the text sequence 302 into a native language script. In some embodiments, the phoneme tokenizer 610 transliterates the language of the text sequence 302 into a native language script based on language embedding 303. The phoneme tokenizer 610 tokenizes the native language script into a phoneme sequence 612. All languages ​​of the phoneme tokenizer 610 share the phoneme set derived from the global speech evaluation method, Speech Alphabet (SAMPA). The phoneme tokenizer 610 provides the phoneme sequence 612 corresponding to the input text sequence 302 as input to the TTS model 310.

[0062] The TTS model 310 includes an encoder section 316 that shares a language embedding (i.e., a language identifier) ​​303 across the first and second languages. The language embedding 303 can be input into the encoder section 316 to improve phoneme embedding extraction. Here, the language embedding can be jointly trained with the TTS model 310. The TTS model 310 also includes a decoder section 318 that shares the language embedding 303 across the first and second languages ​​and shares a speaker embedding 305 for different speakers. The decoder section 318 can also share a local embedding 307.

[0063] Encoder section 316 is configured to encode phoneme sequence 612 to generate encoded phoneme sequences 612, 612E. In some embodiments, encoder section 316 includes an attention network configured to receive phoneme sequence 612 to generate a corresponding encoded phoneme sequence 612E as a fixed-length context vector for each output step of decoder section 318. That is, the attention network at encoder section 316 can generate a fixed-length vector for each frame of the Mel spectrogram (e.g., synthesized speech representation 306) that decoder section 318 will later generate. The attention network can determine the weight of each element of the output of encoder section 316 and generate the fixed-length vector by determining the weighted sum of each element. The attention weights can be changed for each time step of decoder section 318.

[0064] Therefore, decoder section 318 is configured to receive encoded phoneme sequence 317 as input from encoder section 316, speaker embedding 305, and language embedding 303 (and optionally local embedding 317) to generate synthesized speech representation 306. In some examples, decoder section 318 is also conditional on prosodic information extracted from synthesized speech representation 306 (i.e., encoder embedding 412). Synthetic speech representation 306 is in the language specified by language embedding 303 and represents the speech of a target speaker specified by speaker embedding 303 (e.g., a native speaker of first language 304a or a native speaker of second language 304b).

[0065] Decoder section 318 decodes the encoded phoneme sequence 312E to generate either a native-language synthesized speech representation 306a or a cross-language synthesized speech representation 306b. For example, if decoder section 318 receives the language embedding 303 of the first language and the speaker embedding 305 of the native speaker of the first language 304a, decoder section 318 decodes the encoded phoneme sequence 612E and generates the native-language synthesized speech representation 306a. In an alternative example, where decoder section 318 receives the language embedding 303 of the first language and the speaker embedding 305 of the native speaker of the second language 304b, decoder section 318 decodes the encoded phoneme sequence 612E and generates the cross-language synthesized speech representation 306b.

[0066] In the illustrated example, the TTS model 310 provides a synthesized speech representation 306 as input to the VAE 410. The VAE 410 is configured to consume the synthesized speech representation 306 and output either a native speaker audio encoder embedding 412a or a cross-lingual audio encoder embedding 412b. That is, the VAE 410 extracts latent variables (e.g., prosodic information) from the synthesized speech representation 306. For example, if the VAE 410 receives a native speaker synthesized speech representation 306a, the VAE 410 extracts the latent variables of the native speaker synthesized speech representation 306a and generates a native speaker audio encoder embedding 412a. Alternatively, if the VAE 410 receives a cross-lingual synthesized speech representation 306b, the VAE 410 extracts the latent variables of the cross-lingual synthesized speech representation 306b and generates a cross-lingual audio encoder embedding 412b. In some embodiments, the decoder portion 318 is further conditioned on the prosodic information extracted by the VAE 410 from the synthesized speech representation 306. In other words, the decoder section 318 receives the audio encoder embedding 412 and further conditions the synthesized speech representation 306 based on the audio encoder embedding 412. An adversarial loss 422 can be applied to the encoded phoneme sequence 612E to separate prosody from speaker information.

[0067] Classifier 420 receives audio encoder embedding 412 and generates adversarial loss 422. In some examples, classifier 420 includes a language classifier. In other examples, classifier 420 includes an adversarial or speaker classifier. Classifier 420 can be configured to separate prosodic information extracted from synthetic speech representation 306 by applying adversarial loss to speaker classification. TTS model 310 can receive adversarial loss 422 and update parameters based on adversarial loss 422.

[0068] Figure 7 This is a flowchart of an exemplary operational arrangement of a computer-implemented method 700 for training an Automatic Speech Recognition (ASR) model 200. In operation 702, method 700 includes obtaining a multilingual text-to-speech (TTS) model 310. In operation 704, method 700 includes generating native-language synthesized speech representations 306, 306a for a first-language input text sequence 302 using the multilingual TTS model 310, conditioned on speaker characteristics 304a of native speakers of the first language. Here, the first language might be Kannada, a low-resource language with only a few native speakers with limited speaking styles. In operation 706, method 700 includes generating cross-lingual synthesized speech representations 306, 306b for a first-language input text sequence 302 using the TTS model 310, conditioned on speaker characteristics 304b of native speakers of different second languages. The second language could be English, with thousands of native speakers 304b with various speaking styles.

[0069] In operation 708, method 700 includes using ASR model 200 to generate first speech recognition results 312, 312a for native language synthesized speech representation 306a and second speech recognition results 312, 312b for cross-language synthesized speech representation 306b. In operation 710, method 700 includes determining a consistency loss term 352 based on the first speech recognition result 312a and the second speech recognition result 312b. In operation 712, method 700 includes updating the parameters of ASR model 200 based on the consistency loss term 352. Optionally, the parameters of ASR model 200 can be fixed, while the consistency loss term 352 is backpropagated through TTS model 310 for updating the parameters of TTS model 310 based on the consistency loss term 352.

[0070] Figure 8This is a schematic diagram of an example computing device 800 that can be used to implement the systems and methods described in this document. The computing device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0071] Computing device 800 includes a processor 810, a memory 820, a storage device 830, a high-speed interface / controller 840 connected to the memory 820 and a high-speed expansion port 850, and a low-speed interface / controller 860 connected to a low-speed bus 870 and the storage device 830. Each of components 810, 820, 830, 840, 850, and 860 is interconnected using various buses and can be mounted on a common motherboard or otherwise suitably mounted. Processor 810 can process instructions for execution within computing device 800, including instructions stored in memory 820 or on storage device 830 for displaying graphical information for a graphical user interface (GUI) on an external input / output device such as a display 880 coupled to the high-speed interface 840. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and various memory types, can be suitably used. Furthermore, multiple computing devices 800 can be connected, each providing a portion of the necessary operation (e.g., as a server group, a set of blade servers, or a multiprocessor system).

[0072] Memory 820 stores information non-transitorily within computing device 800. Memory 820 may be a computer-readable medium, a volatile storage unit, or a non-volatile storage unit. Non-transitory memory 820 may be a physical device for storing programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 800. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0073] Storage device 830 provides mass storage for computing device 800. In some embodiments, storage device 830 is a computer-readable medium. In various embodiments, storage device 830 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or device array, including devices in a storage area network or other configuration. In other embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 820, storage device 830, or memory on processor 810.

[0074] High-speed controller 840 manages bandwidth-intensive operations of computing device 800, while low-speed controller 860 manages lower bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 840 is coupled to memory 820, display 880 (e.g., via a graphics processor or accelerator), and to high-speed expansion port 850 which can accept various expansion cards (not shown). In some embodiments, low-speed controller 860 is coupled to storage device 830 and low-speed expansion port 890. Low-speed expansion port 890, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or network devices (e.g., switches or routers), for example, via a network adapter.

[0075] The computing device 800 can be implemented in a variety of different forms, as shown in the figure. For example, it can be implemented as a standard server 800a or multiple times in a set of such servers 800a, as a laptop computer 800b, or as part of a rack server system 800c.

[0076] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system, which includes at least one programmable processor, which may be specialized or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0077] A software application (i.e., a software resource) can refer to computer software that instructs a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0078] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” mean any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, which includes a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” means any signal used to provide machine instructions and / or data to a programmable processor.

[0079] The processes and logical flows described in this specification can be executed by one or more programmable processors that perform functions by manipulating input data and generating output. The processes and logical flows can also be executed by special-purpose logic circuitry (e.g., FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit)). For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to them to receive data from or transfer data to, or both. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory can be supplemented by dedicated logic circuits or incorporated into dedicated logic circuits.

[0080] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen) and optionally a keyboard and indicating devices, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending files to and receiving files from the device used by the user (e.g., by sending a webpage to a web browser in response to a request received from a web browser on the user's client device).

[0081] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A computer-implemented method (700), which, when executed on data processing hardware (810), causes the data processing hardware (810) to perform operations for training a speech recognition model (200), the operations comprising: Obtain a multilingual text-to-speech (TTS) model (310); The multilingual TTS model (310) is used to generate a native language synthesized speech representation (306) for the input text sequence (302) of the first language, wherein the native language synthesized speech representation is conditioned on the speaker characteristics (304) of the native speaker of the first language. The multilingual TTS model (310) is used to generate a cross-lingual synthesized speech representation (306) for the input text sequence (302) in the first language, the cross-lingual synthesized speech representation being conditioned on the speaker characteristics (304) of native speakers of different second languages; The speech recognition model (200) is used to generate a first speech recognition result (312) of the native language synthesized speech representation (306) and a second speech recognition result (312) of the cross-language synthesized speech representation (306); Based on the first speech recognition result (312) and the second speech recognition result (312), a consistency loss term (352) is determined; and The parameters of the speech recognition model (200) are updated based on the consistency loss term (352).

2. The computer-implemented method (700) according to claim 1, wherein, The operation also includes: A first cross-entropy loss term (342) is generated based on the first speech recognition result (312) and the input text sequence (302) of the first language; The second cross-entropy loss term (342) is determined based on the second speech recognition result (312) and the input text sequence (302) of the first language; and The parameters of the speech recognition model are updated based on the first cross-entropy loss term (342) and the second cross-entropy loss term (342).

3. The computer-implemented method (700) according to claim 2, wherein, The parameters of the speech recognition model (200) are updated based on the consistency loss term (352) without relying on the first cross-entropy loss term (342) and the second cross-entropy loss term (342).

4. The computer-implemented method (700) according to claim 2, wherein, The operation further includes backpropagating the first cross-entropy loss term (342) and the second cross-entropy loss term (342) through the multilingual TTS model (310). During the backpropagation of the first cross-entropy loss term (342) and the second cross-entropy loss term (342) through the multilingual TTS model (310), the parameters of the speech recognition model are static.

5. The computer-implemented method (700) according to claim 1, wherein, The operation also includes applying data augmentation to at least one of the native language synthesized speech representation (306) or the cross-language synthesized speech representation (306).

6. The computer-implemented method (700) according to claim 1, wherein, The multilingual TTS model (310) includes: The encoder portion (316) shares a language embedding (303) across the first language and the second language; and The decoder part (318) shares the language embedding (303) across the first language and the second language and shares the speaker embedding (303) of both the native speaker of the first language and the native speaker of the second language.

7. The computer-implemented method (700) according to claim 6, wherein, The number of speaker embeddings (303) of the native speaker of the first language is less than the number of speaker embeddings (303) of the native speaker of the second language.

8. The computer-implemented method (700) according to claim 6, wherein, The decoder section (318) is further conditioned on prosodic information extracted from the synthesized speech representation (306) using a variational autoencoder (410).

9. The computer-implemented method (700) according to claim 8, wherein, The prosodic information extracted from the synthesized speech representation (306) using the variational autoencoder (410) is separated from the speaker information by applying an adversarial loss to the speaker classification.

10. The computer-implemented method (700) according to claim 1, wherein, The operation also includes, before generating the native language synthesized speech representation and the cross-language synthesized speech representation (306): Transliterate the input text sequence (302) in the first language into a script in the native language; The native language script lexicalization is converted into a phoneme sequence; The phoneme sequence is encoded using the encoder of the multilingual TTS model (310); as well as The decoder of the multilingual TTS model (310) is used to decode the encoded phoneme sequence to generate either the native language synthesized speech representation (306) or the cross-lingual synthesized speech representation (306).

11. The computer-implemented method (700) according to any one of claims 1-10, wherein, The operation also includes: A native language audio encoder embedding (412) is generated for the native language synthesized speech representation (306) using a variational autoencoder (410); The variational autoencoder (410) is used to generate a cross-language audio encoder embedding (412) for the cross-language synthesized speech representation (306); Based on the native language audio encoder embedding and the cross-language audio encoder embedding (412), an adversarial loss term conditioned on the first language is determined (422); and The parameters of the multilingual TTS model (310) are updated based on the adversarial loss term (422).

12. A system (800) for training a speech recognition model (200), the system comprising: Data processing hardware (810); as well as A memory hardware (820) communicating with the data processing hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations including: Obtain a multilingual text-to-speech (TTS) model (310); The multilingual TTS model (310) is used to generate a native language synthesized speech representation (306) for the input text sequence (302) of the first language, wherein the native language synthesized speech representation is conditioned on the speaker characteristics (304) of the native speaker of the first language. The multilingual TTS model (310) is used to generate a cross-lingual synthesized speech representation (306) for the input text sequence (302) in the first language, the cross-lingual synthesized speech representation being conditioned on the speaker characteristics (304) of native speakers of different second languages; The speech recognition model (200) is used to generate a first speech recognition result (312) of the native language synthesized speech representation (306) and a second speech recognition result (312) of the cross-language synthesized speech representation (306); Based on the first speech recognition result (312) and the second speech recognition result (312), a consistency loss term (352) is determined; and The parameters of the speech recognition model (200) are updated based on the consistency loss term (352).

13. The system (800) according to claim 12, wherein, The operation also includes: A first cross-entropy loss term (342) is generated based on the first language recognition result (312) and the input text sequence (302) of the first language; The second cross-entropy loss term (342) is determined based on the second speech recognition result (312) and the input text sequence (302) of the first language; and The parameters of the speech recognition model (200) are updated based on the first cross-entropy loss term (342) and the second cross-entropy loss term (342).

14. The system (800) according to claim 13, wherein, The parameters of the speech recognition model (200) are updated based on the consistency loss term (352) without relying on the first cross-entropy loss term (342) and the second cross-entropy loss term (342).

15. The system (800) according to claim 13, wherein, The operation further includes backpropagating the first cross-entropy loss term (342) and the second cross-entropy loss term (342) through the multilingual TTS model (310). During the backpropagation of the first cross-entropy loss term (342) and the second cross-entropy loss term (342) through the multilingual TTS model (310), the parameters of the speech recognition model are static.

16. The system (800) according to claim 12, wherein, The operation also includes applying data augmentation to at least one of the native language synthesized speech representation (306) or the cross-language synthesized speech representation (306).

17. The system (800) according to claim 12, wherein, The multilingual TTS model (310) includes: The encoder portion (316) shares a language embedding (303) across the first language and the second language; and The decoder part (318) shares the language embedding (303) across the first language and the second language and shares the speaker embedding (303) of both the native speaker of the first language and the native speaker of the second language.

18. The system (800) according to claim 17, wherein, The number of speaker embeddings (303) of the native speaker of the first language is less than the number of speaker embeddings (303) of the native speaker of the second language.

19. The system (800) according to claim 17, wherein, The decoder section (318) is further conditioned on prosodic information extracted from the synthesized speech representation (306) using a variational autoencoder (410).

20. The system (800) according to claim 19, wherein, The prosodic information extracted from the synthesized speech representation (306) using the variational autoencoder (410) is separated from the speaker information by applying an adversarial loss to the speaker classification.

21. The system (800) according to claim 12, wherein, The operation also includes, before generating the native language synthesized speech representation and the cross-language synthesized speech representation (306): Transliterate the input text sequence (302) in the first language into a script in the native language; The native language script lexicalization is converted into a phoneme sequence; The phoneme sequence is encoded using the encoder of the multilingual TTS model (310); as well as The decoder of the multilingual TTS model (310) is used to decode the encoded phoneme sequence to generate either the native language synthesized speech representation (306) or the cross-lingual synthesized speech representation (306).

22. The system (800) according to any one of claims 12-21, wherein, The operation also includes: A native language audio encoder embedding (412) is generated for the native language synthesized speech representation (306) using a variational autoencoder (410); The variational autoencoder (410) is used to generate a cross-language audio encoder embedding (412) for the cross-language synthesized speech representation (306); Based on the native language audio encoder embedding and the cross-language audio encoder embedding (412), an adversarial loss term conditioned on the first language is determined (422); and The parameters of the multilingual TTS model (310) are updated based on the adversarial loss term (422).

Images