Using non-parallel voice conversion to train speech recognition models.

Abstract

The method includes receiving a set of training utterances, each of which includes a non-synthesized speech representation (304), and generating, for each training utterance, a corresponding speech-converted synthetic speech representation (306) by a speech-conversion model (400). The non-synthesized speech representation and the synthetic speech representation form a corresponding training utterance pair (302). At each of a plurality of output steps for each training utterance pair, the method also includes generating, for output by the speech recognition model (200), a first probability distribution (311) over possible non-synthesized speech recognition hypotheses for the non-synthesized speech representation, and a second probability distribution (312) over possible synthetic speech recognition hypotheses for the synthetic speech representation. The method also includes determining a consistency loss term for the corresponding training utterance pair based on the first and second probability distributions, and updating parameters of the speech recognition model based on the consistency loss term.

Description

[Technical Field]

[0001] The present disclosure relates to using non-parallel speech conversion for speech recognition and joint recognition / conversion models. [Background technology]

[0002] Automatic speech recognition (ASR), the process of taking audio input and transcribing it into text, is a critical technology used in mobile and other devices. Generally, automatic speech recognition attempts to accurately transcribe what a person said by taking audio input (e.g., a spoken utterance) and transcribing the audio input into text. However, one challenge in developing deep learning-based ASR models is that their parameters tend to overfit to the training data, resulting in the ASR model having difficulty generalizing to unseen data when the training data is not sufficiently extensive. For example, speech recognition can be limited by the speaker identities and speaker characteristics present in the training data. As a result, training an ASR model with a larger training dataset improves its accuracy. Synthetic speech and / or data-augmented speech can be incorporated to increase the amount of training data used to train an ASR model. Summary of the Invention

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed by data processing hardware, causes the data processing hardware to perform operations including: receiving a set of training utterances, each including a non-synthesized speech representation of a corresponding utterance; and, for each training utterance, generating a corresponding converted synthetic speech representation by converting the non-synthesized speech representation into a corresponding converted synthetic speech representation of the corresponding utterance using a speech conversion model. The non-synthesized speech representation and the synthetic speech representation form a corresponding training utterance pair. At each of a plurality of output steps for each training utterance pair in the set of training utterance pairs, the operations further include generating, for output by the speech recognition model, a first probability distribution over possible non-synthesized speech recognition hypotheses for the corresponding non-synthesized speech representation of the corresponding utterance; generating, for output by the speech recognition model, a second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance; and determining a consistency loss term for the corresponding training utterance pair based on the first probability distribution over the possible non-synthesized speech recognition hypotheses and the second probability distribution over the possible non-synthesized speech recognition hypotheses. The operations also include updating parameters of the speech recognition model based on the consistency loss term determined at each of the plurality of output steps for each training utterance pair of the set of training utterance pairs.

[0004] Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the operations also include, at each of a plurality of output steps for each training utterance pair in the set of training utterance pairs, generating a non-synthesized speech loss term based on a first probability distribution for possible non-synthesized speech recognition hypotheses for a corresponding non-synthesized speech representation of the corresponding utterance and a ground truth transcription of the corresponding utterance, and generating a synthetic speech loss term based on a second probability distribution for possible synthetic speech recognition hypotheses for a corresponding synthetic speech representation of the corresponding utterance and a ground truth transcription of the corresponding utterance. In these embodiments, parameters of the speech recognition model may be updated for each training utterance pair in the set of training utterance pairs based on the consistent loss term determined at each of the plurality of output steps, independently of the non-synthesized speech loss term and the synthetic speech loss term generated at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs.

[0005] In some examples, the non-synthetic speech representation of the corresponding utterance includes speech spoken by a source speaker and conveys an accent / locality associated with the source speaker, and the speech-conversion synthetic speech representation of the corresponding utterance includes synthetic speech representing the corresponding utterance conveying an accent / locality associated with a target speaker, where the accent / locality associated with the source speaker may differ from the accent / locality associated with the target speaker. Optionally, the speech-conversion synthetic speech representation can convey the same linguistic content as the non-synthetic speech representation of the corresponding utterance. The speech-conversion model may include a non-parallel speech-conversion model.

[0006] In some implementations, the speech conversion model includes a content encoder, a vector quantization-variable autoencoder (VQ-VAE) layer, and a decoder. The content encoder is configured to receive input speech as input and generate an initial latent representation as output. The VQ-VAE layer is configured to receive the initial latent representation as input and generate a latent representation of linguistic content extracted from the input speech, discarding speaking style variations of the input speech, as output. The decoder is conditioned based on the speaker embedding and configured to generate output speech based on the latent representation of the linguistic content. The content encoder may include a pre-trained speech recognition (ASR) encoder previously trained on an ASR loss for a speech recognition task, where parameters of the pre-trained ASR encoder remain fixed during training of the VQ-VAE layer and the decoder of the speech conversion model. Additionally or alternatively, the VQ-VAE layer may be trained using a VQ loss based on the latent representation of linguistic content generated at each time step, where the VQ loss prompts the VQ-VAE layer to minimize the distance between the output and the closest codebook. In some implementations, the decoder is configured to receive, as input, input speech and a latent representation of linguistic content for a speaker embedding, and to generate, as output, output speech comprising a reconstruction of the input speech.

[0007] Another aspect of the present disclosure provides a system that includes data processing hardware and memory hardware in communication with the data processing hardware and storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving a set of training utterances, each utterance including a non-synthesized speech representation of a corresponding utterance, and for each training utterance, generating a corresponding speech-conversion synthetic speech representation by converting the non-synthesized speech representation into a corresponding speech-conversion synthetic speech representation of the corresponding utterance using a speech-conversion model, wherein the non-synthesized speech representation and the synthetic speech representation form a corresponding training utterance pair. At each of the plurality of output steps for each training utterance pair of the set of training utterance pairs, the operations further include generating, for output by the speech recognition model, a first probability distribution over possible non-synthesized speech recognition hypotheses for the corresponding non-synthesized speech representation of the corresponding utterance, generating, for output by the speech recognition model, a second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance, and determining a consistency loss term for the corresponding training utterance pair based on the first probability distribution over the possible non-synthesized speech recognition hypotheses and the second probability distribution over the possible non-synthesized speech recognition hypotheses. The operations also include updating parameters of the speech recognition model based on the consistency loss term determined at each of the plurality of output steps for each training utterance pair of the set of training utterance pairs.

[0008] This aspect may include one or more of the following optional features. In some implementations, the operations also include, at each of a plurality of output steps for each training utterance pair in the set of training utterance pairs, generating a non-synthesized speech loss term based on a first probability distribution for possible non-synthesized speech recognition hypotheses for a corresponding non-synthesized speech representation of the corresponding utterance and a ground truth transcription of the corresponding utterance, and generating a synthetic speech loss term based on a second probability distribution for possible synthetic speech recognition hypotheses for a corresponding synthetic speech representation of the corresponding utterance and a ground truth transcription of the corresponding utterance. In these implementations, parameters of the speech recognition model may be updated for each training utterance pair in the set of training utterance pairs based on the consistent loss term determined at each of the plurality of output steps, independently of the non-synthesized speech loss term and the synthetic speech loss term generated at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs.

[0009] In some examples, the non-synthetic speech representation of the corresponding utterance includes speech spoken by a source speaker and conveys an accent / locality associated with the source speaker, and the speech-conversion synthetic speech representation of the corresponding utterance includes synthetic speech representing the corresponding utterance conveying an accent / locality associated with a target speaker, where the accent / locality associated with the source speaker may differ from the accent / locality associated with the target speaker. Optionally, the speech-conversion synthetic speech representation can convey the same linguistic content as the non-synthetic speech representation of the corresponding utterance. The speech-conversion model may include a non-parallel speech-conversion model.

[0010] In some implementations, the speech conversion model includes a content encoder, a vector quantization-variable autoencoder (VQ-VAE) layer, and a decoder. The content encoder is configured to receive input speech as input and generate an initial latent representation as output. The VQ-VAE layer is configured to receive the initial latent representation as input and generate a latent representation of linguistic content extracted from the input speech, discarding speaking style variations of the input speech, as output. The decoder is conditioned based on the speaker embedding and configured to generate output speech based on the latent representation of the linguistic content. The content encoder may include a pre-trained speech recognition (ASR) encoder previously trained on an ASR loss for a speech recognition task, where parameters of the pre-trained ASR encoder remain fixed during training of the VQ-VAE layer and the decoder of the speech conversion model. Additionally or alternatively, the VQ-VAE layer may be trained using a VQ loss based on the latent representation of linguistic content generated at each time step, where the VQ loss prompts the VQ-VAE layer to minimize the distance between the output and the closest codebook. In some implementations, the decoder is configured to receive, as input, input speech and a latent representation of linguistic content for a speaker embedding, and to generate, as output, output speech comprising a reconstruction of the input speech.

[0011] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of an exemplary speech recognition system. [Figure 2A] FIG. 1 is a schematic diagram of a recurrent neural network-transducer (RNN-T) model architecture. [Figure 2B]1 is a schematic diagram of an attention-based encoder-decoder (AED) model architecture. [Figure 3] FIG. 1 is a schematic diagram of an exemplary training process for promoting a speech recognition model to learn consistent predictions for both unsynthesized and speech-to-text synthesized speech. [Figure 4] FIG. 2 is a schematic diagram of an exemplary non-parallel voice conversion model. [Figure 5] 1 is a flowchart of an exemplary configuration of operations of a method for training a speech recognition model using a speech conversion synthesized speech representation of an utterance converted by a speech conversion model from a non-synthesized speech representation of the same utterance. [Figure 6] FIG. 1 is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0013] Like reference symbols in the various drawings indicate like elements. The introduction of sequence-to-sequence (Seq2Seq) models, which map audio to strings of characters, has made great advances in automatic speech recognition. At the same time, text-to-speech (TTS) or speech synthesis systems have successfully applied Seq2Seq models to obtain state-of-the-art natural and lifelike sounding synthetic speech that is indistinguishable from human speech to the human ear.

[0014] One challenge in developing deep learning-based ASR models is that their parameters tend to overfit to the training data, resulting in difficulty in generalizing to unseen data when the training data is not sufficiently comprehensive. As a result, training an ASR model on a larger training dataset improves its accuracy. For example, using machine learning or other statistical methods, an ASR model can be trained on a training dataset containing 10,000 hours or more of transcribed speech. However, the performance of an ASR model degrades when the domain related to the training data differs from the domain in which the ASR model is deployed during inference. For example, training an ASR model on speech transcribed in a domain related to videoconferencing can result in a loss of speech related to voice search queries and a reduced effectiveness in recognizing such speech, and vice versa. ASR models are further limited by the identity and characteristics of the speakers represented in the training data. For example, speech typically conveys speaker characteristics such as pitch, timbre, and prosody / style of the speaker who spoke the training utterances.

[0015] Embodiments herein are directed to using voice conversion to manipulate speaker characteristics (e.g., pitch, timbre, prosody, style) in spoken training utterances to expose ASR or automatic voice conversion (e.g., speech-to-speech) models to a more diverse range of speakers, thus improving the robustness of the resulting model. Embodiments herein further include using voice conversion extensions to modify the perceived accent of input training utterances to improve speech recognition performance for accented speech or speech from other dialects / regions that are poorly represented / underrepresented in the available transcribed training data (i.e., the spoken utterances paired with the transcriptions).

[0016] Although there are examples where state-of-the-art synthetic speech is generally indistinguishable from human speech, the use of synthetic speech has been shown to affect ASR training differently from human speech. This mismatch between synthetic and human speech stems from the inconsistency of synthetic speech data from human speech data, which arises from the challenging one-to-many mapping problem that both speech conversion and TTS systems attempt to solve. That is, while the overall quality of available synthetic speech is very high, synthetic speech exhibits much less variation than human speech and minimal speech disfluencies. As a result, training an ASR model solely with synthesized speech data presents challenges in generalizing to real-world speech utterances during inference.

[0017] Embodiments herein are further directed to training a speech conversion model (i.e., an ASR model and / or a speech-to-speech (S2S) model) with real / human and synthesized speech representations of the same training utterances, and introducing a consistent loss term to promote consistent predictions (e.g., speech recognition hypotheses) for both the real / human and synthetic representations of the same training utterances. In essence, the consistent loss term between the human representation and the synthetic representation of the same utterance provides an aspect of unsupervised training by encouraging the speech conversion model to behave consistently for both human and synthetic training utterances.

[0018] 1 illustrates an automatic speech recognition (ASR) system 100 that implements an ASR model 200 residing on a user device 102 of a user 104 and / or on a remote computing device 201 (e.g., one or more servers of a distributed system running in a cloud computing environment) that communicates with the user device 102. While the user device 102 is shown as a mobile computing device (e.g., a smartphone), the user device 102 may correspond to any type of computing device, such as, without limitation, a tablet device, a laptop / desktop computer, a wearable device, a digital assistant device, a smart speaker / display, a smart appliance, an automotive infotainment system, or an Internet of Things (IoT) device, and includes data processing hardware 111 and memory hardware 113.

[0019] The user device 102 includes an audio subsystem 108 configured to receive utterances 106 uttered by a user 104 (e.g., the user device 102 may include one or more microphones for recording the spoken utterances 106) and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the ASR system 100. In the illustrated example, the user speaks each utterance 106 in the natural language of English for the phrase "What's the weather like in New York City?", and the audio subsystem 108 converts the utterance 106 into corresponding acoustic frames 110 for input to the ASR system 100. The ASR model 200 then receives the acoustic frames 110 corresponding to the utterances 106 as input and generates / predicts a corresponding transcription 120 (e.g., a recognition result / hypothesis) of the utterance 106 as output. In the illustrated example, the user device 102 and / or the remote computing device 201 also execute a user interface generator 107 configured to present a representation of the transcription 120 of the utterance 106 to the user 104 of the user device 102. In some configurations, the transcription 120 output from the ASR system 100 is processed by a natural language understanding (NLU) module executing, for example, on the user device 102 or the remote computing device 201, to execute user commands. Additionally or alternatively, a text-to-speech system (e.g., executing on any combination of the user device 102 or the remote computing device 201) can convert the transcription into synthetic speech and output audibly on another device. For example, the original utterance 106 corresponds to a message the user 104 sends to a friend, in which the transcription 120 is converted into synthetic speech and output audibly so that the friend can hear the message conveyed in the original utterance 106.

[0020] 2A and 2B, the ASR model 200 may include an end-to-end (E2E) sequence-to-sequence model, such as a frame alignment-based transducer model 200a (FIG. 2A) or an attention-based encoder-decoder (AED) model 200b (FIG. 2B). The ASR model 200 can provide E2E speech recognition by integrating acoustic, pronunciation, and language models into a single neural network, eliminating the need for a lexicon or separate text normalization component. Various architectures and optimization mechanisms can improve accuracy and reduce model training time.

[0021] Referring to FIG. 2A , an exemplary frame alignment-based transducer model 200a includes a recurrent neural network-transducer (RNN-T) model architecture that adheres to latency constraints associated with interactive applications. The RNN-T model 200a provides a small computational footprint and utilizes smaller memory requirements than traditional ASR architectures, making it suitable for performing speech recognition entirely on the user device 102 (e.g., no communication with a remote server is required). The RNN-T model 200a includes an encoder network 210, a prediction network 220, and a joint network 230. The encoder network 210, which is broadly similar to an acoustic model (AM) in traditional ASR systems, includes a recurrent network of stacked long short-term memory (LSTM) layers. For example, the encoder receives a sequence of d-dimensional feature vectors (e.g., acoustic frames 110 ( FIG. 1 )) x=(x1, x2, ..., x T ) where:

[0022]

number

[0023] At each time step, we generate a higher-order representation, which is

[0024]

number

[0025] It is expressed as: Similarly, the prediction network 220 is also an LSTM network, and like the language model (LM), it calculates the sequence of non-blank symbols y0,...,y0 output by the final softmax layer 240. ui-1 A dense representation of

[0026] [Number]

[0027] Finally, using the RNN-T model architecture, the representations generated by the encoder and predictor networks 210, 220 are combined by a joint network 230. The joint network then

[0028]

number

[0029] , which is the distribution for the next output symbol. In other words, at each output step (e.g., time step), the joint network 230 generates a probability distribution for possible speech recognition hypotheses. Here, a "possible speech recognition hypothesis" corresponds to a set of output labels, each representing a symbol / character in a specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, one label for each of the 26 letters in the English alphabet and one label representing a space. Thus, the joint network 230 may output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels. This set of values ​​may be a vector and may indicate a probability distribution for the set of output labels. In some cases, the output labels are graphemes (such as individual characters, potential punctuation marks, and other symbols), but the set of output labels is not limited to this. For example, the set of output labels may include parts of words and / or entire words in addition to or instead of graphemes. The output distribution of the joint network 230 may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output of the joint network 230 (y i ) can contain 100 different probability values, one for each output label. The probability distributions can then be used to select candidate spelling elements (e.g., graphemes, parts of words, and / or words) in a beam search process (e.g., by a softmax layer 240), assign scores, and determine the transcription 120.

[0030] The softmax layer 240 can use any technique to select the output label / symbol with the highest probability from the distribution as the next output symbol predicted by the RNN-T model 200a at the corresponding output step. In this way, the RNN-T model 200a does not make a conditional independence assumption; rather, the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model 200a assumes that the output symbols are independent of future acoustic frames 110. This allows the RNN-T model to be used in a streaming manner.

[0031] In some examples, the encoder network 210 of the RNN-T model 200a may consist of eight 2,048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. The prediction network 220 may have two 2,048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. Finally, the joint network 230 may also have 640 hidden units. Instead of LSTM layers, the encoder and / or prediction networks 210, 220 may instead use multi-head self-attention layers, such as conformer layers, transformer layers, or lightweight convolutional (LConv) layers. The prediction network 220 may instead use an embedding lookup table instead of a network of LSTMs or multi-head self-attention layers. The softmax layer 240 may consist of a set of unified word fragments or graphemes generated using all unique word fragments or graphemes from multiple training datasets 301 (FIG. 3).

[0032] 2B, an exemplary AED model 200b associated with the architecture of a Listen-Attend-And-Spell (LAS) model provides a single neural network including a listener-encoder module 211, which is similar to a traditional acoustic model, an attendant module 221, which acts as an alignment model, and a decoder 231, which is similar to a language model in traditional systems. Specifically, the listener-encoder module 211 takes input features (e.g., acoustic frame 110 (FIG. 1)), x, and converts them into a higher-level feature representation h. enc The encoded feature representation h enc This process of generating f can be done for each of several input frames, representing different input time steps. These time steps are denoted below with the subscript u. Thus, given a set of frames {f1,f2,f3,...f u}, the corresponding set of encoding outputs {h1,h2,h3,...h u}.

[0033] The output of the listener encoder module 211 is passed to an attendant module 221, which, like a dynamic time warping (DTW) alignment module, determines the next output symbol y i To predict h enc In some examples, the attender module 221 is referred to herein as an attender neural network or attender 221. The attender 221 generates a context output c for each of a plurality of output steps i. i Each context output vector c i For u, the attender 221 can compute attention based on the encoding of one or more input steps u, e.g., the encoding of the current input step and the encoding of a previous input step. For example, the attender 221 can compute attention based on the set of all encoder outputs of the utterance, e.g., the entire set {h1, h2, h3, ... h u} and attention context output ci The attention context vector may be a vector that represents a weighted summary of the current and previous encodings of a frame (e.g., part) of speech being recognized.

[0034] Finally, the output of the attender 221 is passed to the decoder 231, which converts the attention context (e.g., context vector or attention distribution), c i , as well as the preceding prediction y to generate the decoder output. i -1 embedding. The decoder output is the embedding of the previous unit {y i-1 ,...,y0} and input x, the current subword unit y i Probability distribution for P(y i |y i-1 ,...,y0,x). Thus, at each output step, decoder 231 generates a probability distribution over possible speech recognition hypotheses. Similar to RNN-T model 200a discussed above with reference to FIG. 2A, a "possible speech recognition hypothesis" corresponds to a set of output symbols, each representing a symbol / character in the specified natural language.

[0035] Although not shown, ASR model 200 may include a softmax layer that receives the output of decoder 231. In some implementations, the softmax layer is separate from decoder 231 and receives the output y i , and the output of the softmax layer is then used in a beam search process to select the orthogonal projection elements. In some implementations, the softmax layer is integrated with the decoder 231, so that the output y i represents the output of the softmax layer.

[0036] The decoder 231 and / or associated softmax layer may be trained to output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels. This set of values ​​may be a vector, indicating a probability distribution over the set of output labels. In some cases, the output labels are graphemes (such as individual characters, potential punctuation marks, or other symbols), but the set of output labels is not so restricted. For example, the set of output labels may include parts of words and / or entire words in addition to or instead of graphemes. The output distribution of the decoder 231 and / or softmax layer may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the decoder output y i or output y i The output of a softmax layer that receives and processes can include 100 different probability values, one for each output label. The probability distributions can then be used to select candidate spelling elements (e.g., graphemes, parts of words, and / or words) in a beam search process, assigning scores, and determining the transcription.

[0037] Referring to FIG. 3 , an exemplary training process 300 for facilitating the ASR model 200 to learn consistent predictions for both non-synthetic speech (e.g., real / human speech) and voice conversion (VC) synthetic speech (e.g., synthesized speech) involves training a loss term (J) that is consistent between training utterance pairs 302 (also referred to as “training audio pairs”) of non-synthetic and VC synthetic speech representations 304, 306 of the same training utterance. consIn essence, the consistent loss terms between the non-synthetic and VC synthetic speech representations of the same training utterances provide an aspect of unsupervised training by encouraging ASR model 200 to behave consistently regardless of whether the training utterance belongs to non-synthetic or VC synthetic speech, and independent of the supervised loss terms between ground truth transcription 320 and each of the non-synthetic and VC synthetic speech recognition hypotheses output by ASR model 200.

[0038] The training process 300 may be executed on the remote computing device 201 of FIG. 1. For example, the remote computing device 201 includes data processing hardware 610 (FIG. 6) and memory hardware 620 (FIG. 6) in communication with the data processing hardware and storing instructions that, when executed on the data processing hardware 610, cause the data processing hardware 610 to perform the operations of the training process 300. In the illustrated example, the training process 300 receives a set of training utterance pairs 302, each of which includes a non-synthesized speech representation 304 (e.g., human / real speech) of a corresponding utterance and a VC synthetic speech representation 306 of the same corresponding utterance. Each training utterance pair 302 is also associated with the same ground truth transcription 320 of the corresponding utterance. In some implementations, the non-synthesized speech representation 304 is initially paired with the ground truth transcription 320. For example, each non-synthesized speech representation 304 may be manually transcribed by a human listener. In these implementations, a non-parallel speech conversion (VC) model 400 may directly convert a non-synthesized speech representation 304 of a corresponding utterance into a VC synthetic speech representation 306, resulting in a corresponding training utterance pair 302. Here, the VC synthetic speech representation 306 is associated with the synthetic speech generated by the VC model 400 by converting the audio associated with the non-synthesized speech representation 304 into synthetic audio without having to perform any intermediate speech recognition. The VC model 400 may apply a speaker embedding 401, z, when converting the non-synthesized speech representation 304 to obtain synthetic speech of a particular accent and / or speaking style and / or prosody associated with the speaker embedding 401. Thus, the VC model 400 may convert the non-synthesized speech representation 304 into any number of new VC synthetic speech representations 306 based on how many different unique speaker embeddings 401 are available during estimation. Here, the non-synthesized speech representation (x) 304 may be used by the VC model 400 to generate one or more new VC synthetic speech representations 306.

[0039]

number

[0040] Each representation has the expectation that it is consistent with the unsynthesized speech representation (x) 304 associated with the ground truth transcription (y*) 320.

[0041] In some examples, the training process 300 applies data augmentation to at least one of the non-synthesized speech representations 304 or the VC-synthesized speech utterance representations 306 of the training utterance pairs 302. Data augmentation may include, but is not limited to, adding noise to the corresponding speech representation, manipulating timing (e.g., stretching), or adding reverberation.

[0042] The ASR model 200 receives as input a non-synthesized speech representation (x) 304 of a corresponding utterance as a sequence of features / vectors (e.g., acoustic frames 110 of FIG. 1 ) and generates as output, for each of a plurality of output steps, a first probability distribution 311 over possible non-synthesized speech recognition hypotheses (y) for the corresponding non-synthesized speech representation (x) 304 of the corresponding utterance. For simplicity, the term “non-synthesized speech recognition result 311” may be used to refer to the first probability distribution 311 over possible non-synthesized speech recognition hypotheses (y) for the corresponding non-synthesized speech representation (x) 304. The ASR model 200 also receives as input a VC synthetic speech representation (x) 304 of the corresponding utterance.

[0043]

number

[0044] as a sequence of features / vectors (e.g., acoustic frames 110 in FIG. 1 ), and as output, for each of a plurality of output steps, a corresponding VC synthesized speech representation of the corresponding utterance:

[0045]

number

[0046] y = y ( y ) ...

[0047]

number

[0048] may be used interchangeably to refer to a second probability distribution 312 over the possible synthetic speech recognition hypotheses (y). Similar to training a conventional end-to-end sequence-to-sequence ASR model, the training process 300 includes training a non-synthetic speech loss term (J) based on the ground truth transcription 320 and the non-synthetic speech recognition results 311 (y) of the corresponding non-synthetic speech representation (x) 304 of the corresponding utterance for output by the ASR model 200 at each of a plurality of output steps. real In the example shown, the training process 300 executes a supervised loss term module 340 configured to receive the ground truth transcription 320 and the first probability distribution 311, and generates a first supervised loss term including a non-synthesized speech loss term (J real (θ)) 342. The non-synthetic speech loss term (J real The first supervised loss term including (θ) 342 may be calculated by the following equation:

[0049]

number

[0050] Further, the training process 300, at each of a number of output steps, produces a ground truth transcription 320 and a corresponding VC synthetic speech representation of the corresponding utterance.

[0051]

number

[0052] VC synthesis speech loss term (J) based on the second probability distribution 312 for the possible VC synthesis speech recognition hypotheses (y). tts In the example shown, the training process 300 executes a supervised loss term module 340 configured to receive the ground truth transcription 320 and the second probability distribution 312, and generates a second supervised loss term (J(θ)) 344. tts (θ)) 344. The VC synthesis speech loss term (J tts The second supervised loss term, which includes (θ) 344, can be calculated by the following equation:

[0053]

number

[0054] where y*,z is the VC synthesized speech representation generated by transforming the unsynthesized speech representation 304 through the VC model 400 using the speaker embedding (z).

[0055]

number

[0056] Shows. The supervised loss term module 340 calculates the non-synthetic speech loss term (J real (θ))342 and VC synthesized speech loss term (J tts(θ)) 344 can be provided to the ASR model 200. For example, the training process 300 can train the ASR model 200 using a stochastic optimization algorithm, such as stochastic gradient descent, to train the ASR model 200 through backpropagation. Here, the stochastic optimization algorithm uses loss terms 342, 344 to define respective loss functions (e.g., cross-entropy loss functions) based on the difference between the actual outputs of the neural network (e.g., non-synthetic and synthetic speech recognition results 311, 312) and the desired outputs (e.g., ground truth transcriptions 320 of the corresponding utterances). For example, the loss functions are calculated for a batch of training examples and then differentiated with respect to each weight of the ASR model 200. In batch training, the non-synthetic speech loss term (J real (θ)) 342 may correspond to the average loss obtained for each batch of unsynthesized speech representations 304, and the VC synthetic speech loss term (J tts (θ) 344 may correspond to the average loss obtained for each batch of synthetic speech representations 306. Furthermore, the ASR model 200 may be trained in parallel on corresponding batches of non-synthetic speech representations 304 and VC-synthetic speech representations 306, such that the supervised loss term module 340 outputs corresponding non-synthetic and VC-synthetic speech loss terms 342, 344 in parallel.

[0057] In the illustrated example, at each of a plurality of output steps for each training utterance pair 302, the training process 300 calculates a consistent loss term (J) for the corresponding training utterance pair 302 based on a first probability distribution 311 for possible non-synthetic speech recognition hypotheses and a second probability distribution 312 for possible VC-synthetic speech recognition hypotheses. cons For example, the training process 300 may use a consistency loss term module 350 configured to receive, at each output step, the corresponding unsynthesized and VC-synthesized speech recognition results 311, 312 output by the ASR model 200 and determine a consistency loss term 352 for the corresponding training utterance pair 302 at the output step.

[0058] In some examples, the training process 300 may include calculating the Kullback-Leibler divergence (D) between a first probability distribution 311 for possible non-synthetic speech recognition hypotheses and a second probability distribution 312 for possible synthetic speech recognition hypotheses. KL ) to determine the consistent loss term 352. D KL The consistent loss term 352 based on can be expressed as:

[0059]

number

[0060] Here, the consistent loss term 352 determined for the training utterance pair 302 at each output step is independent of the accuracy of the ASR model (e.g., independent of the supervised loss terms 342, 344) and can therefore be used to update the parameters of the ASR model 200 to promote consistency between the non-synthetic and VC-synthetic speech representations of the same utterance. In batch training, the consistent loss term 352 may correspond to the average loss term obtained for the batch. In other words, the consistent loss term 352 enables the ASR model 200 to learn to behave in the same way, e.g., make consistent predictions for both non-synthetic speech (e.g., real / human speech) and VC-synthetic speech (e.g., synthetic speech generated by voice conversion) of the same training utterance, regardless of whether the training utterance belongs to non-synthetic or VC-synthetic speech. In the illustrated example, the training process 300 is configured to output corresponding non-synthetic and VC-synthetic speech loss terms 342, 344 from a supervised loss term module 340 and to output a consistent loss term 352 from a consistent loss term module 350 in parallel.

[0061] Advantageously, ASR model 200 can then be further trained on a large dataset of VC synthetic speech without having to rely on non-synthetic speech training examples to generate accurate speech recognition results for the human speech being inferred, or at least the number of non-synthetic speech training examples relative to the number of VC synthetic speech representations can be reduced. Thus, ASR model 200 can quickly recognize new accents / dialects and / or recognize different types of speech (e.g., atypical speech) without having to rely on transcribed non-synthetic speech with desired speech characteristics, because VC synthetic speech can be generated on the fly to train ASR model 200. In one example use case, the accuracy of ASR model 200 can dramatically improve speech recognition in natural languages ​​where transcribed non-synthetic speech is difficult to obtain by increasing the number of speech training examples from source non-synthetic speech and speech-to-synthetic synthetic speech converted by parallel speech-to-speech models from different speaker embeddings.

[0062] 4 is an example of a non-parallel speech conversion (VC) model 400. The term "non-parallel" means that the speech conversion model 400 does not require multiple speakers to speak the same utterance for use in training. The VC model 400 includes a content encoder 410 configured to encode source speech features 402, followed by a vector quantization variational autoencoder (VQ-VAE) layer 420 configured to bottleneck the information contained in the content encoder 410 and facilitate the removal / separation of style content from the source speech features 402, and a decoder 450 conditioned by a speaker identifier (ID) 403 to reconstruct the input speech features 410 as target speech features 480 for synthesized speech.

[0063] The VQ-VAE layer 420 may include a codebook size of 128 and may have two groups. The decoder 450 may include a simple speaker embedding 401 lookup based on speaker ID 403, where each speaker embedding 401 is concatenated with a latent representation of the logical content of the linguistic content 430 output from the VQ-VAE layer 420, tiled across time, forming a concatenated input 440 fed to the decoder 450. In some examples, the decoder 450 includes a network of bidirectional long short-term memory (LSTM) layers followed by a stack of residual convolutional layers with 4x temporal upsampling.

[0064] The VC model 400 may be applied to data augmentation applications, in which a latent representation of linguistic content 430 is extracted from a non-synthesized source speech sample 402 conveying a first locality / accent to generate new speech features 480 from a decoder 450 that convey different extensions of the same linguistic content 430 but convey different localities / accents in the synthesized speech. Here, each new speech feature 480 output from the decoder 450 may contain a locality / accent that is different from the initial locality / accent conveyed by the source speech sample 402 based on each speaker embedding 401 representing the speaker characteristics of each target speaker. An output including the ASR model 200 (or optionally a speaker identification model) may receive the output speech features 480 for training the ASR model 200 (or optionally a speaker identification model). Furthermore, the VC model can modify the source speaker's speech by transferring channel / noise to the resulting target speech features 480 to account for acoustic diversity when training the ASR model 200.

[0065] The content encoder 410 and the VQ-VAE layer 420 are configured to learn latent representations for each discrete time step (e.g., each frame) that capture the linguistic content 430 from the source speech sample 402. Specifically, the content encoder 410 may receive the source speech sample 402 and generate an initial latent representation 415 from the speech sample 402, and the VQ-VAE layer 420 applies an information bottleneck with a linear gradient to the initial latent representation 415 to capture necessary localized information, such as phoneme- or subword-like latent representations, and discard speaking style variations. Thus, the VQ-VAE layer 420 may output a sequence of latent variables that represent the linguistic content 130 from the input speech sample 402. The concatenator 435 may then concatenate the speaker embedding 401 of the target speaker (represented by speaker ID 403) and the latent representation of the linguistic content 430 into a concatenated input 440 that is provided to the decoder 450 for decoding into target speech features 480 that convey the same linguistic content but have speaker characteristics (e.g., accent / locality, speaking style, etc.) associated with the target speaker.

[0066] In some implementations, the content encoder 410 used by the VC model 400 includes a trained ASR encoder. In particular, using a trained ASR encoder as the content encoder 410 simplifies the training process for training the VC model 400, provides robustness to the VC model 400, and helps facilitate targeting of input speech samples in unrecognized languages ​​and domains. As used herein, a trained ASR encoder refers to an ASR encoder implemented by an ASR model to perform speech recognition, whereby the ASR encoder is trained with an ASR objective function that includes only the lexical content of the training data, and speaker identity is ignored by the trained ASR encoder.

[0067] Furthermore, implementing a pre-trained ASR encoder 410 as the content encoder 410 helps the target speech features 480 control accent transformation to convey the accent / accent associated with the target speaker ID 403 without retaining any accent / accent information from the source speech features 402. The speaker ID 403 inventory is associated with the set of speakers available during the training of the VC model performed on the dataset. Through evaluation, we found that training the content encoder from scratch using only VC training data (random initialization of weights) reflects the accent / accent information of the source speech features 402 in the target speech features 480. In other words, the trained ASR encoder is more reliable in removing accent information from the resulting encoded sequence (i.e., initial latent representation) 415, thereby allowing the speaker ID 403 to encode more accent information for enhanced accent control of the VC synthesis target speech features 480 output from the VC model. Accent transformation can then be used as a data augmentation technique to train the ASR model 200 to be more robust to speech from different accents / localities.

[0068] In some implementations, training a variational autoencoder (VAE) uses a variational autoencoder (VAE) to train the VC model 400 for the reconstruction task, such that the source and target speech features 402, 480 are the same. Here, a reconstruction loss module 470 corresponds to the reconstructed speech and calculates a reconstruction loss (L) between the source speech features 402, which serve as ground truth, and the target speech features 480 for use in optimizing the VC model 400 to minimize both the L1-norm squared distance and the L2-norm squared distance between the source speech features 402 and the target speech features 480. recon) 475. In particular, implementing a trained ASR encoder as the content encoder 410 results in freezing the parameters of the content encoder 410 while training the VQ-VAE layer 420, decoder 450, and speaker classifier 460. That is, freezing the parameters of the content encoder 410 trained with the ASR loss can encourage other components to learn different and better representations for the speech conversion task. Notably, the trained ASR encoder is more robust in noisy conditions.

[0069] The VQ-VAE layer 420 can be trained adversarially to ensure that speaker information (i.e., style content) is absent from the latent representation of the linguistic content 430 output by the VQ-VAE layer 420. In some examples, the adversarial training involves sampling random segments of length k of the encoded sequence (i.e., initial latent representation) 415 input to the VQ-VAE layer 420, and performing speaker classification on the randomly sampled segments using a speaker classifier 460. The speaker classifier 460 uses an inverse gradient to update the parameters of the content encoder 410 and the VQ-VAE layer 420 to generate an adversarial loss L adv 462. The speaker classifier 460 may enable inverse gradient backpropagation to maximize the speaker cross-entropy loss L CE 464.

[0070] The trained VC model 400 can further be used for augmentation in zero-shot cross-lingual contexts, whereby the speech conversion model 400 is trained on speech and speakers of one language but is used to augment speech in another language or to generate speech in an unknown language for which only speech data exists. While many applications of the VC model 400 train on speech from the same language used during inference, this is not required. In particular, the VC model 400 still works even when the input utterance is from another language. This simplifies the application of this technique to new, unknown languages ​​and domains. While there may be advantages to training a model-specific VC model, training of ASR models can still benefit from this augmentation technique using a pre-trained VC model from a different language.

[0071] Embodiments are further directed to optimizing VC training and recognizing when training is going well. The primary objective function for training a VC model is reconstruction loss. However, this compromises the model's ability to control the speaker characteristics of the resulting speech. The model 400 is augmented with speaker recognition accuracy—the accuracy of the speaker encoder based on the content encoder output 415 and the complexity (i.e., exponential entropy) of that output—which measures how diverse the encoder output 415 is. Ideal VC performance includes 1) low reconstruction loss 475 (signal is preserved), 2) low speaker classification accuracy (minimal speaker information remains in the encoder output), and 3) high VQ complexity (codebooks are often used—high diversity in the encoder output). While 1 and 2 are included during VC model training, being able to monitor all three independently allows for the identification of effective VC models without any guesswork or human evaluation. Subjective evaluation is only necessary once the model meets these objective criteria.

[0072] 5 is a flowchart of an example configuration of operations of a method 500 for training a speech recognition model 200 using a speech-conversion synthesized speech representation 306 of an utterance converted by a speech-conversion model 400 from a non-synthesized speech representation 304 of the same utterance. The method 500 may be performed on the remote computing device 201 of FIG. 1. For example, the remote computing device 201 includes data processing hardware 610 (FIG. 6) in communication with the data processing hardware, and memory hardware 620 (FIG. 6) that stores instructions that, when executed on the data processing hardware 610, cause the data processing hardware 610 to perform the operations of the method 500.

[0073] At operation 502, method 500 includes receiving a set of training utterances, each including a non-synthesized speech representation 304 of a corresponding utterance, and for each training utterance, generating a corresponding voice-conversion (VC) synthetic speech representation 306 by converting the non-synthesized speech representation 304 into a corresponding VC synthetic speech representation 306 of the corresponding utterance using VC model 400, where the non-synthesized speech representation 304 and the VC synthetic speech representation form a corresponding training utterance pair 302.

[0074] At each of a plurality of output steps for each training utterance pair 302 in the set of training utterance pairs, method 500 performs operations 504, 506, and 508. In operation 504, method 500 includes generating a first probability distribution 311 over possible non-synthesized speech recognition hypotheses (y) for a corresponding non-synthesized speech representation (x) 304 of the corresponding utterance for output by speech recognition model 200. In operation 506, method 500 generates a corresponding VC synthesized speech representation (x) 304 of the corresponding utterance for output by speech recognition model 200.

[0075]

number

[0076] In operation 508, method 500 generates a consistent loss term (J for the corresponding training utterance pairs 302 (also referred to as "training audio pairs") based on the first probability distribution 311 for the possible non-synthetic speech recognition hypotheses (y) and the second probability distribution 312 for the possible synthetic speech recognition hypotheses (y). cons (θ) 352.

[0077] At operation 510, the method includes updating parameters of the speech recognition model 200 based on the consistency loss term 352 determined at each of the multiple output steps for each training utterance pair 302 in the set of training utterance pairs. The speech conversion model 400 may include the non-parallel speech conversion model 400 described above with reference to FIG.

[0078] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0079] Non-transitory memory may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0080] 6 is a schematic diagram of an exemplary computing device 600 that can be used to implement the systems and methods described herein. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the scope of the invention as described and / or claimed in this document.

[0081] Computing device 600 includes a processor 610, memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and may be mounted on a common motherboard or implemented in other ways as needed. Processor 610 processes instructions for execution within computing device 600, including instructions stored in memory 620 or storage device 630, and can display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 680 connected to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used as needed, along with multiple memories and types of memory. Multiple computing devices 600 may also be connected, each performing a portion of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0082] The memory 620 stores non-transitory information within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transitory memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0083] Storage device 630 can provide mass storage for computing device 600. In some embodiments, storage device 630 is a computer-readable medium. In various different implementations, storage device 630 can be a device array including a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional embodiments, a computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 620, storage device 630, or memory on processor 610.

[0084] High-speed controller 640 manages bandwidth-intensive operations for computing device 600, while low-speed controller 660 manages low-bandwidth-intensive operations. This allocation of roles is merely exemplary. In some implementations, high-speed controller 640 is coupled to memory 620, display 680 (e.g., via a graphics processor or accelerator), and high-speed expansion port 650, which can accept various expansion cards (not shown). In some implementations, low-speed controller 660 is coupled to storage device 630 and low-speed expansion port 690. Low-speed expansion port 690 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and can connect to one or more input / output devices, such as a keyboard, pointing device, scanner, or network devices, such as a switch or router, via a network adapter or the like.

[0085] The computing device 600, as shown, can be implemented in many different forms. For example, it may be implemented as a standard server 600a, or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0086] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general-purpose, and may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device, coupled to receive data and instructions from, and transmit data and instructions to, the storage system.

[0087] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0088] The processes and logic flows described herein may be implemented by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose processors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operably connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0089] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device, such as a CRT (cathode ray tube) LCD (liquid crystal display) monitor or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, verbal, or tactile input. Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0090] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (500) that, when executed by data processing hardware (610), causes the data processing hardware (610) to perform operations, the operations comprising: receiving, by a speech conversion model (400), a plurality of training utterances, each of the training utterances comprising a non-synthesized speech representation (304) of a corresponding utterance; for each training utterance, generating a corresponding speech-conversion synthetic speech representation (306) by converting the non-synthesized speech representation (304) into a corresponding speech-conversion synthetic speech representation (306) of the corresponding utterance using the speech-conversion model (400), wherein the non-synthesized speech representation (304) and the speech-conversion synthetic speech representation (306) form a corresponding training utterance pair (302); In each of a plurality of output steps for each training utterance pair (302) of the plurality of training utterance pairs (302), generating a first probability distribution (311) over possible non-synthesized speech recognition hypotheses for the corresponding non-synthesized speech representation (304) of the corresponding utterance for output by a speech recognition model (200); generating a second probability distribution (312) over possible synthetic speech recognition hypotheses for the corresponding speech conversion synthetic speech representation (306) of the corresponding utterance for output by the speech recognition model (200); and determining a consistent loss term (352) for the corresponding training utterance pairs (302) based on a first probability distribution (311) for the possible non-synthetic speech recognition hypotheses and a second probability distribution (312) for the possible synthetic speech recognition hypotheses; and updating parameters of the speech recognition model (200) based on the consistent loss term (352) determined for each training utterance pair (302) at each of the plurality of output steps; The voice conversion model (400) A content encoder (410), receiving, as input, an input speech (402); generating, as output, an initial latent representation (415); the content encoder (410) configured to: A Vector Quantization-Variable Autoencoder (VQ-VAE) layer (420), comprising: receiving as input the initial latent representation (415); and generating, as an output, a latent representation of the linguistic content (430) extracted from the input speech, discarding speaking style variations of the input speech; the VQ-VAE layer (420) configured to perform 1. A computer-implemented method (500) comprising: a decoder (450) conditioned on speaker embeddings (401) and configured to generate output speech (480) based on a latent representation of the linguistic content (430).

2. The operation is In each of the plurality of output steps for each training utterance pair (302) of the training utterance pairs (302), generating a non-synthesized speech loss term (342) based on the first probability distribution (311) over possible non-synthesized speech recognition hypotheses for the corresponding non-synthesized speech representation (304) of the corresponding utterance and a ground truth transcription (320) of the corresponding utterance; and 10. The method of claim 1, further comprising: generating a synthetic speech loss term based on the second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance and the ground truth transcription of the corresponding utterance.

3. 3. The method of claim 2, wherein the parameters of the speech recognition model are updated based on the consistent loss term determined at each of the plurality of output steps for each training utterance pair of the training utterance pairs, independently of the non-synthesized speech loss term and the synthesized speech loss term generated at each of the plurality of output steps for each training utterance pair of the training utterance pairs.

4. the non-synthesized speech representation (304) of the corresponding utterance comprises speech spoken by a source speaker and conveys an accent / locality associated with the source speaker; The method (500) of any one of claims 1 to 3, wherein the speech conversion synthetic speech representation (306) of the corresponding utterance comprises synthetic speech representing the corresponding utterance conveying an accent / locality associated with a target speaker.

5. The method (500) of claim 4, wherein the accent / locality associated with the source speaker is different from the accent / locality associated with the target speaker.

6. 5. The method of claim 4, wherein the converted synthesized speech representation conveys the same linguistic content as the non-synthesized speech representation of the corresponding utterance.

7. The method (500) of any one of claims 1 to 3, wherein the speech conversion model (400) comprises a non-parallel speech conversion model (400).

8. 10. The method of claim 1, wherein the content encoder of the speech conversion model comprises an automatic speech recognition (ASR) encoder pre-trained on an ASR loss for speech recognition tasks.

9. 9. The method (500) of claim 8, wherein parameters of the pre-trained ASR encoder remain fixed during training of the VQ-VAE layer (420) and the decoder (450) of the speech conversion model (400).

10. 2. The method of claim 1, wherein the VQ-VAE layer is trained using a VQ loss based on a latent representation of the linguistic content generated at each time step, the VQ loss encouraging the VQ-VAE layer to minimize the distance between the output and the closest codebook.

11. The decoder (450) receiving as input the speaker embedding (401) and a latent representation of the linguistic content (430) for the input speech; and generating, as an output, the output speech (480) comprising a reconstruction of the input speech; The method (500) of claim 1, configured to:

12. A system (100), comprising: Data processing hardware (610), and Memory hardware (620) in communication with the data processing hardware (610) and storing a program, the program, when executed by the data processing hardware (610), causing the data processing hardware (610) to perform operations, the operations including: receiving, by a speech conversion model (400), a plurality of training utterances, each of the training utterances comprising a non-synthesized speech representation (304) of a corresponding utterance; for each training utterance, generating a corresponding speech-conversion synthetic speech representation (306) by converting the non-synthesized speech representation (304) into a corresponding speech-conversion synthetic speech representation of the corresponding utterance using the speech-conversion model (400), wherein the non-synthesized speech representation (304) and the speech-conversion synthetic speech representation (306) form a corresponding training utterance pair (302); In each of a plurality of output steps for each training utterance pair (302) of the plurality of training utterance pairs (302), generating a first probability distribution (311) over possible non-synthesized speech recognition hypotheses for the corresponding non-synthesized speech representation (304) of the corresponding utterance for output by a speech recognition model (200); generating a second probability distribution (312) over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance for output by the speech recognition model (200); and determining a consistent loss term (352) for the corresponding training utterance pairs (302) based on a first probability distribution (311) for the possible non-synthetic speech recognition hypotheses and a second probability distribution (312) for the possible synthetic speech recognition hypotheses; and updating parameters of the speech recognition model (200) based on the consistent loss term (352) determined for each training utterance pair (302) at each of the plurality of output steps; the memory hardware (620), Including, The voice conversion model (400) A content encoder (410), receiving, as input, an input speech (402); generating, as output, an initial latent representation (415); the content encoder (410) configured to: A Vector Quantization-Variable Autoencoder (VQ-VAE) layer (420), comprising: receiving as input the initial latent representation (415); and generating, as an output, a latent representation of the linguistic content (430) extracted from the input speech, which discards speaking style variations in the input speech; the VQ-VAE layer (420) configured to perform A system (100) comprising: a decoder (450) conditioned on speaker embeddings (401) and configured to generate output speech based on a latent representation of the linguistic content (430).

13. The operation is In each of the plurality of output steps for each training utterance pair (302) of the training utterance pairs (302), generating a non-synthesized speech loss term (342) based on the first probability distribution (311) over possible non-synthesized speech recognition hypotheses for the corresponding non-synthesized speech representation (304) of the corresponding utterance and a ground truth transcription (320) of the corresponding utterance; and 13. The system of claim 12, further comprising: generating a synthetic speech loss term based on the second probability distribution over possible synthetic speech recognition hypotheses for the corresponding speech conversion synthetic speech representation of the corresponding utterance and the ground truth transcription of the corresponding utterance.

14. 14. The system of claim 13, wherein the parameters of the speech recognition model are updated based on the consistent loss term determined at each of the plurality of output steps for each training utterance pair of the training utterance pairs, independently of the non-synthesized speech loss term and the synthesized speech loss term generated at each of the plurality of output steps for each training utterance pair of the training utterance pairs.

15. the non-synthesized speech representation (304) of the corresponding utterance comprises speech spoken by a source speaker and conveys an accent / locality associated with the source speaker; 15. The system (100) of claim 12, wherein the speech conversion synthetic speech representation (306) of the corresponding utterance comprises synthetic speech representing the corresponding utterance conveying an accent / locality associated with a target speaker.

16. 16. The system (100) of claim 15, wherein the accent / locality associated with the source speaker is different from the accent / locality associated with the target speaker.

17. 16. The system (100) of claim 15, wherein the speech-to-text synthesized speech representation (306) conveys the same linguistic content as the non-synthesized speech representation (304) of the corresponding utterance.

18. The system (100) of any one of claims 12 to 14, wherein the speech conversion model (400) comprises a non-parallel speech conversion model (400).

19. 13. The system of claim 12, wherein the content encoder of the speech conversion model comprises an automatic speech recognition (ASR) encoder pre-trained on ASR losses for speech recognition tasks.

20. 20. The system (100) of claim 19, wherein parameters of the pre-trained ASR encoder remain fixed during training of the VQ-VAE layer (420) and the decoder (450) of the speech conversion model (400).

21. 13. The system of claim 12, wherein the VQ-VAE layer is trained using a VQ loss based on a latent representation of the linguistic content generated at each time step, the VQ loss encouraging the VQ-VAE layer to minimize the distance between the output and a closest codebook.

22. The decoder (450) receiving as input the speaker embedding (401) and a latent representation of the linguistic content (430) for the input speech (402); and generating, as an output, the output speech (480) comprising a reconstruction of the input speech; The system (100) of claim 12, configured to:

Images