Using non-parallel voice conversion to train speech recognition models.

Abstract

The method includes receiving a set of training utterances, each of which includes a non-synthesized speech representation (304), and generating, for each training utterance, a corresponding speech-converted synthetic speech representation (306) by a speech-conversion model (400). The non-synthesized speech representation and the synthetic speech representation form a corresponding training utterance pair (302). At each of a plurality of output steps for each training utterance pair, the method also includes generating, for output by the speech recognition model (200), a first probability distribution (311) over possible non-synthesized speech recognition hypotheses for the non-synthesized speech representation, and a second probability distribution (312) over possible synthetic speech recognition hypotheses for the synthetic speech representation. The method also includes determining a consistency loss term for the corresponding training utterance pair based on the first and second probability distributions, and updating parameters of the speech recognition model based on the consistency loss term.

Description

[Technical field]

[0001] The present disclosure relates to using non-parallel speech conversion for speech recognition and joint recognition / conversion models. [Background technology]

[0002] Automatic speech recognition (ASR), the process of taking audio input and transcribing it into text, is a very important technology used in mobile and other devices. In general, automatic speech recognition attempts to accurately transcribe what a person said by taking audio input (e.g., a voice utterance) and transcribing the audio input into text. However, one challenge in developing deep learning-based ASR models is that the parameters of the ASR model tend to overfit the training data, resulting in the ASR model having difficulty generalizing to unseen data when the training data is not extensive enough. For example, speech recognition may be limited by the speaker identities and speaker characteristics that appear in the training data. As a result, training the ASR model with a larger training dataset improves the accuracy of the ASR model. Synthetic speech and / or data augmented speech may be incorporated to increase the amount of training data used to train the ASR model. Summary of the Invention

[0003] One aspect of the disclosure provides a computer-implemented method that, when executed by data processing hardware, causes the data processing hardware to perform operations including: receiving a set of training utterances, each of which includes a non-synthesized speech representation of a corresponding utterance; for each training utterance, generating a corresponding converted speech synthetic speech representation by converting the non-synthesized speech representation into a corresponding converted speech synthetic speech representation of the corresponding utterance using a speech conversion model. The non-synthesized speech representation and the synthetic speech representation form a corresponding training utterance pair. At each of a plurality of output steps for each training utterance pair of the set of training utterance pairs, the operations further include generating a first probability distribution for possible non-synthesized speech recognition hypotheses for the corresponding non-synthesized speech representation of the corresponding utterance for output by the speech recognition model, generating a second probability distribution for possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance for output by the speech recognition model, and determining a consistent loss term for the corresponding training utterance pair based on the first probability distribution for possible non-synthesized speech recognition hypotheses and the second probability distribution for possible non-synthesized speech recognition hypotheses. The operations also include updating parameters of the speech recognition model based on the consistency loss term determined at each of the multiple output steps for each training utterance pair of the set of training utterance pairs.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operations also include, at each of the multiple output steps for each training utterance pair of the set of training utterance pairs, generating a non-synthetic speech loss term based on a first probability distribution for possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation of the corresponding utterance and a ground truth transcription of the corresponding utterance, and generating a synthetic speech loss term based on a second probability distribution for possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance and a ground truth transcription of the corresponding utterance. In these implementations, parameters of the speech recognition model may be updated for each training utterance pair of the set of training utterance pairs based on the consistent loss term determined at each of the multiple output steps, independent of the non-synthetic speech loss term and the synthetic speech loss term generated at each of the multiple output steps for each training utterance pair of the set of training utterance pairs.

[0005] In some examples, the non-synthetic speech representation of the corresponding utterance includes speech spoken by a source speaker and conveys an accent / locality associated with the source speaker, and the speech-conversion synthetic speech representation of the corresponding utterance includes synthetic speech representing the corresponding utterance conveying an accent / locality associated with a target speaker, where the accent / locality associated with the source speaker may differ from the accent / locality associated with the target speaker. Optionally, the speech-conversion synthetic speech representation can convey the same linguistic content as the non-synthetic speech representation of the corresponding utterance. The speech-conversion model may include a non-parallel speech-conversion model.

[0006] In some implementations, the speech conversion model includes a content encoder, a vector quantization-variable autoencoder (VQ-VAE) layer, and a decoder. The content encoder is configured to receive an input speech as an input and generate an initial latent representation as an output. The VQ-VAE layer is configured to receive an initial latent representation as an input and generate a latent representation of linguistic content extracted from the input speech that discards speaking style variations of the input speech as an output. The decoder is configured to generate an output speech based on the latent representation of the linguistic content, conditioned based on the speaker embedding. The content encoder may include a pre-trained speech recognition (ASR) encoder that has been previously trained on an ASR loss for a speech recognition task, where parameters of the pre-trained ASR encoder remain fixed during training the VQ-VAE layer and the decoder of the speech conversion model. Additionally or alternatively, the VQ-VAE layer may be trained using a VQ loss based on the latent representation of the linguistic content generated at each time step, where the VQ loss drives the VQ-VAE layer to minimize the distance between the output and the closest codebook. In some implementations, the decoder is configured to receive as input an input speech and a latent representation of linguistic content for a speaker embedding, and to generate as output an output speech comprising a reconstruction of the input speech.

[0007] Another aspect of the present disclosure provides a system that includes data processing hardware and memory hardware in communication with the data processing hardware and storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving a set of training utterances, each of the training utterances including a non-synthesized speech representation of a corresponding utterance, and for each training utterance, generating a corresponding converted speech synthetic speech representation by converting the non-synthesized speech representation into a corresponding converted speech synthetic speech representation of the corresponding utterance using a speech conversion model, wherein the non-synthesized speech representation and the synthetic speech representation form a corresponding training utterance pair. At each of the plurality of output steps for each training utterance pair of the set of training utterance pairs, the operations further include generating, for output by the speech recognition model, a first probability distribution over possible non-synthesized speech recognition hypotheses for a corresponding non-synthesized speech representation of the corresponding utterance, generating, for output by the speech recognition model, a second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance, and determining a consistency loss term for the corresponding training utterance pair based on the first probability distribution over the possible non-synthesized speech recognition hypotheses and the second probability distribution over the possible non-synthesized speech recognition hypotheses. The operations also include updating parameters of the speech recognition model based on the consistency loss term determined at each of the plurality of output steps for each training utterance pair of the set of training utterance pairs.

[0008] This aspect may include one or more of the following optional features. In some implementations, the operations also include, at each of the multiple output steps for each training utterance pair of the set of training utterance pairs, generating a non-synthetic speech loss term based on a first probability distribution for possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation of the corresponding utterance and a ground truth transcription of the corresponding utterance, and generating a synthetic speech loss term based on a second probability distribution for possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance and a ground truth transcription of the corresponding utterance. In these implementations, parameters of the speech recognition model may be updated for each training utterance pair of the set of training utterance pairs based on the consistent loss term determined at each of the multiple output steps, independent of the non-synthetic speech loss term and the synthetic speech loss term generated at each of the multiple output steps for each training utterance pair of the set of training utterance pairs.

[0009] In some examples, the non-synthetic speech representation of the corresponding utterance includes speech spoken by a source speaker and conveys an accent / locality associated with the source speaker, and the speech-conversion synthetic speech representation of the corresponding utterance includes synthetic speech representing the corresponding utterance conveying an accent / locality associated with a target speaker, where the accent / locality associated with the source speaker may differ from the accent / locality associated with the target speaker. Optionally, the speech-conversion synthetic speech representation can convey the same linguistic content as the non-synthetic speech representation of the corresponding utterance. The speech-conversion model may include a non-parallel speech-conversion model.

[0010] In some implementations, the speech conversion model includes a content encoder, a vector quantization-variable autoencoder (VQ-VAE) layer, and a decoder. The content encoder is configured to receive an input speech as an input and generate an initial latent representation as an output. The VQ-VAE layer is configured to receive an initial latent representation as an input and generate a latent representation of linguistic content extracted from the input speech that discards speaking style variations of the input speech as an output. The decoder is configured to generate an output speech based on the latent representation of the linguistic content, conditioned based on the speaker embedding. The content encoder may include a pre-trained speech recognition (ASR) encoder that has been previously trained on an ASR loss for a speech recognition task, where parameters of the pre-trained ASR encoder remain fixed during training the VQ-VAE layer and the decoder of the speech conversion model. Additionally or alternatively, the VQ-VAE layer may be trained using a VQ loss based on the latent representation of the linguistic content generated at each time step, where the VQ loss drives the VQ-VAE layer to minimize the distance between the output and the closest codebook. In some implementations, the decoder is configured to receive as input an input speech and a latent representation of linguistic content for a speaker embedding, and to generate as output an output speech comprising a reconstruction of the input speech.

[0011] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0012] [Figure 1] 1 is a schematic diagram of an exemplary speech recognition system. [Figure 2A] FIG. 1 is a schematic diagram of a Recurrent Neural Network-Transducer (RNN-T) model architecture. [Figure 2B]FIG. 1 is a schematic diagram of an attention-based encoder-decoder (AED) model architecture. [Diagram 3] FIG. 1 is a schematic diagram of an example training process to facilitate a speech recognition model to learn consistent predictions for both unsynthesized speech and speech-to-text synthesized speech. [Figure 4] FIG. 2 is a schematic diagram of an exemplary non-parallel voice conversion model. [Diagram 5] 1 is a flowchart of an example arrangement of operations of a method for training a speech recognition model using a speech conversion synthetic speech representation of an utterance converted by a speech conversion model from a non-synthetic speech representation of the same utterance. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Like reference symbols in the various drawings indicate like elements. The introduction of sequence-to-sequence (Seq2Seq) models, which map audio to strings of characters, has made great advances in automatic speech recognition. At the same time, text-to-speech (TTS) or speech synthesis systems have successfully applied Seq2Seq models to obtain state-of-the-art natural and lifelike sounding synthetic speech that is indistinguishable from human speech by the human ear.

[0014] One challenge in developing deep learning-based ASR models is that the parameters of the ASR model tend to overfit the training data, resulting in the ASR model having difficulty generalizing to unseen data when the training data is not sufficiently extensive. As a result, training the ASR model on a larger training dataset improves the accuracy of the ASR model. For example, an ASR model can be trained on a training dataset that contains 10,000 hours or more of transcribed speech through the use of machine learning or other statistical methods. However, the performance of the ASR model degrades when the domain associated with the training data is different from the domain in which the ASR model is deployed during inference. For example, training an ASR model on speech transcribed in a domain related to video conferencing makes it less effective at missing and recognizing speech related to voice search queries, and vice versa. ASR models are further limited by the identity and speaker characteristics represented in the training data. For example, speech generally conveys speaker characteristics such as pitch, timbre, and prosody / style of the speaker who spoke the training utterance.

[0015] Embodiments herein are directed to using voice conversion to manipulate speaker characteristics (e.g., pitch, timbre, prosody, style) in spoken training utterances to expose ASR or automatic voice conversion (e.g., speech-to-speech) models to a more diverse range of speakers, thus improving the robustness of the resulting models. Embodiments herein further include using voice conversion extensions to modify the perceived accent of the input training utterances to improve speech recognition performance for accented speech or speech from other dialects / regions that are poorly represented / lacking in the available transcribed training data (i.e., spoken utterances paired with transcriptions).

[0016] In general, despite examples of state-of-the-art synthetic speech being indistinguishable from human speech, the use of synthetic speech has been shown to affect ASR training differently from human speech. This misalignment between synthetic and human speech is due to the mismatch of synthetic speech data from human speech data, which arises from the challenging one-to-many mapping problem that both text-to-speech and text-to-speech systems are trying to solve. That is, while the overall quality of available synthetic speech is very high, synthetic speech exhibits much less variation than human speech and minimal speech disfluencies. As a result, training an ASR model with only synthesized speech data presents difficulties in generalizing to the real speech utterances being inferred.

[0017] Embodiments herein are further directed to training a speech conversion model (i.e., an ASR model and / or a speech-to-speech (S2S) model) on real / human and synthesized speech representations of the same training utterances, and introducing a consistent loss term to promote consistent predictions (e.g., speech recognition hypotheses) for both the real / human and synthetic representations of the same training utterances. In essence, the consistent loss term between the human representation and the synthetic representation of the same utterance provides an aspect of unsupervised training by promoting the speech conversion model to behave consistently for both human and synthetic training utterances.

[0018] 1 illustrates an automatic speech recognition (ASR) system 100 that implements an ASR model 200 residing on a user device 102 of a user 104 and / or on a remote computing device 201 (e.g., one or more servers of a distributed system running in a cloud computing environment) that communicates with the user device 102. Although the user device 102 is illustrated as a mobile computing device (e.g., a smartphone), the user device 102 may correspond to any type of computing device, such as, without limitation, a tablet device, a laptop / desktop computer, a wearable device, a digital assistant device, a smart speaker / display, a smart appliance, an automobile infotainment system, or an Internet of Things (IoT) device, and includes data processing hardware 111 and memory hardware 113.

[0019] The user device 102 includes an audio subsystem 108 configured to receive utterances 106 produced by a user 104 (e.g., the user device 102 may include one or more microphones for recording the spoken utterances 106) and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 processable by the ASR system 100. In the illustrated example, the user speaks each utterance 106 in the natural language of English for the phrase "What's the weather like in New York City," and the audio subsystem 108 converts the utterance 106 into corresponding acoustic frames 110 for input to the ASR system 100. The ASR model 200 then receives the acoustic frames 110 corresponding to the utterances 106 as input and generates / predicts a corresponding transcription 120 (e.g., recognition result / hypothesis) of the utterance 106 as output. In the illustrated example, the user device 102 and / or the remote computing device 201 also execute a user interface generator 107 configured to present a representation of the transcription 120 of the utterance 106 to the user 104 of the user device 102. In some configurations, the transcription 120 output from the ASR system 100 is processed by a natural language understanding (NLU) module, e.g., executing on the user device 102 or the remote computing device 201, to execute user commands. Additionally or alternatively, a text-to-speech system (e.g., executing on any combination of the user device 102 or the remote computing device 201) can convert the transcription into synthetic speech and output audibly on another device. For example, the original utterance 106 corresponds to a message the user 104 sends to a friend in which the transcription 120 is converted into synthetic speech and output audibly so that the friend can hear the message conveyed in the original utterance 106.

[0020] 2A and 2B, the ASR model 200 may include an end-to-end (E2E) sequence-to-sequence model, such as a frame alignment-based transducer model 200a (FIG. 2A) or an attention-based encoder-decoder (AED) model 200b (FIG. 2B). The ASR model 200 may provide E2E speech recognition by integrating acoustic, pronunciation, and language models into a single neural network, and does not require a lexicon or separate text normalization component. Various structures and optimization mechanisms may improve accuracy and reduce model training time.

[0021] Referring to FIG. 2A, an exemplary frame alignment based transducer model 200a includes a recurrent neural network-transducer (RNN-T) model architecture that adheres to latency constraints associated with interactive applications. The RNN-T model 200a provides a small computational footprint and utilizes smaller memory requirements than conventional ASR architectures, making the RNN-T model architecture suitable for performing full speech recognition on the user device 102 (e.g., no communication with a remote server is required). The RNN-T model 200a includes an encoder network 210, a prediction network 220, and a joint network 230. The encoder network 210, which is broadly similar to an acoustic model (AM) of a conventional ASR system, includes a recurrent network of stacked long short-term memory (LSTM) layers. For example, the encoder receives a sequence of d-dimensional feature vectors (e.g., acoustic frames 110 (FIG. 1)) x=(x1, x2, ..., x T ) where:

[0022]

number

[0023] At each time step, we generate a higher-level representation of the feature set.

[0024]

number

[0025] It is expressed as: Similarly, the prediction network 220 is also an LSTM network, and like the language model (LM), it is the sequence of non-blank symbols y0, ..., y0 output by the final softmax layer 240. ui-1 A dense representation of

[0026]

number

[0027] Finally, using the RNN-T model architecture, the representations generated by the encoder and predictor networks 210, 220 are combined by a joint network 230. The joint network then

[0028]

number

[0029] , which is a distribution for the next output symbol. In other words, the joint network 230 generates a probability distribution for possible speech recognition hypotheses at each output step (e.g., time step), where a "possible speech recognition hypothesis" corresponds to a set of output labels each representing a symbol / character of a specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters of the English alphabet and one label representing a space. Thus, the joint network 230 may output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels. This set of values ​​may be a vector and may indicate a probability distribution for the set of output labels. In some cases, the output labels are graphemes (such as individual characters, potential punctuation marks, and other symbols), but the set of output labels is not limited to such. For example, the set of output labels may include parts of words and / or entire words in addition to or instead of graphemes. The output distribution of the joint network 230 may include posterior probability values ​​for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output of the joint network 230 (y i ) may contain 100 different probability values, one for each output label. The probability distributions may then be used to select candidate orthographic elements (e.g., graphemes, parts of words, and / or words) in a beam search process (e.g., by a softmax layer 240), assign scores, and determine the transcription 120.

[0030] The softmax layer 240 can use any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200a at the corresponding output step. In this way, the RNN-T model 200a does not make a conditional independence assumption, but rather the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model 200a assumes that the output symbols are independent of future acoustic frames 110. This allows the RNN-T model to be used in a streaming manner.

[0031] In some examples, the encoder network 210 of the RNN-T model 200a is composed of eight 2,048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. The prediction network 220 may have two 2,048-dimensional LSTM layers, each also followed by a 640-dimensional projection layer. Finally, the joint network 230 may also have 640 hidden units. Instead of LSTM layers, the encoder and / or prediction network 210, 220 may instead use a multi-head self-attention layer, such as a conformer layer, a transformer layer, or a lightweight convolutional (LConv) layer. The prediction network 220 may instead use an embedding lookup table instead of a network of LSTMs or multi-head self-attention layers. The softmax layer 240 may be composed of a set of consolidated word parts or graphemes generated using all unique word parts or graphemes of the multiple training datasets 301 (FIG. 3).

[0032] 2B, an exemplary AED model 200b associated with a Listen-Attend-And-Spell (LAS) model architecture provides a single neural network that includes a listener-encoder module 211, which is similar to a traditional acoustic model, an attendant module 221, which acts as an alignment model, and a decoder 231, which is similar to a language model in traditional systems. Specifically, the listener-encoder module 211 takes input features (e.g., acoustic frame 110 (FIG. 1)), x, and converts them into a higher-level feature representation h enc The encoded feature representation h enc This process of generating f can be done for each of several input frames, which represent different input time steps. These time steps are denoted below with the subscript u. Thus, given a set of frames {f1,f2,f3,...f u}, the corresponding set of encoded outputs {h1,h2,h3,...h u}.

[0033] The output of the listener encoder module 211 is passed to an attendant module 221, which, like a dynamic time warping (DTW) alignment module, aligns the next output symbol y i To predict h enc In some examples, the attender module 221 is referred to herein as an attender neural network or attender 221. The attender 221 determines which encoder features of the context output c for each of a plurality of output steps i. i For each context output vector c i For u, the attender 221 can compute attention based on the encoding of one or more input steps u, e.g., the encoding of the current input step and the encoding of a previous input step. For example, the attender 221 can compute attention based on the set of all encoder outputs of an utterance, e.g., the entire set {h1, h2, h3, ... h u} and attention context output ci The attention context vector may be a vector that represents a weighted summary of the current and previous encodings of a frame (e.g., part) of speech being recognized.

[0034] Finally, the output of the attender 221 is passed to a decoder 231, which outputs the attention context (e.g., a context vector or an attention distribution), c i , as well as the preceding prediction y i The decoder output is the embedding of the previous unit {y i-1 ,...,y0} and input x, the current subword unit y i Probability distribution for P(y i |y i-1 ,...,y0,x). Thus, at each output step, the decoder 231 generates a probability distribution over possible speech recognition hypotheses. Similar to the RNN-T model 200a discussed above with reference to FIG. 2A, a "possible speech recognition hypothesis" corresponds to a set of output symbols, each of which represents a symbol / character in the specified natural language.

[0035] Although not shown, the ASR model 200 may include a softmax layer that receives the output of the decoder 231. In some implementations, the softmax layer is separate from the decoder 231 and receives the output y i , and the output of the softmax layer is then used in a beam search process to select the orthogonal projection elements. In some implementations, the softmax layer is integrated with the decoder 231, so that the output y i represents the output of the softmax layer.

[0036] The decoder 231 and / or associated softmax layer may be trained to output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels. This set of values ​​may be a vector, indicating a probability distribution over the set of output labels. In some cases, the output labels are graphemes (such as individual characters, potential punctuation or other symbols, etc.), but the set of output labels is not so restricted. For example, the set of output labels may include parts of words and / or entire words in addition to or instead of graphemes. The output distribution of the decoder 231 and / or softmax layer may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the decoder output y i or output y i The output of a softmax layer that receives and processes can include 100 distinct probability values, one for each output label. The probability distributions can then be used to select candidate orthographic elements (e.g., graphemes, parts of words, and / or words) in a beam search process and assign scores to determine the transcription.

[0037] Referring to FIG. 3 , an exemplary training process 300 for facilitating the ASR model 200 to learn consistent predictions for both non-synthetic speech (e.g., real / human speech) and voice-conversion (VC) synthetic speech (e.g., synthesized speech) involves training a loss term (J) that is consistent between training utterance pairs 302 (also referred to as “training audio pairs”) of non-synthetic and VC synthetic speech representations 304, 306 of the same training utterance. consIn essence, the consistent loss terms between the non-synthetic and VC synthetic speech representations of the same training utterances provide an aspect of unsupervised training by encouraging ASR model 200 to behave consistently regardless of whether the training utterance belongs to non-synthetic or VC synthetic speech, and independent of the supervised loss terms between the ground truth transcriptions 320 and each of the non-synthetic and VC synthetic speech recognition hypotheses output by ASR model 200.

[0038] The training process 300 may be executed on the remote computing device 201 of FIG. 1. For example, the remote computing device 201 includes data processing hardware 610 (FIG. 6) and memory hardware 620 (FIG. 6) in communication with the data processing hardware and storing instructions that, when executed on the data processing hardware 610, cause the data processing hardware 610 to perform the operations of the training process 300. In the illustrated example, the training process 300 receives a set of training utterance pairs 302, each training utterance pair 302 including a non-synthesized speech representation 304 (e.g., human / real voice) of a corresponding utterance and a VC synthetic speech representation 306 of the same corresponding utterance. Each training utterance pair 302 is also associated with the same ground truth transcription 320 of the corresponding utterance. In some implementations, the non-synthesized speech representation 304 is initially paired with the ground truth transcription 320. For example, each non-synthesized speech representation 304 may be manually transcribed by a human listener. In these implementations, a non-parallel speech conversion (VC) model 400 may directly convert a non-synthesized speech representation 304 of a corresponding utterance into a VC synthetic speech representation 306, resulting in a corresponding training utterance pair 302. Here, the VC synthetic speech representation 306 is associated with the synthetic speech generated by the VC model 400 by converting the audio associated with the non-synthesized speech representation 304 into synthetic audio without having to perform any intermediate speech recognition. The VC model 400 may apply a speaker embedding 401, z, when converting the non-synthesized speech representation 304 to obtain a synthetic speech of a particular accent and / or speaking style and / or prosody associated with the speaker embedding 401. Thus, the VC model 400 may convert the non-synthesized speech representation 304 into any number of new VC synthetic speech representations 306 based on how many different unique speaker embeddings 401 are available during estimation. Here, the non-synthesized speech representation (x) 304 may be associated with one or more new VC synthetic speech representations 306 generated by the VC model 400.

[0039]

number

[0040] Each representation has the expectation that it is consistent with the unsynthesized speech representation (x) 304 associated with a ground truth transcription (y*) 320.

[0041] In some examples, the training process 300 applies data augmentation to at least one of the non-synthesized speech representations 304 or the VC-synthesized speech utterance representations 306 of at least one of the training utterance pairs 302. Data augmentation may include, but is not limited to, adding noise to the corresponding speech representation, manipulating timing (e.g., stretching), or adding reverberation.

[0042] The ASR model 200 receives as input a non-synthesized speech representation (x) 304 of a corresponding utterance as a sequence of features / vectors (e.g., acoustic frames 110 of FIG. 1 ) and generates as output, for each of a plurality of output steps, a first probability distribution 311 over possible non-synthesized speech recognition hypotheses (y) for the corresponding non-synthesized speech representation (x) 304 of the corresponding utterance. For simplicity, the term “non-synthesized speech recognition result 311” may be used to refer to the first probability distribution 311 over possible non-synthesized speech recognition hypotheses (y) for the corresponding non-synthesized speech representation (x) 304. The ASR model 200 also receives as input a VC synthetic speech representation (x) 304 of the corresponding utterance as a sequence of features / vectors (e.g., acoustic frames 110 of FIG. 1 ).

[0043]

number

[0044] as a sequence of features / vectors (e.g., acoustic frames 110 in FIG. 1 ) and as output, for each of a number of output steps, a corresponding VC synthetic speech representation of the corresponding utterance

[0045]

number

[0046] For simplicity, the term "synthetic speech recognition result 312" is used to refer to the corresponding synthetic speech representation

[0047]

number

[0048] may be used interchangeably to refer to a second probability distribution 312 over the possible synthetic speech recognition hypotheses (y). Similar to training a conventional end-to-end sequence-to-sequence ASR model, the training process 300 may include training a non-synthetic speech loss term (J) based on the ground truth transcription 320 and the non-synthetic speech recognition results 311 (y) of the corresponding non-synthetic speech representation (x) 304 of the corresponding utterance for output by the ASR model 200 at each of a number of output steps. real In the illustrated example, the training process 300 executes a supervised loss term module 340 configured to receive the ground truth transcription 320 and the first probability distribution 311 and generate a first supervised loss term including a non-synthetic speech loss term (J(θ)) 342. real (θ)) 342. The non-synthetic speech loss term (J real The first supervised loss term including (θ) 342 may be calculated by the following equation:

[0049]

number

[0050] Further, the training process 300, at each of a number of output steps, outputs a ground truth transcription 320 and a corresponding VC synthetic speech representation of the corresponding utterance.

[0051]

number

[0052] VC synthesis speech loss term (J) based on a second probability distribution 312 for the possible VC synthesis speech recognition hypotheses (y) tts In the illustrated example, the training process 300 executes a supervised loss term module 340 configured to receive the ground truth transcription 320 and the second probability distribution 312 and generate a VC synthesis speech loss term (J tts (θ)) 344. The VC synthesis speech loss term (J tts The second supervised loss term including (θ) 344 may be calculated by the following formula:

[0053]

number

[0054] where y*,z are the VC synthetic speech representations generated by transforming the unsynthesized speech representation 304 through the VC model 400 using the speaker embedding (z).

[0055]

number

[0056] Shows. The supervised loss term module 340 calculates the non-synthetic speech loss term (J real (θ))342 and VC synthesized speech loss term (J tts(θ)) 344 can be provided to the ASR model 200. For example, the training process 300 can train the ASR model 200 using a stochastic optimization algorithm, such as stochastic gradient descent, to train the ASR model 200 through backpropagation. Here, the stochastic optimization algorithm uses loss terms 342, 344 to define respective loss functions (e.g., cross-entropy loss functions) based on the difference between the actual output of the neural network (e.g., the results of non-synthetic and synthetic speech recognition 311, 312) and the desired output (e.g., the ground truth transcription 320 of the corresponding utterance). For example, the loss functions are calculated for a batch of training examples and then differentiated with respect to each weight of the ASR model 200. In batch training, the non-synthetic speech loss term (J real (θ) 342 may correspond to the average loss obtained for each batch of unsynthesized speech representations 304, and the VC synthetic speech loss term (J tts (θ) 344 may correspond to the average loss obtained for each batch of synthetic speech representations 306. Furthermore, the ASR model 200 may be trained in parallel on corresponding batches of non-synthetic speech representations 304 and VC synthetic speech representations 306, such that the supervised loss term module 340 outputs corresponding non-synthetic and VC synthetic speech loss terms 342, 344 in parallel.

[0057] In the illustrated example, at each of a number of output steps for each training utterance pair 302, the training process 300 calculates a consistent loss term (J cons For example, the training process 300 may use a consistency loss term module 350 configured to receive, at each output step, corresponding unsynthesized and VC-synthesized speech recognition results 311, 312 output by the ASR model 200, and determine, at the output step, a consistency loss term 352 for the corresponding training utterance pairs 302.

[0058] In some examples, the training process 300 may include computing the Kullback-Leibler divergence (D) between a first probability distribution 311 for possible non-synthetic speech recognition hypotheses and a second probability distribution 312 for possible synthetic speech recognition hypotheses. KL ) to determine the consistent loss term 352. D KL The consistent loss term 352 based on may be expressed as:

[0059]

number

[0060] Here, the consistent loss term 352 determined for the training utterance pair 302 at each output step is independent of the accuracy of the ASR model (e.g., independent of the supervised loss terms 342, 344) and can therefore be used to update the parameters of the ASR model 200 to promote consistency between the non-synthetic and VC synthetic speech representations of the same utterance. In batch training, the consistent loss term 352 can correspond to the average loss term obtained for the batch. In other words, the consistent loss term 352 enables the ASR model 200 to learn to behave in the same way, e.g., make consistent predictions for both non-synthetic (e.g., real / human) and VC synthetic speech (e.g., synthetic speech generated by voice conversion) of the same training utterance, regardless of whether the training utterance belongs to a non-synthetic or VC synthetic speech. In the illustrated example, the training process 300 is configured to output corresponding non-synthetic and VC-synthetic speech loss terms 342 , 344 from a supervised loss term module 340 and in parallel output a consistent loss term 352 from a consistent loss term module 350 .

[0061] Advantageously, the ASR model 200 can then be further trained on a large dataset of VC synthetic speech without having to rely on non-synthetic speech training examples to generate accurate speech recognition results for the human speech being inferred, or at least the number of non-synthetic speech training examples relative to the number of VC synthetic speech representations can be reduced. Thus, the ASR model 200 can quickly recognize new accents / dialects and / or recognize different types of speech (e.g., atypical speech) without having to rely on transcribed non-synthetic speech with desired speech characteristics, since the VC synthetic speech can be generated on the fly to train the ASR model 200. In one example use case, the accuracy of the ASR model 200 can be dramatically improved for speech recognition in natural languages ​​where transcribed non-synthetic speech is difficult to obtain by increasing the number of speech training examples from the source non-synthetic speech and the speech-to-synthetic synthetic speech converted by a parallel speech-to-synthetic model from different speaker embeddings.

[0062] 4 is an example of a non-parallel speech conversion (VC) model 400. The term "non-parallel" means that the speech conversion model 400 does not require multiple speakers to speak the same utterance for use in training. The VC model 400 includes a content encoder 410 configured to encode source speech features 402, followed by a vector quantization variational autoencoder (VQ-VAE) layer 420 configured to bottleneck the information contained in the content encoder 410 to facilitate removal / separation of style content from the source speech features 402, and a decoder 450 conditioned by a speaker identifier (ID) 403 to reconstruct the input speech features 410 as target speech features 480 of a synthetic speech.

[0063] The VQ-VAE layer 420 may include a codebook size of 128 and may have two groups. The decoder 450 may include a simple speaker embedding 401 lookup based on speaker ID 403, where each speaker embedding 401 is concatenated with a latent representation of the logical content of the linguistic content 430 output from the VQ-VAE layer 420, tiled across time, forming a concatenated input 440 that is fed to the decoder 450. In some examples, the decoder 450 includes a network of bidirectional long short-term memory (LSTM) layers followed by a stack of residual convolutional layers with 4x temporal upsampling.

[0064] The VC model 400 may be applied to data augmentation applications, in which a latent representation of linguistic content 430 is extracted from a non-synthesized source speech sample 402 conveying a first locality / accent to generate new speech features 480 conveying different extensions of the same linguistic content 430 but synthetic speech conveying different localities / accents from the decoder 450. Here, each new speech feature 480 output from the decoder 450 may include a locality / accent different from the initial locality / accent conveyed by the source speech sample 402 based on each speaker embedding 401 representing the speaker characteristics of each target speaker. The output including the ASR model 200 (or optionally a speaker identification model) may receive the output speech features 480 to train the ASR model 200 (or optionally a speaker identification model). Furthermore, the VC model may modify the source speaker's speech by transferring channel / noise to the resulting target speech features 480 for acoustic diversity in training the ASR model 200.

[0065] The content encoder 410 and the VQ-VAE layer 420 are configured to learn a discrete time-step (e.g., frame-by-frame) latent representation that captures the linguistic content 430 from the source speech sample 402. Specifically, the content encoder 410 may receive the source speech sample 402 and generate an initial latent representation 415 from the speech sample 402, and the VQ-VAE layer 420 applies an information bottleneck with a linear gradient to the initial latent representation 415 to capture necessary localized information, such as phoneme or sub-word-like latent representations, and discard speaking style variations. Thus, the VQ-VAE layer 420 may output a sequence of latent variables that represent the linguistic content 130 from the input speech sample 402. The concatenator 435 may then concatenate the speaker embedding 401 of the target speaker (represented by speaker ID 403) and the latent representation of the linguistic content 430 into a concatenated input 440 that is provided to the decoder 450 for decoding into target speech features 480 that convey the same linguistic content but have speaker characteristics associated with the target speaker (e.g., accent / locality, speaking style, etc.).

[0066] In some implementations, the content encoder 410 used by the VC model 400 includes a trained ASR encoder. In particular, using a trained ASR encoder as the content encoder 410 simplifies the training process for training the VC model 400, provides robustness to the VC model 400, and helps facilitate targeting of input speech samples in unrecognized languages ​​and domains. As used herein, a trained ASR encoder refers to an ASR encoder implemented by an ASR model to perform speech recognition, whereby the ASR encoder is trained with an ASR objective function to include only the lexical content of the training data, and the identity of the speaker is ignored by the trained ASR encoder.

[0067] Furthermore, implementing a pre-trained ASR encoder 410 as the content encoder 410 helps the target speech features 480 control accent transformation to convey the locality / accent associated with the target speaker ID 403 without retaining any locality / accent information from the source speech features 402. The inventory of speaker IDs 403 is associated with a set of speakers that were available during the training of the VC model performed on the dataset. Through evaluation, we found that training the content encoder from scratch using only VC training data (random initialization of weights) reflects the locality / accent information of the source speech features 402 in the target speech features 480. In other words, the trained ASR encoder is more reliable in removing accent information from the resulting encoding sequence (i.e., initial latent representation) 415, which allows the speaker ID 403 to encode more accent information to enhance accent control of the VC synthesis target speech features 480 output from the VC model. Accent transformation can then be used as a data augmentation technique to train the ASR model 200 to be more robust to speech from different accents / localities.

[0068] In some implementations, training a variational autoencoder (VAE) trains the VC model 400 on the reconstruction task using a variational autoencoder (VAE) such that the source and target speech features 402, 480 are the same. Here, the reconstruction loss module 470 corresponds to the reconstructed speech and calculates a reconstruction loss (L L ) between the source speech features 402, which serve as ground truth, and the target speech features 480 for use in optimizing the VC model 400 to minimize both the L1 norm squared distance and the L2 norm squared distance between the source speech features 402 and the target speech features 480. recon) 475. In particular, implementing the trained ASR encoder as the content encoder 410 results in freezing the parameters of the content encoder 410 while training the VQ-VAE layer 420, the decoder 450, and the speaker classifier 460. That is, freezing the parameters of the content encoder 410 trained with the ASR loss can encourage other components to learn different and better representations for the speech conversion task. Notably, the trained ASR encoder is more robust in noisy conditions.

[0069] The VQ-VAE layer 420 can be trained adversarially to ensure that no speaker information (i.e., style content) is present in the latent representation of the linguistic content 430 output by the VQ-VAE layer 420. In some examples, the adversarial training involves sampling random segments of length k of the encoded sequence (i.e., the initial latent representation) 415 input to the VQ-VAE layer 420, and performing speaker classification on the randomly sampled segments using a speaker classifier 460. The speaker classifier 460 uses the inverse gradient to update the parameters of the content encoder 410 and the VQ-VAE layer 420 to obtain the adversarial loss L adv 462. The speaker classifier 460 may enable inverse gradient backpropagation to maximize the speaker cross-entropy loss L CE 464.

[0070] The trained VC model 400 can be further used for augmentation in zero-shot cross-lingual contexts, whereby the speech conversion model 400 is trained on speech and speakers of one language, but is used to augment speech in another language, or to generate speech in an unknown language for which only speech data exists. Many applications of the VC model 400 train on speech from the same language used during inference, but this is not required. In particular, the VC model 400 still works even if the input utterance is from another language. This simplifies the application of this technique to new unknown languages ​​and domains. Although there may be advantages in training a VC model specific to the model, training of an ASR model can still benefit from this augmentation technique using a pre-trained VC model from a different language.

[0071] The embodiment is further directed to optimizing VC training and recognizing when training is going well. The main objective function for training a VC model is the reconstruction loss. However, this is questionable if the model can control the speaker characteristics of the resulting speech. The model 400 is extended with speaker recognition accuracy - the accuracy of the speaker encoder based on the content encoder output 415 and the complexity of its output (i.e., exponential entropy), thereby measuring how diverse the encoder output 415 is. Ideal VC performance includes 1) low reconstruction loss 475 (signal is preserved), 2) low speaker classification accuracy (minimal speaker information remains in the encoder output), and 3) high VQ complexity (codebooks are often used - high diversity of encoder output). While 1 and 2 are included during training of the VC model, being able to monitor all three independently allows for identifying effective VC models without any guesswork and human evaluation. Subjective evaluation is only necessary after the model meets these objective criteria.

[0072] 5 is a flowchart of an exemplary configuration of operations of a method 500 for training a speech recognition model 200 using a speech conversion synthetic speech representation 306 of an utterance converted by a speech conversion model 400 from a non-synthesized speech representation 304 of the same utterance. The method 500 may be performed on the remote computing device 201 of FIG. 1. For example, the remote computing device 201 includes data processing hardware 610 (FIG. 6) in communication with the data processing hardware, and memory hardware 620 (FIG. 6) that stores instructions that, when executed on the data processing hardware 610, cause the data processing hardware 610 to perform the operations of the method 500.

[0073] At operation 502, the method 500 includes receiving a set of training utterances, each of which includes a non-synthesized speech representation 304 of a corresponding utterance, and for each training utterance, generating a corresponding voice-conversion (VC) synthetic speech representation 306 by converting the non-synthesized speech representation 304 into a corresponding VC synthetic speech representation 306 of the corresponding utterance using the VC model 400, where the non-synthesized speech representation 304 and the VC synthetic speech representation form a corresponding training utterance pair 302.

[0074] At each of a number of output steps for each training utterance pair 302 in the set of training utterance pairs, the method 500 performs operations 504, 506, and 508. In operation 504, the method 500 includes generating a first probability distribution 311 over possible non-synthesized speech recognition hypotheses (y) for a corresponding non-synthesized speech representation (x) 304 of the corresponding utterance, for output by the speech recognition model 200. In operation 506, the method 500 generates a corresponding VC synthetic speech representation (y) of the corresponding utterance, for output by the speech recognition model 200.

[0075]

number

[0076] In operation 508, the method 500 generates a consistency loss term (J for the corresponding training utterance pairs 302 (also referred to as “training audio pairs”) based on the first probability distribution 311 for the possible non-synthetic speech recognition hypotheses (y) and the second probability distribution 312 for the possible synthetic speech recognition hypotheses (y). cons (θ) 352.

[0077] At operation 510, the method includes updating parameters of the speech recognition model 200 based on the consistency loss term 352 determined at each of the multiple output steps for each training utterance pair 302 of the set of training utterance pairs. The speech conversion model 400 may include the non-parallel speech conversion model 400 described above with reference to FIG.

[0078] A software application (i.e., software resource) may be referred to as computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0079] Non-transient memory may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0080] 6 is a schematic diagram of an exemplary computing device 600 that can be used to implement the systems and methods described herein. Computing device 600 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0081] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be mounted on a common motherboard or otherwise implemented as desired. Processor 610 may process instructions for execution within computing device 600, including instructions stored in memory 620 or storage device 630, to display graphical information for a graphical user interface (GUI) on an external input / output device such as a display 680 connected to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used as desired, along with multiple memories and types of memories. Also, multiple computing devices 600 may be connected, with each device performing some of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0082] The memory 620 stores non-transient information within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0083] The storage device 630 can provide mass storage for the computing device 600. In some embodiments, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 can be a device array, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional embodiments, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0084] The high-speed controller 640 manages the bandwidth-intensive operations of the computing device 600, and the low-speed controller 660 manages the low-bandwidth intensive operations. Such an allocation of roles is merely an example. In some implementations, the high-speed controller 640 is coupled to the memory 620, the display 680 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and the low-speed expansion port 690. The low-speed expansion port 690 may include various communication ports (USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and can connect, via a network adapter or the like, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0085] The computing device 600 can be implemented in many different forms, as shown in the figure: for example, it can be implemented as a standard server 600a, or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0086] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general purpose and may include implementations in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0087] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0088] The processes and logic flows described herein can be implemented by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be implemented by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, for example, both general purpose and special purpose processors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0089] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device, such as a CRT (cathode ray tube) LCD (liquid crystal display) monitor, or a touch screen, for displaying information to the user, and optionally a keyboard and a pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, spoken language, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0090] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (500) that, when executed by data processing hardware (610), causes the data processing hardware (610) to perform operations, the operations comprising: receiving a set of a plurality of training utterances, each of the training utterances including a non-synthesized speech representation (304) of a corresponding utterance; for each training utterance, generating a corresponding speech-conversion synthetic speech representation (306) by converting the non-synthesized speech representation (304) into a corresponding speech-conversion synthetic speech representation (306) of the corresponding utterance using a speech-conversion model (400), wherein the non-synthesized speech representation (304) and the speech-conversion synthetic speech representation (306) form a corresponding training utterance pair (302); In each of a plurality of output steps for each training utterance pair (302) of the set of a plurality of training utterance pairs (302), generating a first probability distribution (311) over possible non-synthesized speech recognition hypotheses for the corresponding non-synthesized speech representation (304) of the corresponding utterance, for output by a speech recognition model (200); generating a second probability distribution (312) over possible synthetic speech recognition hypotheses for the corresponding speech conversion synthetic speech representation (306) of the corresponding utterance for output by the speech recognition model (200); and determining a consistent loss term (352) for the corresponding training utterance pairs (302) based on a first probability distribution (311) for the possible non-synthetic speech recognition hypotheses and a second probability distribution (312) for the possible non-synthetic speech recognition hypotheses; and updating parameters of the speech recognition model (200) based on the consistent loss term (352) determined at each of the plurality of output steps for each training utterance pair (302) of the set of training utterance pairs (302); A computer-implemented method (500) comprising:

2. The operation includes: At each of the plurality of output steps for each training utterance pair (302) of the set of training utterance pairs (302), generating a non-synthesized speech loss term (342) based on the first probability distribution (311) over possible non-synthesized speech recognition hypotheses for the corresponding non-synthesized speech representation (304) of the corresponding utterance and a ground truth transcription (320) of the corresponding utterance; and generating a synthetic speech loss term (342) based on the second probability distribution (312) over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance and the ground truth transcription (320) of the corresponding utterance; The method of claim 1 , further comprising:

3. 3. The method of claim 2, wherein the parameters of the speech recognition model are updated based on the consistent loss term determined at each of the multiple output steps for each training utterance pair of the set of training utterance pairs, independently of the non-synthesized speech loss term and the synthesized speech loss term generated at each of the multiple output steps for each training utterance pair of the set of training utterance pairs.

4. the non-synthesized speech representation (304) of the corresponding utterance comprises speech spoken by a source speaker and conveys an accent / locality associated with the source speaker; The method (500) of any one of claims 1 to 3, wherein the speech conversion synthetic speech representation (306) of the corresponding utterance comprises synthetic speech representing the corresponding utterance conveying an accent / locality associated with a target speaker.

5. The method (500) of claim 4, wherein the accent / locality associated with the source speaker is different from the accent / locality associated with the target speaker.

6. 6. The method (500) of claim 4 or 5, wherein the converted synthetic speech representation (306) conveys the same linguistic content as the non-synthesized speech representation (304) of the corresponding utterance.

7. The method (500) of any one of claims 1 to 4, wherein the speech conversion model (400) comprises a non-parallel speech conversion model (400).

8. The voice conversion model (400) A content encoder (410), receiving, as input, an input speech (402); generating, as output, an initial latent representation (415); the content encoder (410), configured to A vector quantization-variable autoencoder (VQ-VAE) layer (420), comprising: receiving, as input, the initial latent representation (415); and generating, as an output, a latent representation of the linguistic content (430) extracted from the input speech, discarding speaking style variations of the input speech; the VQ-VAE layer (420) configured to perform a decoder (450) conditioned on the speaker embeddings (401) and configured to generate output speech (480) based on a latent representation of the linguistic content (430); The method (500) of any one of claims 1 to 4 and 7, comprising:

9. 10. The method of claim 8, wherein the content encoder of the speech conversion model comprises a pre-trained automatic speech recognition (ASR) encoder that has been previously trained on ASR losses for a speech recognition task.

10. 10. The method (500) of claim 9, wherein parameters of the pre-trained ASR encoder remain fixed during training of the VQ-VAE layer (420) and the decoder (450) of the speech conversion model (400).

11. The method (500) of any one of claims 8 to 10, wherein the VQ-VAE layer (420) is trained using a VQ loss based on a latent representation of the linguistic content (430) generated at each time step, the VQ loss encouraging the VQ-VAE layer (420) to minimize the distance between the output and the closest codebook.

12. The decoder (450): receiving as input the speaker embeddings (401) and a latent representation of the linguistic content (430) for the input speech; and generating, as an output, the output speech (480) comprising a reconstruction of the input speech; The method (500) of any one of claims 8 to 11, configured to:

13. A system (100), comprising: Data processing hardware (610); and memory hardware (620) in communication with the data processing hardware (610) and storing instructions that, when executed by the data processing hardware (610), cause the data processing hardware (610) to perform operations, the operations including: receiving a set of a plurality of training utterances, each of the training utterances including a non-synthesized speech representation (304) of a corresponding utterance; for each training utterance, generating a corresponding speech-conversion synthetic speech representation (306) by converting the non-synthesized speech representation (304) into a corresponding speech-conversion synthetic speech representation of the corresponding utterance using a speech-conversion model (400), wherein the non-synthesized speech representation (304) and the speech-conversion synthetic speech representation (306) form a corresponding training utterance pair (302); At each of a plurality of output steps for each training utterance pair (302) of the set of a plurality of training utterance pairs (302), generating a first probability distribution (311) over possible non-synthesized speech recognition hypotheses for the corresponding non-synthesized speech representation (304) of the corresponding utterance, for output by a speech recognition model (200); generating a second probability distribution (312) over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance, for output by the speech recognition model (200); and determining a consistent loss term (352) for the corresponding training utterance pairs (302) based on a first probability distribution (311) for the possible non-synthetic speech recognition hypotheses and a second probability distribution (312) for the possible non-synthetic speech recognition hypotheses; and updating parameters of the speech recognition model (200) based on the consistent loss term (352) determined at each of the plurality of output steps for each training utterance pair (302) of the set of training utterance pairs (302); the memory hardware (620), A system (100).

14. The operation includes: At each of the plurality of output steps for each training utterance pair (302) of the set of training utterance pairs (302), generating a non-synthesized speech loss term (342) based on the first probability distribution (311) over possible non-synthesized speech recognition hypotheses for the corresponding non-synthesized speech representation (304) of the corresponding utterance and a ground truth transcription (320) of the corresponding utterance; and generating a synthetic speech loss term (344) based on the second probability distribution (312) over possible synthetic speech recognition hypotheses for the corresponding speech conversion synthetic speech representation (306) of the corresponding utterance and the ground truth transcription (320) of the corresponding utterance; The system (100) of claim 13, further comprising:

15. 15. The system of claim 14, wherein the parameters of the speech recognition model are updated based on the consistent loss term determined at each of the multiple output steps for each training utterance pair of the set of training utterance pairs, independent of the non-synthesized speech loss term and the synthesized speech loss term generated at each of the multiple output steps for each training utterance pair of the set of training utterance pairs.

16. the non-synthesized speech representation (304) of the corresponding utterance comprises speech spoken by a source speaker and conveys an accent / locality associated with the source speaker; The system (100) of any one of claims 13 to 15, wherein the voice conversion synthetic speech representation (306) of the corresponding utterance comprises synthetic speech representing the corresponding utterance conveying an accent / locality associated with a target speaker.

17. 17. The system (100) of claim 16, wherein the accent / locality associated with the source speaker is different from the accent / locality associated with the target speaker.

18. 18. The system (100) of claim 16 or 17, wherein the converted synthesized speech representation (306) conveys the same linguistic content as the non-synthesized speech representation (304) of the corresponding utterance.

19. The system (100) of any one of claims 13 to 15, wherein the speech conversion model (400) comprises a non-parallel speech conversion model (400).

20. The voice conversion model (400) A content encoder (410), receiving, as input, an input speech (402); generating, as output, an initial latent representation (415); the content encoder (410), configured to A vector quantization-variable autoencoder (VQ-VAE) layer (420), comprising: receiving, as input, the initial latent representation (415); and generating, as an output, a latent representation of the linguistic content (430) extracted from the input speech, discarding speaking style variations in the input speech; The VQ-VAE layer (420) is configured to perform a decoder (450) conditioned on the speaker embeddings (401) and configured to generate output speech based on a latent representation of the linguistic content (430); The system (100) according to any one of claims 13 to 16 and 19, comprising:

21. 21. The system of claim 20, wherein the content encoder of the speech conversion model comprises a pre-trained automatic speech recognition (ASR) encoder that has been previously trained on ASR losses for a speech recognition task.

22. 22. The system (100) of claim 21, wherein parameters of the pre-trained ASR encoder remain fixed during training of the VQ-VAE layer (420) and the decoder (450) of the speech conversion model (400).

23. 23. The system (100) of any one of claims 20 to 22, wherein the VQ-VAE layer (420) is trained using a VQ loss based on a latent representation of the linguistic content (430) generated at each time step, the VQ loss encouraging the VQ-VAE layer (420) to minimize the distance between the output and the closest codebook.

24. The decoder (450): receiving as input the speaker embeddings (401) and a latent representation of the linguistic content (430) for the input speech (402); and generating, as an output, the output speech (480) comprising a reconstruction of the input speech; The system (100) according to any one of claims 20 to 23, configured to:

Images