Facilitating the Use of Text and Speech in ASR Pre-training with Consistency and Contrastive Losses

Abstract

The method (600) includes receiving training data including a non-spoken text utterance (320), an untranscribed non-synthetic speech utterance (306), and a transcribed non-synthetic speech utterance (304). Each non-spoken text utterance is not paired with any corresponding spoken utterance of the non-synthetic speech. Each untranscribed non-synthetic speech utterance is not paired with a corresponding transcription. Each transcribed non-synthetic speech utterance is paired with a corresponding transcription (302). The method also includes generating a corresponding synthetic speech representation (332) of each non-spoken text utterance of the received training data using a text-to-speech model (330). The method also includes pre-training an audio encoder (210) based on the generated synthetic speech representations for the non-spoken text utterance, the untranscribed non-synthetic speech utterance, and the transcribed non-synthetic speech utterance to teach the audio encoder to jointly learn the shared speech and text representations.

Description

[Technical Field]

[0001] The present disclosure relates to advancing the use of text and speech in automatic speech recognition (ASR) pre-training through consistency and contrastive losses. [Background technology]

[0002] Automatic speech recognition (ASR), the process of taking audio input and transcribing it into text, has become a very important technology used in mobile devices and other devices. Generally, automatic speech recognition attempts to provide an accurate transcription of what a person said by taking audio input (e.g., a voice utterance) and transcribing the audio input into text. Based on the ongoing development of deep neural networks, modern ASR models continue to improve in both accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay between a user's utterance and the transcription). However, one challenge in developing deep learning-based ASR models is that the parameters of the ASR model tend to overfit to the training data, which causes the ASR model to have difficulty generalizing to unseen data when the training data is not sufficiently large. As a result, training an ASR model with a relatively large training dataset improves the accuracy of the ASR model. To increase the amount of training data used to train an ASR model, synthetic speech and / or data-augmented speech can be incorporated. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for pre-training an audio encoder to jointly learn a shared representation of speech and text. The operations include receiving training data including a non-spoken text utterance, an untranscribed non-synthetic speech utterance, and a transcribed non-synthetic speech utterance. Each non-spoken text utterance is not paired with any corresponding spoken utterance of the non-synthetic speech. Each untranscribed non-synthetic speech utterance is not paired with a corresponding transcription. Each transcribed non-synthetic speech utterance is paired with a corresponding transcription. The operations also include generating a corresponding synthetic speech representation for each non-spoken text utterance in the received training data using a text-to-speech model. The operations also include pre-training the audio encoder based on the synthetic speech representations generated for the non-spoken text utterance, the untranscribed non-synthetic speech utterance, and the transcribed non-synthetic speech utterance to teach the audio encoder to jointly learn the shared speech and text representation.

[0004] Implementations of this disclosure may include one or more of the following optional features: In some implementations, the audio encoder includes a stack of self-attention layers, each including a multi-head self-attention mechanism; In some examples, pre-training the audio encoder includes: for each untranscribed, non-synthetic speech utterance, generating a corresponding coded representation of the untranscribed speech representation and pre-training the audio encoder based on a contrastive loss applied to the corresponding coded representation of the untranscribed, non-synthetic speech utterance; for each synthetic speech representation, generating a corresponding coded representation of the synthetic speech representation and pre-training the audio encoder based on a contrastive loss applied to the corresponding coded representation of the synthetic speech representation; and for each transcribed non-synthetic speech utterance, generating a corresponding coded representation of the transcribed non-synthetic speech utterance and pre-training the audio encoder based on a contrastive loss applied to the corresponding coded representation of the untranscribed, non-synthetic speech utterance.

[0005] In some implementations, pre-training the audio encoder includes, at each of a plurality of time steps of each synthetic speech representation, using an auxiliary decoder, generating a first probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation, determining a synthetic speech loss term based on the first probability distribution over the possible synthetic speech recognition hypotheses and a non-spoken text utterance corresponding to the corresponding synthetic speech representation, pre-training the audio encoder based on the synthetic speech loss term, at each of a plurality of time steps of each transcribed non-synthetic speech utterance, using the auxiliary decoder, generating a second probability distribution over the possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance, determining a non-synthetic speech loss term based on the second probability distribution over the possible non-synthetic speech recognition hypotheses and a corresponding transcription paired with the transcribed non-synthetic speech utterance, and pre-training the audio encoder based on the non-synthetic speech loss term. Here, a first probability distribution over the possible synthetic speech recognition hypotheses includes one of the possible phoneme labels or the possible wordpiece labels, and a second probability distribution over the possible non-synthetic speech recognition hypotheses includes one of the possible phoneme labels or the possible wordpiece labels.

[0006] In these implementations, pre-training the audio encoder may further include: at each of a plurality of time steps of each synthetic speech representation, using another auxiliary decoder, generating a third probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation, including the other of the possible phoneme labels or the possible wordpiece labels; determining another synthetic speech loss term based on the third probability distribution over the possible synthetic speech recognition hypotheses and a non-spoken text utterance corresponding to the corresponding synthetic speech representation; pre-training the audio encoder based on the other synthetic speech loss term; at each of a plurality of time steps of each transcribed non-synthetic speech utterance, using the other auxiliary decoder, generating a fourth probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance, including the other of the possible phoneme labels or the possible wordpiece labels; determining another non-synthetic speech loss term based on the fourth probability distribution over the possible non-synthetic speech recognition hypotheses and a corresponding transcription paired with the transcribed non-synthetic speech utterance; and pre-training the audio encoder based on the non-synthetic speech loss term. The auxiliary decoder includes one of a Connection Temporal Classification (CTC) decoder, a Listen Attend Spell (LAS) decoder, or a Recurrent Neural Network-Transducer (RNN-T) decoder.

[0007] In some examples, the operations further include obtaining a set of training utterance pairs, each including a corresponding one of the transcribed non-synthetic speech utterances of the received training data and a synthetic speech representation of the corresponding pair of transcribed non-synthetic speech utterances generated by performing text-to-speech conversion on the corresponding transcription paired with the transcribed non-synthetic speech utterance by the text-to-speech model. In these examples, pre-training the audio encoder includes, at each of a plurality of output steps for each training utterance pair in the set of training utterance pairs, generating, using an auxiliary decoder, a first probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance, generating, using the auxiliary decoder, a second probability distribution over possible synthetic speech recognition hypotheses for the corresponding pair of synthetic speech representations, determining a consistency loss term for the corresponding training utterance pair based on the first probability distribution over the possible non-synthetic speech recognition hypotheses and the second probability distribution over the possible synthetic speech recognition hypotheses, and pre-training the audio encoder based on the non-synthetic speech loss term. The operations may further include augmenting one or more of the synthetic speech representations before pre-training the audio encoder based on the synthetic speech representations.

[0008] In some implementations, the non-spoken text utterances are generated and / or selected using one or more language models. In some examples, the non-spoken text utterances are generated using a background language model and a domain language model trained on transcribed speech utterances associated with the target domain. The operations may further include pre-training an audio encoder and then fine-tuning the pre-trained encoder on the transcribed speech utterances.

[0009] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving training data including a non-speech text utterance, an untranscribed non-synthetic speech utterance, and a transcribed non-synthetic speech utterance. Each non-speech text utterance is not paired with any corresponding speech utterance of the non-synthetic speech. Each untranscribed non-synthetic speech utterance is not paired with a corresponding transcription. Each transcribed non-synthetic speech utterance is paired with a corresponding transcription. The operations also include generating a corresponding synthetic speech representation of each non-speech text utterance in the received training data using a text-to-speech model. The operations also include pre-training an audio encoder based on the generated synthetic speech representations for the non-speech text utterance, the untranscribed non-synthetic speech utterance, and the transcribed non-synthetic speech utterance to teach the audio encoder to jointly learn the shared speech and text representations.

[0010] Implementations of this disclosure may include one or more of the following optional features: In some implementations, the audio encoder includes a stack of self-attention layers each including a multi-head self-attention mechanism; In some examples, pre-training the audio encoder includes: for each untranscribed, non-synthetic speech utterance, generating a corresponding coded representation of the untranscribed speech representation and pre-training the audio encoder based on a contrastive loss applied to the corresponding coded representation of the untranscribed, non-synthetic speech utterance; for each synthetic speech representation, generating a corresponding coded representation of the synthetic speech representation and pre-training the audio encoder based on a contrastive loss applied to the corresponding coded representation of the synthetic speech representation; and for each transcribed non-synthetic speech utterance, generating a corresponding coded representation of the transcribed non-synthetic speech utterance and pre-training the audio encoder based on a contrastive loss applied to the corresponding coded representation of the transcribed, non-synthetic speech utterance.

[0011] In some implementations, pre-training the audio encoder includes, at each of a plurality of time steps of each synthetic speech representation, using an auxiliary decoder, generating a first probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation, determining a synthetic speech loss term based on the first probability distribution over the possible synthetic speech recognition hypotheses and a non-spoken text utterance corresponding to the corresponding synthetic speech representation, pre-training the audio encoder based on the synthetic speech loss term, at each of a plurality of time steps of each transcribed non-synthetic speech utterance, using the auxiliary decoder, generating a second probability distribution over the possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance, determining a non-synthetic speech loss term based on the second probability distribution over the possible non-synthetic speech recognition hypotheses and a corresponding transcription paired with the transcribed non-synthetic speech utterance, and pre-training the audio encoder based on the non-synthetic speech loss term. Here, a first probability distribution over the possible synthetic speech recognition hypotheses includes one of the possible phoneme labels or the possible wordpiece labels, and a second probability distribution over the possible non-synthetic speech recognition hypotheses includes one of the possible phoneme labels or the possible wordpiece labels.

[0012] In these implementations, pre-training the audio encoder may further include: at each of a plurality of time steps of each synthetic speech representation, using another auxiliary decoder, generating a third probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation, including the other of the possible phoneme labels or the possible wordpiece labels; determining another synthetic speech loss term based on the third probability distribution over the possible synthetic speech recognition hypotheses and a non-spoken text utterance corresponding to the corresponding synthetic speech representation; pre-training the audio encoder based on the other synthetic speech loss term; at each of a plurality of time steps of each transcribed non-synthetic speech utterance, using the other auxiliary decoder, generating a fourth probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance, including the other of the possible phoneme labels or the possible wordpiece labels; determining another non-synthetic speech loss term based on the fourth probability distribution over the possible non-synthetic speech recognition hypotheses and a corresponding transcription paired with the transcribed non-synthetic speech utterance; and pre-training the audio encoder based on the non-synthetic speech loss term. The auxiliary decoder includes one of a Connection Temporal Classification (CTC) decoder, a Listen Attend Spell (LAS) decoder, or a Recurrent Neural Network-Transducer (RNN-T) decoder.

[0013] In some examples, the operations further include obtaining a set of training utterance pairs, each including a corresponding one of the transcribed non-synthetic speech utterances of the received training data and a synthetic speech representation of the corresponding pair of transcribed non-synthetic speech utterances generated by performing text-to-speech conversion on the corresponding transcription paired with the transcribed non-synthetic speech utterance by the text-to-speech model. In these examples, pre-training the audio encoder includes, at each of a plurality of output steps for each training utterance pair in the set of training utterance pairs, generating, using an auxiliary decoder, a first probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance, generating, using the auxiliary decoder, a second probability distribution over possible synthetic speech recognition hypotheses for the corresponding pair of synthetic speech representations, determining a consistency loss term for the corresponding training utterance pair based on the first probability distribution over the possible non-synthetic speech recognition hypotheses and the second probability distribution over the possible synthetic speech recognition hypotheses, and pre-training the audio encoder based on the non-synthetic speech loss term. The operations may further include augmenting one or more of the synthetic speech representations before pre-training the audio encoder based on the synthetic speech representations.

[0014] In some implementations, the non-spoken text utterances are generated and / or selected using one or more language models. In some examples, the non-spoken text utterances are generated using a background language model and a domain language model trained on transcribed speech utterances associated with the target domain. The operations may further include pre-training an audio encoder and then fine-tuning the pre-trained encoder on the transcribed speech utterances.

[0015] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a schematic diagram of an exemplary speech recognition system. [Figure 2] FIG. 1 is a schematic diagram of a recurrent neural network transducer (RNN-T) model architecture. [Figure 3A] FIG. 1 is a schematic diagram illustrating an example training process for pre-training an audio encoder for a speech recognition model. [Figure 3B] FIG. 1 is a schematic diagram illustrating an example training process for pre-training an audio encoder for a speech recognition model. [Figure 3C] FIG. 1 is a schematic diagram illustrating an example training process for pre-training an audio encoder for a speech recognition model. [Figure 4] FIG. 1 is a schematic diagram illustrating an exemplary non-spoken text selection process for selecting non-spoken text utterances related to a particular domain. [Figure 5] 1 is an exemplary projection space encoder representation of non-synthetic and synthetic speech. [Figure 6] 1 is a flowchart illustrating an example arrangement of operations for a method for pre-training an audio encoder to jointly learn a shared representation of speech and text. [Figure 7] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0017] Like reference symbols in the various drawings indicate like elements.

[0018] Automated speech recognition has made tremendous progress with the introduction of sequence-to-sequence (Seq2Seq) models that map from audio to character sequences. At the same time, text-to-speech (TTS) or speech synthesis systems have successfully applied Seq2Seq models to obtain state-of-the-art natural, realistic-sounding synthesized speech that may be indistinguishable from human speech to the human ear.

[0019] One challenge in developing deep learning-based ASR models is that their parameters tend to overfit to the training data, which can cause the ASR model to have difficulty generalizing to unknown data when the training data is not sufficiently large. Therefore, training an ASR model with a relatively large training dataset improves its accuracy. For example, by using machine learning or other statistical methods, an ASR model can be trained with a training dataset containing 10,000 hours or more of transcribed speech. However, when the domain associated with the training data differs from the domain in which the ASR model is deployed during inference, the performance of the ASR model degrades. For example, training an ASR model with speech transcribed in a domain associated with videoconferencing can be less effective at recognizing speech related to voice search queries, and vice versa.

[0020] Synthetic speech has the potential to drastically limit the amount of labeled human speech required to train an ASR model, while also providing flexibility for moving ASR models between different domains. Although state-of-the-art synthetic speech examples are generally indistinguishable from human speech, the use of synthetic speech has been shown to impact ASR training differently than human speech. This gap between synthetic and human speech stems from the mismatch between human and synthetic speech data, which arises from the challenging one-to-many mapping problem that TTS systems attempt to solve. That is, while the overall quality of available synthetic speech is very high, synthetic speech has much less variability than human speech, minimizing speech artifacts. As a result, training ASR models exclusively on synthetic speech data makes it difficult to generalize to real-world speech utterances during inference.

[0021] Implementations herein are directed to using synthetic speech to train an ASR model to recognize speech when a large amount of transcribed speech (e.g., non-synthetic speech) in a target domain and / or language for training the ASR model is unavailable or not widely available, in order to maintain the accuracy of the ASR model. More specifically, implementations are directed to pre-training an audio encoder of the ASR model with training data including untranscribed non-synthetic speech utterances, non-spoken text utterances for generating corresponding synthetic speech representations, and transcribed non-synthetic speech utterances for jointly learning speech and text representations, and then fine-tuning (e.g., warm-start training) the pre-trained ASR model using available transcribed non-synthetic speech utterances. As will become apparent, pre-training the audio encoder includes updating parameters of the audio encoder based on a combination of contrastive self-supervised loss, supervised loss, and consistency loss derived from the training data.

[0022] A contrastive self-supervised loss may be derived from latent speech representations generated by an audio encoder from corresponding untranscribed non-synthetic speech utterances, synthetic speech representations, and transcribed non-synthetic speech utterances to facilitate language learning. Meanwhile, a supervised loss may be derived from speech recognition labels predicted by one or more auxiliary decoders based on latent speech representations generated by an audio encoder from corresponding synthetic speech representations and transcribed non-synthetic speech utterances. Here, the corresponding transcriptions paired with the transcribed non-synthetic speech utterances and the corresponding non-spoken text utterances used to generate the synthetic speech representations serve as ground truth labels for deriving the supervised loss. Finally, a consistency loss may be derived from each transcribed non-synthetic speech utterance and its corresponding synthetic speech representation to facilitate consistent predictions (e.g., latent speech representations) by an audio encoder for both non-synthetic speech representations (e.g., real / human) and synthetic speech representations of the same utterance. That is, consistency loss between human (non-synthetic) and synthetic (synthetic) representations of the same utterance provides an aspect of unsupervised training by encouraging audio encoders to behave consistently across human and synthetic training utterances. In particular, a text-to-speech (TTS) model can convert the corresponding transcription paired with each transcribed non-synthetic speech utterance into a corresponding synthetic speech representation of the same utterance.

[0023] Additional implementations include applying data augmentation techniques, such as synthesizing diverse realizations of training utterances by varying the characteristics of the synthesized speakers to promote robustness to speaker differences. The techniques described herein are particularly useful when relatively little transcribed human speech is available in the target domain and / or language.

[0024] 1 illustrates an automated speech recognition (ASR) system 100 that implements an ASR model 200 that resides on a user device 102 of a user 104 and / or on a remote computing device 201 (e.g., one or more servers of a distributed system executing in a cloud computing environment) that communicates with the user device 102. While the user device 102 is shown as a mobile computing device (e.g., a smartphone), the user device 102 may correspond to any type of computing device, such as, but not limited to, a tablet device, a laptop / desktop computer, a wearable device, a digital assistant device, a smart speaker / display, a smart appliance, an in-vehicle infotainment system, or an Internet of Things (IoT) device, and comprises data processing hardware 111 and memory hardware 113.

[0025] The user device 102 includes an audio subsystem 108 configured to receive utterances 106 spoken by the user 104 (e.g., the user device 102 may include one or more microphones for recording the spoken utterances 106) and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the ASR system 100. In the illustrated example, the user speaks each utterance 106 in natural English for the phrase "What is the weather in New York City?", and the audio subsystem 108 converts the utterance 106 into a corresponding acoustic frame 110 for input to the ASR system 100. The ASR model 200 then receives the acoustic frame 110 corresponding to the utterance 106 as input and generates / predicts a corresponding transcription 120 (e.g., a recognition result / hypothesis) of the utterance 106 as output. In the illustrated example, the user device 102 and / or the remote computing device 201 also execute a user interface generator 107 configured to present a representation of a transcription 120 of the utterance 106 to the user 104 of the user device 102. In some configurations, the transcription 120 output from the ASR system 100 is processed by a natural language understanding (NLU) module executing on the user device 102 or the remote computing device 201, for example, to execute user commands. Additionally or alternatively, a text-to-speech system (e.g., executing on any combination of the user device 102 or the remote computing device 201) may convert the transcription into synthesized speech for audible output by another device. For example, the original utterance 106 may correspond to a message the user 104 is sending to a friend, where the transcription 120 is converted into synthesized speech for audible output to the friend, who should hear the message conveyed in the original utterance 106.

[0026] Referring to FIG. 2, an exemplary frame alignment-based transducer model 200a includes a recurrent neural network transducer (RNN-T) model architecture that adheres to latency constraints associated with interactive applications. The use of the RNN-T model architecture is exemplary, and the frame alignment-based transducer model 200 may include other architectures, such as transformer-transducer and conformer-transducer model architectures, among others. The RNN-T model 200 enables a small computational footprint and utilizes fewer memory requirements than traditional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device 102 (e.g., without requiring communication with a remote server). The RNN-T model 200 includes an encoder network 210, a prediction network 220, and a collaboration network 230. The encoder network 210 is generally similar to the acoustic model (AM) of traditional ASR systems and includes a stack of self-attention layers (e.g., conformer or transformer layers) or a recurrent network of stacked long short-term memory (LSTM) layers. For example, the encoder may generate a sequence of d-dimensional feature vectors (e.g., acoustic frame 110 (FIG. 1)) x=(x1, x2,..., x T ) and in the formula, x t ∈

[0027]

number

[0028] and at each output step, we generate a higher-level representation, which is

[0029]

number

[0030] , ...,

[0031]

number

[0032] It is written as follows.

[0033] Similarly, the prediction network 220 is also an LSTM network, which, like a language model (LM), uses the sequence of non-empty symbols output so far by the final softmax layer 240, i.e., y 0 , ... , y ui-1 A dense representation of

[0034]

number

[0035] Finally, using the RNN-T model architecture, the representations generated by the encoder and predictor / decoder networks 210, 220 are combined by the combination network 230. The prediction network 220 can be replaced with an embedding lookup table to improve latency by outputting a sparse embedding that is looked up instead of processing a dense representation. The combination network then

[0036]

number

[0037] , which is a distribution over the next output symbol. In other words, at each output step (e.g., time step), the collaborative network 230 generates a probability distribution over possible speech recognition hypotheses. Here, a "possible speech recognition hypothesis" corresponds to a set of output labels, each representing a symbol / character in a specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters of the English alphabet, and one label specifying a space. Thus, the collaborative network 230 may output a set of values ​​indicating the likelihood of each occurrence of a given set of output labels. This set of values ​​may be a vector and may indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels may include wordpieces and / or entire words in addition to, or instead of, graphemes. The output distribution of the collaborative network 230 may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output y of the collaborative network 230 i may contain 100 different probability values, one for each output label. The probability distribution may thus be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and / or words) in a beam search process (e.g., by softmax layer 240) to determine transcription 120.

[0038] The softmax layer 240 may utilize any technique for selecting the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200 at the corresponding output step. In this manner, the RNN-T model 200 does not make a conditional independence assumption; rather, the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model 200 assumes that the output symbols are independent of future acoustic frames 110, which allows the RNN-T model to be used in a streaming manner.

[0039] In some examples, the encoder network (i.e., audio encoder) 210 of the RNN-T model 200 includes a stack of self-attention layers / blocks, such as conformer blocks. Here, each conformer block includes a series of multi-head self-attention, depthwise convolutional, and feedforward layers. The prediction network 220 may have two 2,048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. Alternatively, the prediction network 220 may include a stack of transformer or conformer blocks, or an embedded lookup table, instead of an LSTM layer. Finally, the joint network 230 may also have 640 hidden units. The softmax layer 240 may consist of a joint word piece or grapheme set, generated using all unique word pieces or graphemes in multiple training datasets.

[0040] 3A-3C illustrate an exemplary training process 300 for pre-training the audio encoder 210 of the ASR model 200 (FIG. 2). The training process 300 involves pre-training a non-spoken text utterance (X text ) a set of 320 transcribed non-synthesized speech utterances (X sup ) 304 sets, and untranscribed non-synthesized speech utterances (X unsupThe audio encoder 210 can be pre-trained using available training data, including the non-spoken training text utterances 320, 302, 304, 306, 310, 312, 314, 316, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, ​​383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405,

[0041] For simplicity, the training process 300 includes a contrastive self-supervised loss section 300a (FIG. 3A), a supervised loss section 300b (FIG. 3B), and a consistency regularizer section 300c (FIG. 3C). The training process 300 is performed on a non-spoken text utterance (X text )320, transcribed non-synthesized speech utterances (X sup ) 304 corpora, and untranscribed non-synthetic speech utterances (X unsup ) 306 using the contrastive self-supervised loss unit 300a. w2v )316, Non-spoken text utterances (X text )320 and transcribed non-synthesized speech utterances (X sup ) 304 using the supervised loss unit 300b. aux ) 344, as well as the consistency loss (L cons ) 352 based on the total loss (L tts4pretrain2 ) to pre-train the audio encoder 210.

[0042] 3A, the contrastive self-supervised loss portion 300a of the training process 300 may employ a text-to-speech (TTS) system 330 configured to generate, at each of a plurality of output steps, a synthetic speech representation (e.g., synthetic speech) 332 for each of a plurality of non-spoken training text utterances 320. The non-spoken training text utterances 320 (also referred to simply as "non-spoken text utterances 320") may be generated by generating a synthetic speech representation (e.g., synthetic speech) 332 for each of a plurality of non-spoken text utterances (e.g., X text ) 320 includes non-spoken text, which is text-only data, i.e., unpaired data, such that the non-spoken text utterances 320 are not paired with any synthetic or unsynthetic speech. Accordingly, the TTS system 330 generates a corresponding synthetic speech representation 332 for each non-spoken text utterance 320. In particular, the synthetic speech representation 332 may include mel-frequency spectrogram frames for training the audio encoder 210, thereby eliminating the need for the training process 300 to include a vocoder and / or synthesizer to synthesize the mel-frequency spectrogram frames into synthetic speech.

[0043] The TTS system 330 may apply the speaker embedding z when converting the non-spoken text utterance 320 to generate a synthesized speech representation 332 having a particular speaking style and prosody associated with the speaker embedding. The TTS system 330 may apply multiple different speaker embeddings z, each associated with a different speaker characteristic of the resulting utterance of the synthesized synthesized speech representation 332. Similarly, the TTS system 330 may vary the prosody and other production qualities of the synthesized utterance.

[0044] In some examples, the training process 300 applies data augmentation to at least one of the sample utterances of the synthetic speech representation 332. Data augmentation may include, but is not limited to, adding noise, manipulating timing (e.g., stretching), or adding reverberation to the corresponding speech representation. Data augmentation may add different synthesis recording conditions to the synthetic speech representation 332.

[0045] This pre-training batch generation process for generating sample utterances for synthetic speech representation 332 advantageously samples new speaker and prosodic conditioning values ​​each time a non-spoken text utterance 320 is observed during training, resulting in a variety of synthetic utterances in subsequent observations. Each batch therefore contains both synthetic and real (non-synthetic) utterances. The loss contribution can be masked by using a loss mask σ (see Equation 4), so that the loss is calculated for the appropriate batch elements.

[0046] In some examples, the audio encoder 210 includes a stack of self-attention layers, each including a multi-head self-attention mechanism. For example, the stack of self-attention layers may include a stack of conformer layers or transformer layers. In the illustrated example, the audio encoder 210 includes a conformer encoder including a stack of conformer blocks, each of which includes a series of multi-head self-attention layers, a depthwise convolutional layer, and a feedforward layer. The conformer encoder 210 can naturally be divided into a feature encoder including a convolutional subsampling block 212 and a context network including a stack of linear layers 214 and conformer blocks 216. In some implementations, the convolutional subsampling block 212 has two 2D convolutional layers, both with stride (2, 2), resulting in a quarter-fold reduction in feature sequence length. The convolutional subsampling block 212 receives as input a sequence of input features / vectors (e.g., Mel-frequency spectrograms such as acoustic frame 110 in FIG. 1 ) associated with each synthesized speech representation 332, each transcribed non-synthesized speech utterance 304, and each untranscribed non-synthesized speech utterance 306, and generates as output, for each of a plurality of output steps, an encoding feature 211 corresponding to one of the synthesized speech representations 332, one of the transcribed non-synthesized speech utterances 304, or one of the untranscribed non-synthesized speech utterances 306, respectively.

[0047] The encoding features 211 output from the convolutional subsampling block 212 are supplied to a masking module 218, where a portion of the encoding features 211 is randomly selected and replaced with a trained feature vector shared among all masked time steps to provide corresponding masked encoding features 211m. In some examples, the masking module 218 masks the randomly selected encoding features 211 for masking by randomly sampling a percentage p of all time steps without replacement as a starting index, and then masking the subsequent M consecutive time steps from all sample indexes, which may result in some overlapping spans. After masking is applied, the linear layer 214 of the context network and the conformer block 216 receive the masked encoding features 211m and output a corresponding contrastive context vector (i.e., a corresponding encoded representation) 215 from the masked encoding features 211m. Furthermore, a quantizer 217 receives the encoding features 211 as input and generates a quantized vector (i.e., a target context vector) 219 as output. Then, the contrastive loss module 315 calculates the contrastive loss (L w2v )316 is derived as follows:

[0048]

number

[0049] In the formula, c t is the contrastive context vector 215 centered at the masked time step t, and q t , q t represents the target context vector 219 at time step t in a set of K+1 candidate target context vectors 219 containing K distractors. The distractors may be uniformly sampled from other masked time steps of the same utterance.

[0050] The contrastive loss 316 is optimized between the contrastive context vector 215 at the mask location and the target context vector 219. After the pre-trained audio encoder 210 converges on the untranscribed non-synthetic speech utterance 306, the pre-training procedure is repeated on both the synthetic speech representation 332 and the transcribed non-synthetic speech utterance 304. Thus, the contrastive loss 316 is optimized for both real / human (non-synthetic) features and synthetic (TTS audio) features, with additional auxiliary losses for the transcribed non-synthetic speech utterance 304 and the synthetic speech representation 332, as described in further detail below with reference to FIG. 3B. Thus, the training process 300 pre-trains the audio encoder 210 based on the derived contrastive loss 316 applied to the corresponding coding features 211 associated with each synthetic speech representation 332, each transcribed non-synthetic speech utterance 304, and each untranscribed non-synthetic speech utterance 306 provided as input to the audio encoder 210. Pre-training the audio encoder 210 may include updating parameters of the audio encoder based on the contrastive loss.

[0051] Referring to FIG. 3B , the supervised loss portion 300b of the training process 300 is configured to inject lexical information into the audio encoder 210 during pre-training based on supervised loss terms 342, 344 derived from the transcribed unsynthesized speech utterance 304 and the synthetic speech representation 332 generated by the TTS system 330 for the unspoken text utterance 320. In particular, the supervised loss portion 300b utilizes one or more auxiliary decoders 390 to generate the supervised loss terms 344, 346. The auxiliary decoders 390 may include a Connectionist Temporal Classification (CTC) decoder, a Listen Attend Spell (LAS) decoder, or an RNN-T decoder. These auxiliary decoders 390 may include at least one of a phoneme decoder configured to decode a sequence of phonemes or a wordpiece decoder configured to decode a sequence of wordpieces. The auxiliary decoder 390 may also include a grapheme decoder configured to decode a sequence of graphemes. In some examples, the training process 300 applies data augmentation to at least one of the sample utterances of the synthetic speech representation 332 to provide one or more lexically diverse synthetic speech representations 332 for a given non-speech training text utterance 320. Data augmentation may include, but is not limited to, adding noise, manipulating timing (e.g., stretching), or adding reverberation to the corresponding speech representation. Data augmentation may add different synthesis recording conditions to the synthetic speech representation 332.

[0052] During the supervised loss portion 300b, the audio encoder 210 receives as input each synthetic speech representation 332 generated from the non-spoken text utterance 320 as a sequence of features / vectors (e.g., Mel-frequency spectrograms such as the acoustic frames 110 of FIG. 1 ), and for each of a plurality of time steps generates a first encoded representation (e text) 312 as an output. An auxiliary decoder 390, including a phoneme decoder or wordpiece decoder, receives as input each first coded representation 312 output from the audio encoder 310 and generates as an output a first probability distribution 392 over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation 332 at the corresponding time step. In some examples, the first probability distribution 392 over the possible synthetic speech recognition hypotheses includes one of possible phoneme labels or possible wordpiece labels. The supervised loss module 340 may then determine a synthetic speech loss term 342 based on the first probability distribution 392 over the possible synthetic speech recognition hypotheses and the corresponding non-spoken text utterance 320. Here, the corresponding non-spoken text utterance 320 from which the synthetic speech representation 332 is generated also serves as a ground truth transcription. The supervised loss unit 300b may pre-train the audio encoder 210 based on the synthetic speech loss term 342 by updating parameters of the audio encoder 210.

[0053] Similarly, during the supervised loss portion 300b, the audio encoder 210 receives as input each transcribed non-synthesized speech utterance 304 as a sequence of features / vectors (e.g., Mel-frequency spectrograms such as the acoustic frame 110 in FIG. 1 ) and, for each of a plurality of time steps, generates a second coded representation (e sup) 314 as an output. An auxiliary decoder 390, including a phoneme decoder or wordpiece decoder, receives as input each second coded representation 314 output from the audio encoder 310 and generates as an output a second probability distribution 394 over the possible non-synthesis speech recognition hypotheses for the corresponding transcribed non-synthesized speech utterance 304 at the corresponding time step. In some examples, the second probability distribution 394 over the possible synthetic speech recognition hypotheses includes one of the possible phoneme labels or the possible wordpiece labels. The supervised loss module 340 may then determine a non-synthesis speech loss term 344 based on the second probability distribution 394 over the possible non-synthesis speech recognition hypotheses and the corresponding transcription 302 paired with the transcribed non-synthesized speech utterance 304. Here, the corresponding transcription 302 serves as a ground truth transcription and may include a sequence of target phonemes and / or target wordpieces. The supervised loss unit 300b may pre-train the audio encoder 210 based on the non-synthesized speech loss term 344 by updating the parameters of the audio encoder 210.

[0054] In some implementations, the supervised loss portion 300b of the training process 300 uses a separate auxiliary decoder 390 to decode the first coded representation (e text) 312, generates a third probability distribution 393 over the possible synthetic speech recognition hypotheses, whereby the supervised loss module 340 may determine another synthetic speech loss term 342 based on the third probability distribution and the non-spoken text utterance 320 corresponding to the synthetic speech representation. Here, the other auxiliary decoder 390 includes the other of the phoneme decoder or the wordpiece decoder, and the third probability distribution 393 over the possible synthetic speech recognition hypotheses includes the other of the possible phoneme labels or the possible wordpiece labels. In these implementations, the other auxiliary decoder 290 also generates a fourth probability distribution 395 over the possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance 304 at the corresponding time step, whereby the supervised loss module 340 may determine another non-synthetic speech loss term 344 based on the fourth probability distribution 395 and the corresponding transcription 302 paired with the transcribed non-synthetic speech utterance 304. Here, a fourth probability distribution 395 over the possible non-synthetic speech recognition hypotheses includes the other of the possible phoneme labels or possible wordpiece labels. The supervised loss portion 300b of the training process 300 can similarly pre-train the audio encoder 210 based on other synthetic speech loss terms 342 and other non-synthetic speech loss terms 344.

[0055] The untranscribed non-synthesized speech utterance 306 and the non-spoken text utterance 320 are each a non-spoken text utterance (X text )320 derived from the contrast loss (L w2v ) 316 (FIG. 3A) as the supervised loss J associated with the synthetic speech loss term 342 aux Combined with this, we use the non-spoken text loss function J as follows: text corresponds to the "unpaired" training data from which J text =L w2v (x│θ e )+L aux (y│x,θ e ,θ_d ) (2) Similarly, untranscribed non-synthesized speech utterances (X unsup )306 derived from the contrast loss (Lw2v ) 316 (Figure 3A) to calculate the unsupervised speech loss function, J unsup_speech can be expressed as follows: J unsup_speech = J w2v (x*│θ e ) (3)

[0056] During pre-training of the audio encoder 210, synthetic speech representations 332 and untranscribed non-synthetic speech utterances 306 are mixed within each batch. To force the audio encoder 210 to learn representations that are valid for both synthetic and non-synthetic (human / real) speech, the loss function J in Equation 2 and Equation 3 is text and J. unsup_speech When combining, a loss mask σ is applied to the unpaired data loss function J unpaired is obtained as follows: J unpaired =σ J text +(1-σ) J speech (4)

[0057] The transcribed non-synthetic speech utterances 304 correspond to the “paired” and “supervised” training data, which allows the derived contrastive loss (L w2v ) 316 (FIG. 3A) and the derived supervised loss J associated with the non-synthetic speech loss term 344 aux are combined to form a paired data loss function J paired can be obtained as follows: J paired =L w2v (x│θ e )+L aux (y│x,θ e ,θ d ) (5)

[0058] Referring to FIG. 3C, the consistency regularizer portion 300c of the training process 300 performs a consistency regularization on the transcribed non-synthesized speech utterances (X sup ) 304 and the corresponding transcribed non-synthetic speech utterance 304 and the synthetic speech representation 334 of the same utterance pair. consThe auxiliary decoder 390 is configured to generate a consistency loss term (θ) 352 to encourage the audio encoder 210 to learn a consistency prediction between non-synthetic speech (e.g., real / human speech) and synthetic speech. Thus, the transcribed non-synthetic speech utterance 304 and the paired synthetic speech representation of each training utterance are associated with the same ground truth transcription. In essence, the consistency loss term between the non-synthetic and synthetic speech representations of the same training utterance provides an aspect of unsupervised training by encouraging the audio encoder 210 to operate consistently, regardless of whether the training utterance belongs to non-synthetic or synthetic speech, and regardless of the supervised loss term between the ground truth transcription 302 and each of the non-synthetic and synthetic speech recognition hypotheses output by the auxiliary decoder 390.

[0059] Similar to the synthetic speech representation 332 generated from the non-spoken text utterance 320 in FIG. 3B , the TTS system 330 may generate each paired synthetic speech representation 334 by performing text-to-speech conversion on the corresponding transcription 302 paired with the transcribed non-synthetic speech utterance 304. Here, the transcribed non-synthetic speech utterance 304 is associated with the synthetic speech generated by the TTS system 330 by converting the text associated with the ground truth transcription 302 into synthetic audio. The TTS system 330 may apply the speaker embedding z when converting the ground truth transcription (y*) 302 to obtain synthetic speech having the particular speaking style and prosody associated with the speaker embedding. Here, the ground truth transcription (y*) 302 is associated with a source for supervised data augmentation, and the TTS system 330 may generate the transcribed non-synthetic speech utterance (X*) associated with the ground truth transcription (y*) 302. sup) 304. In some examples, the training process 300 applies data augmentation to at least one of the transcribed non-synthesized speech utterances 304 or the synthetic speech representations 334 of at least one pair of training utterance pairs 301. Data augmentation may include, but is not limited to, adding noise to the corresponding speech representation, manipulating timing (e.g., stretching), or adding reverberation.

[0060] In the consistency regularizer 300c, the audio encoder 210 receives as input each pair of synthetic speech representations 334 as a sequence of features / vectors (e.g., Mel-frequency spectrograms such as the acoustic frame 110 in FIG. 1 ), and for each of a plurality of time steps, generates an extended coded representation (e * sup ) 313 as output. An auxiliary decoder 390, which may include a phoneme decoder or a wordpiece decoder, receives as input each extended coded representation 313 output from the audio encoder 210 and produces as output a first probability distribution 311 over the possible synthetic speech recognition hypotheses for the corresponding pair of synthetic speech representations 334 at the corresponding time step. In some examples, the first probability distribution 311 over the possible synthetic speech recognition hypotheses includes one of the possible phoneme labels or the possible wordpiece labels.

[0061] Similarly, the audio encoder 210 receives as input each transcribed non-synthesized speech utterance 304 as a sequence of features / vectors (e.g., Mel-frequency spectrograms such as the acoustic frame 110 in FIG. 1 ) and, for each of a plurality of time steps, generates an unaugmented coded representation (e sup) 314 as an output. An auxiliary decoder 390, which may include a phoneme decoder or a wordpiece decoder, receives as input each unaugmented coded representation 314 output from the audio encoder 310 and produces as output a second probability distribution 394 over the possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance 304 at the corresponding time step. In some examples, the second probability distribution 394 over the possible synthetic speech recognition hypotheses includes one of the possible phoneme labels or the possible wordpiece labels.

[0062] Continuing with reference to FIG. 3C , consistency regularizer 300 c of training process 300 calculates a consistency loss term (J ) for each training utterance pair 301 at each of a plurality of time steps for the corresponding training utterance pair 301 based on a first probability distribution 311 over the possible non-synthetic speech recognition hypotheses and a second probability distribution 394 over the possible non-synthetic speech recognition hypotheses. cons For example, the training process 300 may utilize a consistency loss term module 350 configured to receive, at each time step, corresponding unsynthesized and synthesized speech recognition results 311, 394 output by the auxiliary decoder 390 and determine a consistency loss term 352 for the corresponding training utterance pair 301 at the time step.

[0063] In some examples, the consistency regularizer portion 300c of the training process 300 calculates the Kullback-Leibler divergence (D) between a first probability distribution 311 over the possible synthetic speech recognition hypotheses and a second probability distribution 394 over the possible non-synthetic speech recognition hypotheses. KL ) to determine the consistency loss term 352. KL The consistency loss term 352 based on can be expressed as:

[0064]

number

[0065] Here, the consistency loss term 352 determined for the training utterance pair 301 at each time step provides an “unsupervised” loss term that does not depend on the accuracy of the auxiliary decoder 390 (e.g., does not depend on the supervised loss terms 342, 344 of FIG. 3B ) and thus may be utilized to update parameters of the audio encoder 210 to promote consistency between the non-synthetic and synthetic speech representations of the same utterance. In batch training, the consistency loss term 352 may correspond to the average loss term obtained for the batch. In other words, the consistency loss term 352 allows the audio encoder 210 to learn to behave in the same way regardless of whether the training utterance belongs to non-synthetic or synthetic speech, e.g., to make consistent coding representation predictions for both non-synthetic (e.g., real / human) and synthetic (e.g., synthetic) speech of the same training utterance.

[0066] Finally, the training process 300 calculates the unpaired data loss function (J unpaired ), paired data loss functions (J paired ), and the consistency loss term (J cons ) to form an overall loss term J, which can be expressed as tts4pretrain2 can be obtained. J tts4pretrain2 = J unpaired +λ1Jpaired+λ2J cons (7) where λ1 may be equal to 1.0 and λ2 may be equal to 0.1. The training process 300 updates the parameters of the audio encoder 210 to reduce the overall loss term J tts4pretrain2The audio encoder 210 may be pre-trained using the training process 300, effectively teaching the audio encoder 210 to learn a shared representation between speech and text. After pre-training the audio encoder 210, the training process 300 may fine-tune the pre-trained audio encoder on transcribed speech utterances, which may include both synthetic (e.g., synthetic speech) and non-synthetic (e.g., human speech) supervised training samples.

[0067] In some implementations, the training process 300 for pre-training the audio encoder 210 applies encoder consistency regularization. Unlike decoder consistency regularization, which is applied to the auxiliary decoder during the consistency regularization section 300c, which requires hypothesis labels (e.g., transcription 302 and non-spoken text utterances 320), encoder consistency regularization has the advantage of not requiring hypothesis labels and therefore being applicable to all training data 304, 306, 320. Encoder consistency regularization can be applied via a hierarchical contrastive consistency regularization (HCCR) technique, in which encoder activations e, e* from the original / unaugmented and augmented speech are projected through an auxiliary network to generate z, z*. Then, positive-negative pairs are constructed and a contrastive loss l is applied. t,z,z* is calculated as follows:

[0068]

number

[0069] Specific to HCCR, a convolutional neural network (CNN) projection network computes projections of encoder activations e in increasing length segments (30, 50, and 120 ms) to generate three views (V), drawing negative examples from the same utterance in the shorter segments and from other utterances in the batch with 120 ms segments. Thus, for a transcribed non-synthetic speech utterance 304 (paired speech), an untranscribed non-synthetic speech utterance 306 (unpaired speech), and a synthetic speech representation (synthetic speech) generated from the unspoken text utterance 320, the HCCR loss may be computed as follows:

[0070]

number

[0071] The HCCR loss calculated by Equation 9 is the overall loss term J for use in pre-training the audio encoder 210. tts4pretrain2 can be added to equation 7 by a factor of 1e-3 as part of

[0072] 4, a contrastive non-spoken text selection process 400 can select non-spoken text utterances 320 from a large non-spoken text corpus 402 to be used for pre-training the audio encoder 210, such that the selected non-spoken text utterances 320 are most similar to the particular domain that the audio encoder 210 is being pre-trained to learn from. That is, the text selection process 400 can identify non-spoken text within and near a domain from the non-spoken text corpus 402 for inclusion in the non-spoken text utterances 320 to be used for pre-training the audio encoder 210. In particular, the non-spoken text utterances 320 selected by the text selection process 400 allow different utterances to be synthesized on the fly during batch construction, such that each time a non-spoken text utterance 320 is in a batch, a new speaker embedding z and the latent variable Z can be sampled.

[0073] The corpus of non-spoken text 402 includes a large number of non-spoken training text utterances 320, 320a-n from across a wide range of domains, encompassing a much greater linguistic diversity than the specific domain in which the audio encoder 210 is trained to learn. As previously mentioned, the set of transcribed non-synthesized speech utterances 304 may be domain-specific, in that they are associated with a particular domain, and each transcribed non-synthesized speech utterance 304 is paired with a corresponding transcription 302. The corpus of non-spoken text 402 may be stored in the same or a different data store 401 as the spoken transcribed non-synthesized speech utterances (i.e., training utterances) 304. The corpus of non-spoken text 402 may change dynamically to incorporate new non-spoken text utterances 320. Simply using all the non-spoken text utterances 320 in the non-spoken text corpus 402 is not feasible for the following reasons: i) for each sentence, the speech modality requires much more memory to be encoded than the text, thereby making it infeasible to convert all the text in the non-spoken text corpus 402; ii) the enormous amount of difference between the transcriptions 302 paired with the transcribed non-synthesized speech utterances 304 and the non-spoken text utterances 320 in the non-spoken text corpus 402 requires an intelligent strategy to balance their contributions.

[0074] The text selection process 400 aims to select a subset of available non-spoken text utterances 320 from a non-spoken text corpus 402 as data for TTS synthesis, resulting in the synthetic speech representations 332 generated for pre-training the audio encoder 210 during the contrastive loss and supervised loss portions 300a, 300b of the training process 300 described above with reference to Figures 3A and 3B. In other words, the text selection process 400 aims to improve the match between the selected subset of available non-spoken text utterances 320 and the specific domain being targeted, thereby, in turn, reducing the computational resources required to leverage large amounts of non-domain-specific data. Thus, the text selection process 400 reduces computational and memory costs by selecting non-spoken text utterances 320 that best match the specific domain that the audio encoder 210 is trained to learn.

[0075] In some examples, the text selection process 400 selects a subset of available non-spoken text utterances 320 from the non-spoken text corpus 402 that best matches a particular region by simply providing a region identifier (not shown) associated with the particular region as input to a background LM 406 that has been pre-trained across the entire non-spoken text corpus 402. As previously mentioned, the non-spoken text corpus 402 spans a number of different regions. In these examples, the background LM 406 may include a maximum entropy (MaxEnt LM) that can optionally accept a region identifier as input, as described in U.S. Patent No. 9,842,592, filed February 12, 2014, the entire contents of which are incorporated herein by reference. Here, the region identifier associated with a particular region may enable the MaxEnt LM to output a subset of available non-spoken text utterances 320 from the non-spoken text corpus 402 that may contain words and / or phrases related to the particular region. In some configurations, rather than assessing word likelihood, the statistical language model operates in reverse mode to randomly generate text phrases that match the statistical distribution of words associated with a particular domain.

[0076] In a further example, and as shown in FIG. 4 , a text selection process 400 uses transcriptions 302 paired with transcribed non-synthesized speech utterances 304 spoken by a human speaker to select a subset of available non-spoken text utterances 320 from the non-spoken text corpus 402 that best matches a particular domain, where the transcribed non-synthesized speech utterances 304 include words, phrases, and / or other terminology related to the particular domain. Optionally, in addition to or instead of the transcriptions 302 paired with the transcribed non-synthesized speech utterances 304, a different set of transcribed utterances related to the particular domain can be used to select the non-spoken text utterances 320. This provides the advantage of not requiring all transcribed non-synthesized speech utterances 304 to belong to a particular domain.

[0077] During the first stage (Stage A), the non-spoken text selection process 400 assembles two language models 404, 406 to enable contrastive selection of non-spoken text utterances 320. Here, a domain-specific language LM 410 is trained on each transcription 302 in the set of transcribed non-synthesized speech utterances 304. The set of transcribed non-synthesized speech utterances 304 is assumed to belong to the specific domain that the audio encoder 210 is trained to learn. Meanwhile, a background LM 406 is trained on each non-spoken text utterance 320 in the entire non-spoken text corpus 402. As mentioned above, the non-spoken text corpus 402 spans many different domains. In some examples, the first stage uses n-gram language model training to build the two language models 404, 406. In other examples, the first stage uses neural network language model training to build the two language models 404, 406.

[0078] During a second state (Phase B), the unspoken text selection process 400 calculates a first probability associated with each word in the unspoken text utterance 320 that appears in the domain-specific LM 404, i.e.,

[0079]

number

[0080] and determining a second probability associated with each word in the non-spoken text utterance 320 that appears in the background LM 406, i.e.

[0081]

number

[0082] The two contrasting LMs 404, 406 are used to evaluate each non-spoken text utterance 320 in the non-spoken text corpus 402 by determining a probability, S, of each non-spoken text utterance 320. Then, for each non-spoken text utterance 320 in the non-spoken text corpus 402, the text selection process 400 determines a score, S, in a scorer 408, based on the first probability, the second probability, and the number of words, #(w), that appear in the corresponding non-spoken text utterance 320. For example, the score S for each non-spoken text utterance 320 may be calculated as follows:

[0083]

number

[0084] After determining the scores, the non-spoken text selection process 400 selects the non-spoken text utterances 320 with the N best scores S as these non-spoken text utterances 320 best match the particular domain. The non-spoken text corpus 402 may contain billions of non-spoken text utterances 320. The non-spoken text utterances 320 selected by the text selection process 400 can include millions of utterances, far exceeding the number of untranscribed, non-synthesized speech utterances 306 spoken by human speakers. As described above, the content of the non-spoken text utterances 320 increases linguistic diversity for the particular domain that the audio encoder 210 is trained to learn, and the corresponding synthetic speech representations 332 generated from the non-spoken text utterances 320 increase acoustic / lexical diversity for the speech that the audio encoder 210 is encoding as part of the speech recognition process when the audio encoder 210 is integrated into the ASR model 200.

[0085] Figure 5 shows an example projection space 500 of encoder representations of synthetic (TTS) and non-synthetic (real / human) speech utterances. After introducing consistency regularization via the consistency regularizer 300c of Figure 3C for pre-training the audio encoder, the resulting learned speech and text encoder representations remain much closer to each other compared to the speech and text encoder representations when consistency regularization is not applied. Thus, projection space 500 demonstrates that using supervised training data (i.e., transcribed non-synthetic speech utterances) to pre-train the audio encoder 210 effectively produces improved shared speech and text representations.

[0086] 6 is a flowchart of an exemplary arrangement of operations for method 600 of pre-training audio encoder 210 to jointly learn a shared representation of speech and text. Method 600 may be executed on data processing hardware 710 (FIG. 7) using instructions stored in memory hardware 720 (FIG. 7). Data processing hardware 710 and memory hardware 720 may reside on remote computer / server 201 of FIG. 1, which corresponds to computing device 700 (FIG. 7).

[0087] At operation 602, the method 600 may include receiving training data including unspoken text utterances 320, untranscribed unsynthesized speech utterances 306, and transcribed unsynthesized speech utterances 304. Each unspoken text utterance 320 is not paired with any corresponding spoken utterance of unsynthesized speech. Each untranscribed unsynthesized speech utterance 306 is not paired with a corresponding transcription. Each transcribed unsynthesized speech utterance 304 is paired with a corresponding transcription 302.

[0088] At operation 604, the method 600 also includes generating a corresponding synthetic speech representation 332 for each non-spoken text utterance 320 in the received training data using a text-to-speech (TTS) system 330. At operation 606, the method also includes pre-training the audio encoder 210 based on the synthetic speech representations 332 generated for the non-spoken text utterance 320, the untranscribed non-synthetic speech utterance 306, and the transcribed non-synthetic speech utterance 304 to teach the audio encoder 210 to jointly learn the shared speech and text representations. The pre-training may include pre-training the audio encoder 210 based on a contrastive loss 315 derived from each of the synthetic speech representation 332, the untranscribed non-synthetic speech utterance 306, and the transcribed non-synthetic speech utterance 304. The pre-training may also include pre-training the audio encoder 210 based on supervised losses 342, 344 (e.g., auxiliary decoder losses) derived from the synthetic speech representation 332 and the transcribed non-synthetic speech utterance 304. Finally, the pre-training may further include pre-training the audio encoder 210 based on a consistency loss 352 derived from the transcribed non-synthetic speech utterance 304.

[0089] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0090] Non-transitory memory may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transitory memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0091] 7 is a schematic diagram of an exemplary computing device 700 that may be used to implement the systems and methods described herein. Computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality are intended to be exemplary only and are not intended to limit the implementation of the invention described and / or claimed herein.

[0092] Computing device 700 includes a processor 710, a memory 720, a storage device 730, a high-speed interface / controller 740 that connects to memory 720 and a high-speed expansion port 750, and a low-speed interface / controller 760 that connects to a low-speed bus 770 and storage device 730. Each of components 710, 720, 730, 740, 750, and 760 are interconnected using various buses and may be mounted on a common motherboard or in other manners as appropriate. Processor 710 can process instructions for execution within computing device 700, including instructions stored in memory 720 or on storage device 730, for displaying graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 780 coupled to high-speed interface 740. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memory, as needed. Also, multiple computing devices 700 may be connected, each providing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0093] Memory 720 stores information non-transiently within computing device 700. Memory 720 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-transient memory 720 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0094] The storage device 730 is capable of providing mass storage for the computing device 700. In some implementations, the storage device 730 is a computer-readable medium. In various different implementations, the storage device 730 can be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or devices in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 720, the storage device 730, or memory on the processor 710.

[0095] The high-speed controller 740 manages bandwidth-intensive operations for the computing device 700, while the low-speed controller 760 manages less bandwidth-intensive operations. Such allocation of duties is merely exemplary. In some implementations, the high-speed controller 740 is coupled to memory 720, a display 780 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 750 that may receive various expansion cards (not shown). In some implementations, the low-speed controller 760 is coupled to a storage device 730 and a low-speed expansion port 790. The low-speed expansion port 790 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), but may also be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, etc., or to a network device, such as a switch or router, for example, through a network adapter.

[0096] The computing device 700 may be implemented in several different forms, as shown in the figure. For example, the computing device 700 may be implemented as a standard server 700a, or multiple times in a group of such servers 700a, or as a laptop computer 700b, or as part of a rack server system 700c.

[0097] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0098] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0099] The processes and logic flows described herein may be performed by one or more programmable processors, also referred to as data processing hardware, that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special-purpose logic circuitry, for example, an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operably coupled to a mass storage device to receive data from, transfer data to, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0100] To enable interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Furthermore, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0101] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0102] 100 Automated Speech Recognition (ASR) Systems 102 User Devices 104 users 106 Vocalization 107 User Interface Generator 108 Audio Subsystem 110 Acoustic Frame 111 Data Processing Hardware 113 Memory Hardware 120 Transcription 200 ASR model 200 RNN-T model 200a transducer model 201 Remote Computing Devices 210 Encoder Network 210 Audio Encoder 211 Encoding Features 211m Masked coding features 212 Convolution Subsampling Block 214 Linear Layer 215 contrastive context vectors 215 encoded representation 216 Conformer Block 217 Quantizer 218 Masking Module 219 Target Context Vector 220 Prediction Network 230 Collaborative Network 240 softmax layers 300 training processes 301 Training Vocal Pairs 302 Transcription 300a Symmetric self-supervised loss part 300b Supervised loss part 300c Consistency Regularizer 301 Training Vocal Pairs 302 Transcription 304 Transcribed Non-Synthesized Speech Utterances 306 Untranscribed non-synthesized speech utterances 312 First Encoded Representation 314 Second Encoded Representation 315 Contrastive Loss Module 316 Contrasting Losses 320 Non-speech Text Utterances 330 Text-to-Speech (TTS) System 332 Synthetic speech expression 334 pairs of synthetic speech representations 340 Supervised Loss Module 342 Supervised loss term 344 Supervised loss term 346 Supervised Loss Term 350 Consistency Loss Term Module 352 Consistency Loss 390 Auxiliary Decoder 392 First Probability Distribution 393 Third Probability Distribution 394 Second Probability Distribution 395 Fourth Probability Distribution 400 Contrastive Non-Speech Text Selection Process 401 Datastore 402 Non-speech text corpus 404 Domain Language Model 404 Domain-specific LM 406 Background Language Model 408 Scorer 410 Domain-Specific Language LM 500 projection space 600 ways 700 computing devices 700a Server 700b laptop computer 700c Rack Server System 710 Data Processing Hardware 710 processor 720 memory 730 Storage Devices 740 High-Speed ​​Interface / Controller 750 High-Speed ​​Expansion Port 760 Low-Speed ​​Interface / Controller 770 Slow Bus 780 Display 790 Low-Speed ​​Expansion Port

Claims

1. A computer-implemented method (600) that, when executed on data processing hardware (710), causes the data processing hardware (710) to perform operations, the operations comprising: non-spoken text utterances (320), each of which is not paired with any corresponding spoken utterance of the non-synthesized voice; untranscribed, non-synthesized speech utterances (306), each untranscribed, non-synthesized speech utterance (306) not paired with a corresponding transcription; Transcribed non-synthesized speech utterances (304), each transcribed non-synthesized speech utterance (304) paired with a corresponding transcription (302); receiving training data (304, 306, 320) including: generating a corresponding synthetic speech representation (332) for each non-spoken text utterance (320) of the received training data (304, 306, 320) using a text-to-speech model (330); pre-training the audio encoder (210) based on the non-speech text utterance (320), the untranscribed non-synthesized speech utterance (306), and the synthetic speech representation (332) generated for the transcribed non-synthesized speech utterance (304) to teach the audio encoder (210) to jointly learn a shared representation between speech and text; Including, Pre-training the audio encoder (210) At each of a plurality of time steps of each synthesized speech representation (332), generating, using an auxiliary decoder (390), a first probability distribution (392) over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation (332); determining a synthetic speech loss term (342) based on the first probability distribution (392) over possible synthetic speech recognition hypotheses and the unspoken text utterance (320) corresponding to the corresponding synthetic speech representation (332); pre-training the audio encoder (210) based on the synthetic speech loss term (342); At each of a plurality of time steps of each transcribed non-synthesized speech utterance (304), generating, using the auxiliary decoder (390), a second probability distribution (394) over possible non-synthesized speech recognition hypotheses for the corresponding transcribed non-synthesized speech utterance (304); determining a non-synthetic speech loss term (344) based on the second probability distribution (394) over possible non-synthetic speech recognition hypotheses and the corresponding transcription (302) paired with the transcribed non-synthetic speech utterance (304); pre-training the audio encoder (210) based on the non-synthetic speech loss term (344); A computer-implemented method (600) comprising:

2. 10. The computer-implemented method of claim 1, wherein the audio encoder includes a stack of self-attention layers, each of which includes a multi-head self-attention mechanism.

3. A computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform an operation, the operation comprising: non-spoken text utterances (320), each of which is not paired with any corresponding spoken utterance of the non-synthesized voice; untranscribed, non-synthesized speech utterances (306), each untranscribed, non-synthesized speech utterance (306) not paired with a corresponding transcription; Transcribed non-synthesized speech utterances (304), each transcribed non-synthesized speech utterance (304) paired with a corresponding transcription (302); receiving training data (304, 306, 320) including: generating a corresponding synthetic speech representation (332) for each non-spoken text utterance (320) of the received training data (304, 306, 320) using a text-to-speech model (330); pre-training the audio encoder (210) based on the non-speech text utterance (320), the untranscribed non-synthesized speech utterance (306), and the synthetic speech representation (332) generated for the transcribed non-synthesized speech utterance (304) to teach the audio encoder (210) to jointly learn a shared representation between speech and text; Including, Pre-training the audio encoder (210) For each untranscribed non-synthesized speech utterance (306), generating a corresponding coded representation (215) of the untranscribed, non-synthesized speech utterance (306); pre-training the audio encoder (210) based on contrastive losses (316) applied to the corresponding coded representations (215) of the untranscribed, non-synthetic speech utterances (306), wherein the contrastive losses applied to the corresponding coded representations (215) of the untranscribed, non-synthetic speech utterances (306) include contrastive losses (316) between the corresponding coded representations (215) of mask positions and a target context vector of the untranscribed, non-synthetic speech utterances (306); For each synthesized speech representation (332), generating a corresponding coded representation (215) of said synthetic speech representation (332); pre-training the audio encoder (210) based on contrastive losses (316) applied to the corresponding coded representations (215) of the synthetic speech representations (332), wherein the contrastive losses (316) applied to the corresponding coded representations (215) of the synthetic speech representations (332) include contrastive losses (316) between the corresponding coded representations (215) of mask positions and target context vectors of the synthetic speech representations (332); For each transcribed non-synthesized speech utterance (304), generating a corresponding coded representation (215) of the transcribed non-synthesized speech utterance (304); pre-training the audio encoder (210) based on a contrastive loss (316) applied to the corresponding coded representation (215) of the transcribed non-synthetic speech utterance (304), wherein the contrastive loss (316) applied to the corresponding coded representation (215) of the transcribed non-synthetic speech utterance (304) comprises a contrastive loss between the corresponding coded representation (215) of a mask position and a target context vector of the transcribed non-synthetic speech utterance (304); 10. The computer-implemented method (600) of claim 1, comprising:

4. the first probability distribution (392) over possible synthetic speech recognition hypotheses includes one of possible phoneme labels or possible wordpiece labels; the second probability distribution (394) over possible non-synthetic speech recognition hypotheses includes one of the possible phoneme labels or the possible wordpiece labels.

10. The computer-implemented method (600) of claim 1.

5. Pre-training the audio encoder (210) At each of the plurality of time steps of each synthesized speech representation (332), generating, using another auxiliary decoder (390), a third probability distribution (393) over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation (332), the third probability distribution (393) over possible synthetic speech recognition hypotheses including the other of the possible phoneme labels or the possible wordpiece labels; determining another synthetic speech loss term (342) based on the third probability distribution (393) over possible synthetic speech recognition hypotheses and the unspoken text utterance (320) corresponding to the corresponding synthetic speech representation (332); and pre-training the audio encoder (210) based on the different synthetic speech loss terms (342); At each of the plurality of time steps of each transcribed non-synthesized speech utterance (304), generating, using the further auxiliary decoder (390), a fourth probability distribution (395) over possible non-synthesized speech recognition hypotheses for the corresponding transcribed non-synthesized speech utterance (304), the fourth probability distribution (395) over possible non-synthesized speech recognition hypotheses including the other of the possible phoneme labels or the possible wordpiece labels; determining another non-synthetic speech loss term (344) based on the fourth probability distribution (395) over possible non-synthetic speech recognition hypotheses and the corresponding transcription (302) paired with the transcribed non-synthetic speech utterance (304); pre-training the audio encoder (210) based on the non-synthetic speech loss term (344); 5. The computer-implemented method (600) of claim 4, further comprising:

6. 10. The computer-implemented method of claim 1, wherein the auxiliary decoder comprises one of a Connection Temporal Classification (CTC) decoder, a Listen Attend Spell (LAS) decoder, or a Recurrent Neural Network-Transducer (RNN-T) decoder.

7. The operation is obtaining a set of training utterance pairs (301), each of which is a corresponding one of the transcribed non-synthesized speech utterances (304) of the received training data (304, 306, 320); and a pair of synthetic speech representations (334) of the corresponding transcribed non-synthetic speech utterances (304), the pair of synthetic speech representations (334) being generated by the text-to-speech model (330) performing text-to-speech conversion on the corresponding transcriptions (302) paired with the transcribed non-synthetic speech utterances (304); and further comprising obtaining the pre-training the audio encoder (210) includes, at each of a plurality of output steps for each training utterance pair (301) in the set of training utterance pairs (301), generating, using an auxiliary decoder (390), a first probability distribution (311) over possible non-synthesized speech recognition hypotheses for the corresponding transcribed non-synthesized speech utterance (304); generating, using an auxiliary decoder (390), a second probability distribution (394) over possible synthetic speech recognition hypotheses for the corresponding pair of synthetic speech representations (334); determining a consistency loss term (352) for the corresponding training utterance pairs (301) based on the first probability distribution (311) over possible non-synthetic speech recognition hypotheses and the second probability distribution (394) over possible synthetic speech recognition hypotheses; pre-training the audio encoder (210) based on the consistency loss term (352); 10. The computer-implemented method (600) of claim 1, comprising:

8. 2. The computer-implemented method of claim 1, wherein the operations further comprise augmenting one or more of the synthetic speech representations before pre-training the audio encoder based on the synthetic speech representations.

9. 10. The computer-implemented method of claim 1, wherein the non-spoken text utterances are generated and / or selected using one or more language models.

10. 10. The computer-implemented method of claim 1, wherein the non-spoken text utterances are generated using a background language model and a domain language model trained on transcribed speech utterances associated with the target domain.

11. 10. The computer-implemented method of claim 1, wherein the operations further comprise pre-training the audio encoder and then fine-tuning the pre-trained encoder on a transcribed speech utterance.

12. A system (100), data processing hardware (710); and memory hardware (720) in communication with the data processing hardware (710), the memory hardware (720) storing instructions that, when executed on the data processing hardware (710), cause the data processing hardware (710) to perform operations, the operations including: non-spoken text utterances (320), each of which is not paired with any corresponding spoken utterance of the non-synthesized voice; untranscribed, non-synthesized speech utterances (306), each untranscribed, non-synthesized speech utterance (306) not paired with a corresponding transcription; Transcribed non-synthesized speech utterances (304), each transcribed non-synthesized speech utterance (304) paired with a corresponding transcription (302); receiving training data (304, 306, 320) including: generating a corresponding synthetic speech representation (332) for each non-spoken text utterance (320) of the received training data (304, 306, 320) using a text-to-speech model (330); pre-training the audio encoder (210) based on the non-speech text utterance (320), the untranscribed non-synthesized speech utterance (306), and the synthetic speech representation (332) generated for the transcribed non-synthesized speech utterance (304) to teach the audio encoder (210) to jointly learn a shared representation between speech and text; Including, Pre-training the audio encoder (210) At each of a plurality of time steps of each synthesized speech representation (332), generating, using an auxiliary decoder (390), a first probability distribution (392) over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation (332); determining a synthetic speech loss term (342) based on the first probability distribution (392) over possible synthetic speech recognition hypotheses and the unspoken text utterance (320) corresponding to the corresponding synthetic speech representation (332); pre-training the audio encoder (210) based on the synthetic speech loss term (342); At each of a plurality of time steps of each transcribed non-synthesized speech utterance (304), generating, using the auxiliary decoder (390), a second probability distribution (394) over possible non-synthesized speech recognition hypotheses for the corresponding transcribed non-synthesized speech utterance (304); determining a non-synthetic speech loss term (344) based on the second probability distribution (394) over possible non-synthetic speech recognition hypotheses and the corresponding transcription (302) paired with the transcribed non-synthetic speech utterance (304); pre-training the audio encoder (210) based on the non-synthetic speech loss term (344); A system (100) including:

13. 13. The system of claim 12, wherein the audio encoder includes a stack of self-attention layers, each of which includes a multi-head self-attention mechanism.

14. A system (100), comprising: data processing hardware (710); and memory hardware (720) in communication with the data processing hardware (710), the memory hardware (720) storing instructions that, when executed on the data processing hardware (710), cause the data processing hardware (710) to perform operations, the operations including: non-spoken text utterances (320), each of which is not paired with any corresponding spoken utterance of the non-synthesized voice; untranscribed, non-synthesized speech utterances (306), each untranscribed, non-synthesized speech utterance (306) not paired with a corresponding transcription; Transcribed non-synthesized speech utterances (304), each transcribed non-synthesized speech utterance (304) paired with a corresponding transcription (302); receiving training data (304, 306, 320) including: generating a corresponding synthetic speech representation (332) for each non-spoken text utterance (320) of the received training data (304, 306, 320) using a text-to-speech model (330); pre-training the audio encoder (210) based on the non-speech text utterance (320), the untranscribed non-synthesized speech utterance (306), and the synthetic speech representation (332) generated for the transcribed non-synthesized speech utterance (304) to teach the audio encoder (210) to jointly learn a shared representation between speech and text; Including, Pre-training the audio encoder (210) For each untranscribed non-synthesized speech utterance (306), generating a corresponding coded representation (215) of the untranscribed, non-synthesized speech utterance (306); pre-training the audio encoder (210) based on contrastive losses (316) applied to the corresponding coded representations (215) of the untranscribed, non-synthetic speech utterances (306), wherein the contrastive losses applied to the corresponding coded representations (215) of the untranscribed, non-synthetic speech utterances (306) include contrastive losses (316) between the corresponding coded representations (215) of mask positions and a target context vector of the untranscribed, non-synthetic speech utterances (306); For each synthesized speech representation (332), generating a corresponding coded representation (215) of said synthetic speech representation (332); pre-training the audio encoder (210) based on contrastive losses (316) applied to the corresponding coded representations (215) of the synthetic speech representations (332), wherein the contrastive losses (316) applied to the corresponding coded representations (215) of the synthetic speech representations (332) include contrastive losses (316) between the corresponding coded representations (215) of mask positions and target context vectors of the synthetic speech representations (332); For each transcribed non-synthesized speech utterance (304), generating a corresponding coded representation (215) of the transcribed non-synthesized speech utterance (304); pre-training the audio encoder (210) based on contrastive losses (316) applied to the corresponding coded representations (215) of the transcribed non-synthetic speech utterances (304); the contrastive losses (316) applied to the corresponding coded representations (215) of the transcribed non-synthetic speech utterances (304) including the corresponding coded representations (215) of mask positions; A system (100) including:

15. the first probability distribution (392) over possible synthetic speech recognition hypotheses includes one of possible phoneme labels or possible wordpiece labels; the second probability distribution (394) over possible non-synthetic speech recognition hypotheses includes one of the possible phoneme labels or the possible wordpiece labels.

13. The system (100) of claim 12.

16. Pre-training the audio encoder (210) At each of the plurality of time steps of each synthetic speech representation (332), generating, using another auxiliary decoder (390), a third probability distribution (393) over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation (332), the third probability distribution (393) over possible synthetic speech recognition hypotheses including the other of the possible phoneme labels or the possible wordpiece labels; determining another synthetic speech loss term (342) based on the third probability distribution (393) over possible synthetic speech recognition hypotheses and the unspoken text utterance (320) corresponding to the corresponding synthetic speech representation (332); and pre-training the audio encoder (210) based on the different synthetic speech loss terms (342); At each of the plurality of time steps of each transcribed non-synthesized speech utterance (304), generating, using the further auxiliary decoder (390), a fourth probability distribution (395) over possible non-synthesized speech recognition hypotheses for the corresponding transcribed non-synthesized speech utterance (304), the fourth probability distribution (395) over possible non-synthesized speech recognition hypotheses including the other of the possible phoneme labels or the possible wordpiece labels; determining another non-synthetic speech loss term (344) based on the fourth probability distribution (395) over possible non-synthetic speech recognition hypotheses and the corresponding transcription (302) paired with the transcribed non-synthetic speech utterance (304); pre-training the audio encoder (210) based on the non-synthetic speech loss term (344); 16. The system (100) of claim 15, further comprising:

17. 13. The system of claim 12, wherein the auxiliary decoder comprises one of a Connection Temporal Classification (CTC) decoder, a Listen Attend Spell (LAS) decoder, or a Recurrent Neural Network-Transducer (RNN-T) decoder.

18. The operation is obtaining a set of training utterance pairs (301), each of which is a corresponding one of the transcribed non-synthesized speech utterances (304) of the received training data (304, 306, 320); and a pair of synthetic speech representations (334) of the corresponding transcribed non-synthetic speech utterances (304), the pair of synthetic speech representations (334) being generated by the text-to-speech model (330) performing text-to-speech conversion on the corresponding transcriptions (302) paired with the transcribed non-synthetic speech utterances (304); and further comprising obtaining the pre-training the audio encoder (210) includes, at each of a plurality of output steps for each training utterance pair (301) in the set of training utterance pairs (301), generating, using an auxiliary decoder (390), a first probability distribution (311) over possible non-synthesized speech recognition hypotheses for the corresponding transcribed non-synthesized speech utterance (304); generating, using an auxiliary decoder (390), a second probability distribution (394) over possible synthetic speech recognition hypotheses for the corresponding pair of synthetic speech representations (334); determining a consistency loss term (352) for the corresponding training utterance pairs (301) based on the first probability distribution (311) over possible non-synthetic speech recognition hypotheses and the second probability distribution (394) over possible synthetic speech recognition hypotheses; pre-training the audio encoder (210) based on the consistency loss term (352); 13. The system (100) of claim 12, further comprising:

19. 13. The system (100) of claim 12, wherein the operations further include augmenting one or more of the synthetic speech representations (332) before pre-training the audio encoder (210) based on the synthetic speech representations (332).

20. 13. The system (100) of claim 12, wherein the non-spoken text utterances (320) are generated and / or selected using one or more language models (404, 406).

21. 13. The system of claim 12, wherein the non-spoken text utterances are generated using a background language model and a domain language model trained on transcribed speech utterances associated with the target domain.

22. 14. The system (100) of claim 13, wherein the operations further include, after pre-training the audio encoder (210), fine-tuning the pre-trained encoder (210) on a transcribed speech utterance (304).

23. A computer program for causing a computer to execute the computer-implemented method described in any one of claims 1 to 11.

Images