Text Insertion in Self-Supervised Speech Pre-Training

Abstract

The method (500) includes receiving training data including non-spoken text utterances (320) and non-transcribed non-synthetic speech utterances (306). Each non-spoken text utterance is not paired with any corresponding spoken utterance of non-synthetic speech. Each non-transcribed non-synthetic speech utterance is not paired with a corresponding transcription. The method also includes generating a corresponding synthetic speech representation (332) for each non-spoken text utterance of the received training data using a text-to-speech model (330). The method also includes pre-training an audio encoder (210) on the generated synthetic speech representations for the non-spoken text utterances and the non-transcribed non-synthetic speech utterances to teach the audio encoder to jointly learn the shared speech and text representations.

Description

[Technical Field]

[0001] The present disclosure relates to text insertion in self-supervised speech pre-training. [Background technology]

[0002] Automatic speech recognition (ASR), the process of taking audio input and transcribing it into text, has become a very important technology used in mobile devices and other devices. Generally, automatic speech recognition attempts to provide an accurate transcription of what a person said by taking audio input (e.g., a voice utterance) and transcribing the audio input into text. Based on the ongoing development of deep neural networks, modern ASR models continue to improve in both accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay between a user's utterance and the transcription). However, one challenge in developing deep learning-based ASR models is that the parameters of the ASR model tend to overfit to the training data, which causes the ASR model to have difficulty generalizing to unseen data when the training data is not sufficiently large. As a result, training an ASR model with a relatively large training dataset improves the accuracy of the ASR model. To increase the amount of training data used to train an ASR model, synthetic speech and / or data-augmented speech can be incorporated. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for pre-training an audio encoder to jointly learn a shared representation of speech and text. The operations include receiving training data including non-spoken text utterances and untranscribed non-synthetic speech utterances. Each non-spoken text utterance is not paired with any corresponding spoken utterance of the non-synthetic speech. Each untranscribed non-synthetic speech utterance is not paired with a corresponding transcription. The operations also include generating a corresponding synthetic speech representation of each non-spoken text utterance in the received training data using a text-to-speech model. The operations also include pre-training the audio encoder based on the synthetic speech representations generated for the non-spoken text utterances and the untranscribed non-synthetic speech utterances to teach the audio encoder to jointly learn the shared speech and text representations.

[0004] Implementations of this disclosure may include one or more of the following optional features: In some implementations, the audio encoder includes a stack of self-attention layers, each including a multi-head self-attention mechanism; In some examples, pre-training the audio encoder includes: for each untranscribed, non-synthetic speech utterance, generating a corresponding coded representation of the untranscribed speech representation and pre-training the audio encoder based on a contrastive loss applied to the corresponding coded representation of the untranscribed, non-synthetic speech utterance; and for each synthetic speech representation, generating a corresponding coded representation of the synthetic speech representation and pre-training the audio encoder based on a contrastive loss applied to the corresponding coded representation of the synthetic speech representation.

[0005] In some implementations, pre-training the audio encoder includes, at each of a plurality of time steps of each synthetic speech representation, generating, using an auxiliary decoder, a first probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation, determining a synthetic speech loss term based on the first probability distribution over the possible synthetic speech recognition hypotheses and a non-spoken text utterance corresponding to the corresponding synthetic speech representation, and pre-training the audio encoder based on the synthetic speech loss term, where the first probability distribution over the possible synthetic speech recognition hypotheses may include one of possible phoneme labels or possible wordpiece labels.

[0006] In these implementations, pre-training the audio encoder may further include, at each of a plurality of time steps of each synthetic speech representation, generating, using another auxiliary decoder, a second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation, including the other of the possible phoneme labels or the possible wordpiece labels, determining another synthetic speech loss term based on the second probability distribution over the possible synthetic speech recognition hypotheses and the non-spoken text utterance corresponding to the corresponding synthetic speech representation, and pre-training the audio encoder based on the other synthetic speech loss term. The auxiliary decoder and the other auxiliary decoder may each include one of a Connection Temporal Classification (CTC) decoder, a Listen Attend Spell (LAS) decoder, or a Recurrent Neural Network-Transducer (RNN-T) decoder.

[0007] In some examples, the operations further include augmenting one or more of the synthetic speech representations before pre-training the audio encoder based on the synthetic speech representations. Here, pre-training the audio encoder based on the synthetic speech representations may include pre-training the audio encoder based on the augmented one or more synthetic speech representations. The non-spoken text utterances may be generated and / or selected using one or more language models.

[0008] In some implementations, the non-speech text utterances are generated using a background language model and a domain language model trained on transcribed speech utterances associated with the target domain. Additionally or alternatively, the operations may further include pre-training the audio encoder and then fine-tuning the pre-trained audio encoder on the transcribed speech utterances.

[0009] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving training data including non-spoken text utterances and untranscribed, non-synthetic speech utterances. Each non-spoken text utterance is not paired with any corresponding speech utterance of the non-synthetic speech. Each untranscribed, non-synthetic speech utterance is not paired with a corresponding transcription. The operations also include generating a corresponding synthetic speech representation of each non-spoken text utterance in the received training data using a text-to-speech model. The operations also include pre-training an audio encoder based on the generated synthetic speech representation for the non-spoken text utterance and the untranscribed, non-synthetic speech utterance to teach the audio encoder to jointly learn the shared speech and text representations.

[0010] This aspect of the present disclosure may include one or more of the following optional features: In some implementations, the audio encoder includes a stack of self-attention layers, each including a multi-head self-attention mechanism. In some examples, pre-training the audio encoder includes: for each untranscribed, non-synthetic speech utterance, generating a corresponding coded representation of an untranscribed speech representation and pre-training the audio encoder based on a contrastive loss applied to the corresponding coded representation of the untranscribed, non-synthetic speech utterance; and for each synthetic speech representation, generating a corresponding coded representation of a synthetic speech representation and pre-training the audio encoder based on a contrastive loss applied to the corresponding coded representation of the synthetic speech representation.

[0011] In some implementations, pre-training the audio encoder includes, at each of a plurality of time steps of each synthetic speech representation, generating, using an auxiliary decoder, a first probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation, determining a synthetic speech loss term based on the first probability distribution over the possible synthetic speech recognition hypotheses and a non-spoken text utterance corresponding to the corresponding synthetic speech representation, and pre-training the audio encoder based on the synthetic speech loss term, where the first probability distribution over the possible synthetic speech recognition hypotheses may include one of possible phoneme labels or possible wordpiece labels.

[0012] In these implementations, pre-training the audio encoder may further include, at each of a plurality of time steps of each synthetic speech representation, generating, using another auxiliary decoder, a second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation, including the other of the possible phoneme labels or the possible wordpiece labels, determining another synthetic speech loss term based on the second probability distribution over the possible synthetic speech recognition hypotheses and the non-spoken text utterance corresponding to the corresponding synthetic speech representation, and pre-training the audio encoder based on the other synthetic speech loss term. The auxiliary decoder and the other auxiliary decoder may each include one of a Connection Temporal Classification (CTC) decoder, a Listen Attend Spell (LAS) decoder, or a Recurrent Neural Network-Transducer (RNN-T) decoder.

[0013] In some examples, the operations further include augmenting one or more of the synthetic speech representations before pre-training the audio encoder based on the synthetic speech representations. Here, pre-training the audio encoder based on the synthetic speech representations may include pre-training the audio encoder based on the augmented one or more synthetic speech representations. The non-spoken text utterances may be generated and / or selected using one or more language models.

[0014] In some implementations, the non-speech text utterances are generated using a background language model and a domain language model trained on transcribed speech utterances associated with the target domain. Additionally or alternatively, the operations may further include pre-training the audio encoder and then fine-tuning the pre-trained audio encoder on the transcribed speech utterances.

[0015] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a schematic diagram of an exemplary speech recognition system. [Figure 2] FIG. 1 is a schematic diagram of a recurrent neural network transducer (RNN-T) model architecture. [Figure 3A] FIG. 1 is a schematic diagram illustrating an example training process for pre-training an audio encoder for a speech recognition model. [Figure 3B] FIG. 1 is a schematic diagram illustrating an example training process for pre-training an audio encoder for a speech recognition model. [Figure 4] FIG. 1 is a schematic diagram illustrating an exemplary non-spoken text selection process for selecting non-spoken text utterances related to a particular domain. [Figure 5] 1 is a flowchart illustrating an example arrangement of operations for a method for pre-training an audio encoder to jointly learn a shared representation of speech and text. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0017] Like reference symbols in the various drawings indicate like elements.

[0018] Automated speech recognition has made tremendous progress with the introduction of sequence-to-sequence (Seq2Seq) models that map from audio to character sequences. At the same time, text-to-speech (TTS) or speech synthesis systems have successfully applied Seq2Seq models to obtain state-of-the-art natural, realistic-sounding synthesized speech that may be indistinguishable from human speech to the human ear.

[0019] One challenge in developing deep learning-based ASR models is that their parameters tend to overfit to the training data, which can cause the ASR model to have difficulty generalizing to unknown data when the training data is not sufficiently large. Therefore, training an ASR model with a relatively large training dataset improves its accuracy. For example, by using machine learning or other statistical methods, an ASR model can be trained with a training dataset containing 10,000 hours or more of transcribed speech. However, when the domain associated with the training data differs from the domain in which the ASR model is deployed during inference, the performance of the ASR model degrades. For example, training an ASR model with speech transcribed in a domain associated with videoconferencing can be less effective at recognizing speech related to voice search queries, and vice versa.

[0020] Synthetic speech has the potential to drastically limit the amount of labeled human speech required to train an ASR model, while also providing flexibility for moving ASR models between different domains. Although state-of-the-art synthetic speech examples are generally indistinguishable from human speech, the use of synthetic speech has been shown to impact ASR training differently than human speech. This gap between synthetic and human speech stems from the mismatch between human and synthetic speech data, which arises from the challenging one-to-many mapping problem that TTS systems attempt to solve. That is, while the overall quality of available synthetic speech is very high, synthetic speech has much less variability than human speech, minimizing speech artifacts. As a result, training ASR models exclusively on synthetic speech data makes it difficult to generalize to real-world speech utterances during inference.

[0021] Implementations herein are directed to using synthetic speech to train an ASR model to recognize speech when a large amount of transcribed speech (e.g., non-synthetic speech) in a target domain and / or language for training the ASR model is unavailable or not widely available, in order to maintain the accuracy of the ASR model. More specifically, implementations are directed to pre-training an audio encoder of the ASR model with training data including untranscribed non-synthetic speech utterances, non-spoken text utterances for generating corresponding synthetic speech representations, and transcribed non-synthetic speech utterances for jointly learning speech and text representations, and then fine-tuning (e.g., warm-start training) the pre-trained ASR model using available transcribed non-synthetic speech utterances. As will become apparent, pre-training the audio encoder includes updating parameters of the audio encoder based on a combination of contrastive self-supervised loss, supervised loss, and consistency loss derived from the training data.

[0022] A contrastive self-supervised loss may be derived from latent speech representations generated by an audio encoder from corresponding untranscribed non-synthetic speech utterances, synthetic speech representations, and transcribed non-synthetic speech utterances to facilitate language learning. Meanwhile, a supervised loss may be derived from speech recognition labels predicted by one or more auxiliary decoders based on latent speech representations generated by an audio encoder from corresponding synthetic speech representations and transcribed non-synthetic speech utterances. Here, the corresponding transcriptions paired with the transcribed non-synthetic speech utterances and the corresponding non-spoken text utterances used to generate the synthetic speech representations serve as ground truth labels for deriving the supervised loss. Finally, a consistency loss may be derived from each transcribed non-synthetic speech utterance and its corresponding synthetic speech representation to facilitate consistent predictions (e.g., latent speech representations) by an audio encoder for both non-synthetic speech representations (e.g., real / human) and synthetic speech representations of the same utterance. That is, consistency loss between human (non-synthetic) and synthetic (synthetic) representations of the same utterance provides an aspect of unsupervised training by encouraging audio encoders to behave consistently across human and synthetic training utterances. In particular, a text-to-speech (TTS) model can convert the corresponding transcription paired with each transcribed non-synthetic speech utterance into a corresponding synthetic speech representation of the same utterance.

[0023] Additional implementations include applying data augmentation techniques, such as synthesizing diverse realizations of training utterances by varying the characteristics of the synthesized speakers to promote robustness to speaker differences. The techniques described herein are particularly useful when relatively little transcribed human speech is available in the target domain and / or language.

[0024] 1 illustrates an automated speech recognition (ASR) system 100 that implements an ASR model 200 that resides on a user device 102 of a user 104 and / or on a remote computing device 201 (e.g., one or more servers of a distributed system executing in a cloud computing environment) that communicates with the user device 102. While the user device 102 is shown as a mobile computing device (e.g., a smartphone), the user device 102 may correspond to any type of computing device, such as, but not limited to, a tablet device, a laptop / desktop computer, a wearable device, a digital assistant device, a smart speaker / display, a smart appliance, an in-vehicle infotainment system, or an Internet of Things (IoT) device, and comprises data processing hardware 111 and memory hardware 113.

[0025] The user device 102 includes an audio subsystem 108 configured to receive utterances 106 spoken by the user 104 (e.g., the user device 102 may include one or more microphones for recording the spoken utterances 106) and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the ASR system 100. In the illustrated example, the user speaks each utterance 106 in natural English for the phrase "What is the weather in New York City?", and the audio subsystem 108 converts the utterance 106 into a corresponding acoustic frame 110 for input to the ASR system 100. The ASR model 200 then receives the acoustic frame 110 corresponding to the utterance 106 as input and generates / predicts a corresponding transcription 120 (e.g., a recognition result / hypothesis) of the utterance 106 as output. In the illustrated example, the user device 102 and / or the remote computing device 201 also execute a user interface generator 107 configured to present a representation of a transcription 120 of the utterance 106 to the user 104 of the user device 102. In some configurations, the transcription 120 output from the ASR system 100 is processed by a natural language understanding (NLU) module executing on the user device 102 or the remote computing device 201, for example, to execute user commands. Additionally or alternatively, a text-to-speech system (e.g., executing on any combination of the user device 102 or the remote computing device 201) may convert the transcription into synthesized speech for audible output by another device. For example, the original utterance 106 may correspond to a message the user 104 is sending to a friend, where the transcription 120 is converted into synthesized speech for audible output to the friend, who should hear the message conveyed in the original utterance 106.

[0026] Referring to FIG. 2, an exemplary frame alignment-based transducer model 200a includes a recurrent neural network transducer (RNN-T) model architecture that adheres to latency constraints associated with interactive applications. The use of the RNN-T model architecture is exemplary, and the frame alignment-based transducer model 200 may include other architectures, such as transformer-transducer and conformer-transducer model architectures, among others. The RNN-T model 200 enables a small computational footprint and utilizes fewer memory requirements than traditional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device 102 (e.g., without requiring communication with a remote server). The RNN-T model 200 includes an encoder network 210, a prediction network 220, and a collaboration network 230. The encoder network 210 is generally similar to the acoustic model (AM) of traditional ASR systems and includes a stack of self-attention layers (e.g., conformer or transformer layers) or a recurrent network of stacked long short-term memory (LSTM) layers. For example, the encoder may generate a sequence of d-dimensional feature vectors (e.g., acoustic frame 110 (FIG. 1)) x=(x1, x2,..., x T ) is read, and in the formula,

[0027]

number

[0028] and at each output step, we generate a higher-level representation, which is

[0029]

number

[0030] It is written as follows.

[0031] Similarly, the prediction network 220 is also an LSTM network, which, like a language model (LM), calculates the sequence of non-null symbols output so far by the final softmax layer 240, i.e., y 0 , ....., y ui-1 A dense representation of

[0032]

number

[0033] Finally, using the RNN-T model architecture, the representations generated by the encoder and predictor / decoder networks 210, 220 are combined by the combination network 230. The prediction network 220 can be replaced with an embedding lookup table to improve latency by outputting a sparse embedding that is looked up instead of processing a dense representation. The combination network then

[0034]

number

[0035] , which is a distribution over the next output symbol. In other words, at each output step (e.g., time step), the collaborative network 230 generates a probability distribution over possible speech recognition hypotheses. Here, a "possible speech recognition hypothesis" corresponds to a set of output labels, each representing a symbol / character in a specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters of the English alphabet, and one label specifying a space. Thus, the collaborative network 230 may output a set of values ​​indicating the likelihood of each occurrence of a given set of output labels. This set of values ​​may be a vector and may indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels may include wordpieces and / or entire words in addition to, or instead of, graphemes. The output distribution of the collaborative network 230 may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output y of the collaborative network 230 i may contain 100 different probability values, one for each output label. The probability distribution may thus be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and / or words) in a beam search process (e.g., by softmax layer 240) to determine transcription 120.

[0036] The softmax layer 240 may utilize any technique for selecting the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200 at the corresponding output step. In this manner, the RNN-T model 200 does not make a conditional independence assumption; rather, the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model 200 assumes that the output symbols are independent of future acoustic frames 110, which allows the RNN-T model to be used in a streaming manner.

[0037] In some examples, the encoder network (i.e., audio encoder) 210 of the RNN-T model 200 includes a stack of self-attention layers / blocks, such as conformer blocks. Here, each conformer block includes a series of multi-head self-attention, depthwise convolutional, and feedforward layers. The prediction network 220 may have two 2,048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. Alternatively, the prediction network 220 may include a stack of transformer or conformer blocks, or an embedded lookup table, instead of an LSTM layer. Finally, the joint network 230 may also have 640 hidden units. The softmax layer 240 may consist of a joint word piece or grapheme set, generated using all unique word pieces or graphemes in multiple training datasets.

[0038] 3A and 3B illustrate an example training process 300 for pre-training the audio encoder 210 of the ASR model 200 (FIG. 2). The training process 300 involves pre-training a non-spoken text utterance (X text ) 320 sets, and untranscribed non-synthesized speech utterances (X unsupThe audio encoder 210 can be pre-trained using available training data, including a set of non-spoken training text utterances 320. Each non-spoken training text utterance 320 includes text-only data (i.e., unpaired data) such that each non-spoken training text utterance 320 is not paired with any corresponding spoken audio representation (speech) of the utterance. Each untranscribed, unsynthesized speech utterance 306 (also simply referred to as an "untranscribed speech utterance 306") includes audio-only data (i.e., unpaired data) such that each untranscribed speech utterance 306 is not paired with any corresponding transcription.

[0039] For simplicity, the training process 300 includes a contrastive self-supervised loss portion 300a (FIG. 3A) and a supervised loss portion 300b (FIG. 3B). The training process 300 is performed by training non-spoken text utterances (X text ) 320, and untranscribed non-synthesized speech utterances (X unsup ) 306 using the contrastive self-supervised loss unit 300a. w2v ) 316, as well as non-spoken text utterances (X text ) 320 using the supervised loss unit 300b. aux )342 based unpaired data loss J unpaired The audio encoder 210 is pre-trained based on the

[0040] 3A, the contrastive self-supervised loss portion 300a of the training process 300 may employ a text-to-speech (TTS) system 330 configured to generate, at each of a plurality of output steps, a synthetic speech representation (e.g., synthetic speech) 332 for each of a plurality of non-spoken training text utterances 320. The non-spoken training text utterances 320 (also referred to simply as "non-spoken text utterances 320") may be generated by generating a synthetic speech representation (e.g., synthetic speech) 332 for each of a plurality of non-spoken text utterances (e.g., X text) 320 includes non-spoken text, which is text-only data, i.e., unpaired data, such that the non-spoken text utterances 320 are not paired with any synthetic or unsynthetic speech. Accordingly, the TTS system 330 generates a corresponding synthetic speech representation 332 for each non-spoken text utterance 320. In particular, the synthetic speech representation 332 may include mel-frequency spectrogram frames for training the audio encoder 210, thereby eliminating the need for the training process 300 to include a vocoder and / or synthesizer to synthesize the mel-frequency spectrogram frames into synthetic speech.

[0041] The TTS system 330 may apply the speaker embedding z when converting the non-spoken text utterance 320 to generate a synthesized speech representation 332 having a particular speaking style and prosody associated with the speaker embedding. The TTS system 330 may apply multiple different speaker embeddings z, each associated with a different speaker characteristic of the resulting utterance of the synthesized synthesized speech representation 332. Similarly, the TTS system 330 may vary the prosody and other production qualities of the synthesized utterance.

[0042] In some examples, the training process 300 applies data augmentation to at least one of the sample utterances of the synthetic speech representation 332. Data augmentation may include, but is not limited to, adding noise, manipulating timing (e.g., stretching), or adding reverberation to the corresponding speech representation. Data augmentation may add different synthesis recording conditions to the synthetic speech representation 332.

[0043] This pre-training batch generation process for generating sample utterances for synthetic speech representation 332 advantageously samples new speaker and prosodic conditioning values ​​each time a non-spoken text utterance 320 is observed during training, resulting in a variety of synthetic utterances in subsequent observations. Each batch therefore contains both synthetic and real (non-synthetic) utterances. The loss contribution can be masked by using a loss mask σ (see Equation 4), so that the loss is calculated for the appropriate batch elements.

[0044] In some examples, the audio encoder 210 includes a stack of self-attention layers, each including a multi-head self-attention mechanism. For example, the stack of self-attention layers may include a stack of conformer layers or transformer layers. In the illustrated example, the audio encoder 210 includes a conformer encoder including a stack of conformer blocks, each of which includes a series of multi-head self-attention layers, a depthwise convolutional layer, and a feedforward layer. The conformer encoder 210 can naturally be divided into a feature encoder including a convolutional subsampling block 212 and a context network including a stack of linear layers 214 and conformer blocks 216. In some implementations, the convolutional subsampling block 212 has two 2D convolutional layers, both with stride (2, 2), resulting in a quarter-fold reduction in feature sequence length. The convolutional subsampling block 212 receives as input a sequence of input features / vectors (e.g., Mel-frequency spectrograms such as acoustic frame 110 in FIG. 1 ) associated with each synthesized speech representation 332 and each untranscribed, non-synthesized speech utterance 306, and generates as output, for each of a plurality of output steps, coding features (also referred to as “coding representations”) 211 corresponding to one of the synthesized speech representations 332 or one of the untranscribed, non-synthesized speech utterances 306, respectively.

[0045] The encoding features 211 output from the convolutional subsampling block 212 are supplied to a masking module 218, where a portion of the encoding features 211 is randomly selected and replaced with a trained feature vector shared among all masked time steps to provide corresponding masked encoding features 211m. In some examples, the masking module 218 masks the randomly selected encoding features 211 for masking by randomly sampling a percentage p of all time steps without replacement as a starting index, and then masking the subsequent M consecutive time steps from all sample indexes, which may result in some overlapping spans. After masking is applied, the linear layer 214 of the context network and the conformer block 216 receive the masked encoding features 211m and output a corresponding contrastive context vector 215 from the masked encoding features 211m. Furthermore, a quantizer 217 receives the encoding features 211 as input and generates a quantized vector (i.e., a target context vector) 219 as output. Then, the contrastive loss module 315 calculates the contrastive loss (L w2v )316 is derived as follows:

[0046]

number

[0047] In the formula, c t is the contrastive context vector 215 centered at the masked time step t, and q t , q t represents the target context vector 219 at time step t in a set of K+1 candidate target context vectors 219 containing K distractors. The distractors may be uniformly sampled from other masked time steps of the same utterance.

[0048] The contrastive loss 316 is optimized between the contrastive context vector 215 at the mask location and the target context vector 219. After the pre-trained audio encoder 210 converges on the untranscribed, non-synthetic speech utterance 306, the pre-training procedure is repeated on the synthetic speech representation 332. Thus, the contrastive loss 316 is optimized for both real / human (non-synthetic) features and synthetic (TTS audio) features, with an additional auxiliary loss for the synthetic speech representation 332, as described in further detail below with reference to FIG. 3B. Thus, the training process 300 pre-trains the audio encoder 210 based on each synthetic speech representation 332 provided as input to the audio encoder 210 and the derived contrastive loss 316 applied to the corresponding coding feature 211 associated with each untranscribed, non-synthetic speech utterance 306. Pre-training the audio encoder 210 may include updating parameters of the audio encoder based on the contrastive loss.

[0049] Referring to FIG. 3B , the supervised loss portion 300b of the training process 300 is configured to inject lexical information into the audio encoder 210 during pre-training based on supervised loss terms 342 derived from synthetic speech representations 332 generated by a TTS system 330 for non-spoken text utterances 320. In particular, the supervised loss portion 300b utilizes one or more auxiliary decoders 390 to generate the supervised loss terms 342. The auxiliary decoders 390 may include a Connectionist Temporal Classification (CTC) decoder, a Listen Attend Spell (LAS) decoder, or an RNN-T decoder. These auxiliary decoders 390 may include at least one of a phoneme decoder configured to decode a sequence of phonemes or a wordpiece decoder configured to decode a sequence of wordpieces. The auxiliary decoder 390 may also include a grapheme decoder configured to decode a sequence of graphemes. In some examples, the training process 300 applies data augmentation to at least one of the sample utterances of the synthetic speech representation 332 to provide one or more lexically diverse synthetic speech representations 332 for a given non-speech training text utterance 320. Data augmentation may include, but is not limited to, adding noise, manipulating timing (e.g., stretching), or adding reverberation to the corresponding speech representation. Data augmentation may add different synthesis recording conditions to the synthetic speech representation 332.

[0050] During the supervised loss portion 300b, the audio encoder 210 receives as input each synthetic speech representation 332 generated from the non-spoken text utterance 320 as a sequence of features / vectors (e.g., Mel-frequency spectrograms such as the acoustic frames 110 of FIG. 1 ), and for each of a plurality of time steps generates a first encoded representation (e text) 312 as an output. An auxiliary decoder 390, including a phoneme decoder or wordpiece decoder, receives as input each first coded representation 312 output from the audio encoder 310 and generates as an output a first probability distribution 392 over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation 332 at the corresponding time step. In some examples, the first probability distribution 392 over the possible synthetic speech recognition hypotheses includes one of possible phoneme labels or possible wordpiece labels. The supervised loss module 340 may then determine a synthetic speech loss term 342 based on the first probability distribution 392 over the possible synthetic speech recognition hypotheses and the corresponding non-spoken text utterance 320. Here, the corresponding non-spoken text utterance 320 from which the synthetic speech representation 332 is generated also serves as a ground truth transcription. The supervised loss unit 300b may pre-train the audio encoder 210 based on the synthetic speech loss term 342 by updating parameters of the audio encoder 210.

[0051] In some implementations, the supervised loss portion 300b of the training process 300 uses a separate auxiliary decoder 390 to decode the first coded representation (e text ) 312, generates a second probability distribution 393 over the possible synthetic speech recognition hypotheses, whereby the supervised loss module 340 determines another synthetic speech loss term 342 based on the second probability distribution and the unspoken text utterance 320 corresponding to the synthetic speech representation, where the other auxiliary decoder 390 includes the other of a phoneme decoder or a wordpiece decoder, and the second probability distribution 393 over the possible synthetic speech recognition hypotheses includes the other of the possible phoneme labels or the possible wordpiece labels.

[0052] The untranscribed non-synthesized speech utterance 306 and the non-spoken text utterance 320 are each a non-spoken text utterance (X text )320 derived from the contrast loss (L w2v ) 316 (FIG. 3A) as the supervised loss J associated with the synthetic speech loss term 342aux Combined with this, we use the non-spoken text loss function J as follows: text corresponds to the "unpaired" training data from which J text =L w2v (x│θ e )+L aux (y│x,θ e ,θ_d) (2) Similarly, untranscribed non-synthesized speech utterances (X unsup )306 derived from the contrast loss (L w2v ) 316 (Figure 3A) to calculate the unsupervised speech loss function, J unsup_speech can be expressed as follows: J unsup_speech = J w2v (x*│θ e ) (3)

[0053] During pre-training of the audio encoder 210, synthetic speech representations 332 and untranscribed non-synthetic speech utterances 306 are mixed within each batch. To force the audio encoder 210 to learn representations that are valid for both synthetic and non-synthetic (human / real) speech, the loss function J in Equation 2 and Equation 3 is text and J. unsup_speech When combining, a loss mask σ is applied to the unpaired data loss function J unpaired is obtained as follows: J unpaired =σ J text +(1-σ) J unsup_speech (4) Finally, the training process 300 updates the parameters of the audio encoder 210 to obtain the unpaired data loss function J unpairedThe audio encoder 210 may be pre-trained using the training process 300, effectively teaching the audio encoder 210 to learn a shared representation between speech and text. After pre-training the audio encoder 210, the training process 300 may fine-tune the pre-trained audio encoder on transcribed speech utterances, which may include both synthetic (e.g., synthetic speech) and non-synthetic (e.g., human speech) supervised training samples.

[0054] In some implementations, the training process 300 for pre-training the audio encoder 210 applies encoder consistency regularization. Unlike decoder consistency regularization, which is applied to the auxiliary decoder during the consistency regularization section 300c, which requires hypothesis labels (e.g., non-spoken text utterances 320), encoder consistency regularization has the advantage of not requiring hypothesis labels and therefore being applicable to all training data 306, 320. Encoder consistency regularization can be applied via a hierarchical contrastive consistency regularization (HCCR) technique, in which encoder activations e, e* from the original / unaugmented and augmented speech are projected through an auxiliary network to generate z, z*. Then, positive-negative pairs are constructed and a contrastive loss l is applied. t,z,z* is calculated as follows:

[0055]

number

[0056] Specific to HCCR, a convolutional neural network (CNN) projection network computes projections of encoder activations e in increasing length segments (30, 50, 120 ms) to generate three views (V), drawing negative examples from the same utterance in the shorter segments and from other utterances in the batch with 120 ms segments. Thus, for a synthetic speech representation (synthetic speech) generated from an untranscribed non-synthetic speech utterance 306 (unpaired speech) and an unspoken text utterance 320, the HCCR loss can be computed as follows:

[0057]

number

[0058] The HCCR loss calculated by Equation 6 may be added to Equation 4 by a factor of 1e-3 as part of the unpaired data loss function for use in pre-training the audio encoder 210.

[0059] 4, a contrastive non-spoken text selection process 400 can select non-spoken text utterances 320 from a large non-spoken text corpus 402 to be used for pre-training the audio encoder 210, such that the selected non-spoken text utterances 320 are most similar to the particular domain that the audio encoder 210 is being pre-trained to learn from. That is, the text selection process 400 can identify non-spoken text within and near a domain from the non-spoken text corpus 402 for inclusion in the non-spoken text utterances 320 to be used for pre-training the audio encoder 210. In particular, the non-spoken text utterances 320 selected by the text selection process 400 allow different utterances to be synthesized on the fly during batch construction, such that each time a non-spoken text utterance 320 is in a batch, a new speaker embedding z and the latent variable Z can be sampled.

[0060] The corpus of non-spoken text 402 includes a large number of non-spoken training text utterances 320, 320a-n from across a wide range of domains, encompassing a much greater linguistic diversity than the specific domain for which the audio encoder 210 is trained to learn. The corpus of non-spoken text 402 may be stored in the same or a different data store 401 as the spoken transcribed non-synthesized speech utterances (i.e., training utterances) 304, so as to belong to the specific domain for which the audio encoder 210 is trained to learn. Each spoken transcribed non-synthesized speech utterance 304 is paired with a corresponding transcription 302. The corpus of non-spoken text 402 may change dynamically to incorporate new non-spoken text utterances 320. Simply using all the non-spoken text utterances 320 in the non-spoken text corpus 402 is not feasible for the following reasons: i) for each sentence, the speech modality requires much more memory to be encoded than the text, thereby making it infeasible to convert all the text in the non-spoken text corpus 402; ii) the enormous amount of difference between the transcriptions 302 paired with the transcribed non-synthesized speech utterances 304 and the non-spoken text utterances 320 in the non-spoken text corpus 402 requires an intelligent strategy to balance their contributions.

[0061] The text selection process 400 aims to select a subset of available non-spoken text utterances 320 from a non-spoken text corpus 402 as data for TTS synthesis, resulting in the synthetic speech representations 332 generated for pre-training the audio encoder 210 during the contrastive loss and supervised loss portions 300a, 300b of the training process 300 described above with reference to Figures 3A and 3B. In other words, the text selection process 400 aims to improve the match between the selected subset of available non-spoken text utterances 320 and the specific domain being targeted, thereby, in turn, reducing the computational resources required to leverage large amounts of non-domain-specific data. Thus, the text selection process 400 reduces computational and memory costs by selecting non-spoken text utterances 320 that best match the specific domain that the audio encoder 210 is trained to learn.

[0062] In some examples, the text selection process 400 selects a subset of available non-spoken text utterances 320 from the non-spoken text corpus 402 that best matches a particular region by simply providing a region identifier (not shown) associated with the particular region as input to a background LM 406 that has been pre-trained across the entire non-spoken text corpus 402. As previously mentioned, the non-spoken text corpus 402 spans a number of different regions. In these examples, the background LM 406 may include a maximum entropy (MaxEnt LM) that can optionally accept a region identifier as input, as described in U.S. Patent No. 9,842,592, filed February 12, 2014, the entire contents of which are incorporated herein by reference. Here, the region identifier associated with a particular region may enable the MaxEnt LM to output a subset of available non-spoken text utterances 320 from the non-spoken text corpus 402 that may contain words and / or phrases related to the particular region. In some configurations, rather than assessing word likelihood, the statistical language model operates in reverse mode to randomly generate text phrases that match the statistical distribution of words associated with a particular domain.

[0063] In a further example, and as shown in FIG. 4 , a text selection process 400 uses transcriptions 302 paired with transcribed non-synthesized speech utterances 304 spoken by a human speaker to select a subset of available non-spoken text utterances 320 from the non-spoken text corpus 402 that best matches a particular domain, where the transcribed non-synthesized speech utterances 304 include words, phrases, and / or other terminology related to the particular domain. Optionally, in addition to or instead of the transcriptions 302 paired with the transcribed non-synthesized speech utterances 304, a different set of transcribed utterances related to the particular domain can be used to select the non-spoken text utterances 320. This provides the advantage of not requiring all transcribed non-synthesized speech utterances 304 to belong to a particular domain.

[0064] During the first stage (Stage A), the non-spoken text selection process 400 assembles two language models 404, 406 to enable contrastive selection of non-spoken text utterances 320. Here, a domain-specific language LM 410 is trained on each transcription 302 in the set of transcribed non-synthesized speech utterances 304. The set of transcribed non-synthesized speech utterances 304 is assumed to belong to the specific domain that the audio encoder 210 is trained to learn. Meanwhile, a background LM 406 is trained on each non-spoken text utterance 320 in the entire non-spoken text corpus 402. As mentioned above, the non-spoken text corpus 402 spans many different domains. In some examples, the first stage uses n-gram language model training to build the two language models 404, 406. In other examples, the first stage uses neural network language model training to build the two language models 404, 406.

[0065] During a second state (Phase B), the unspoken text selection process 400 calculates a first probability associated with each word in the unspoken text utterance 320 that appears in the domain-specific LM 404, i.e.,

[0066]

number

[0067] and determining a second probability associated with each word in the non-spoken text utterance 320 that appears in the background LM 406, i.e.

[0068]

number

[0069] The two contrasting LMs 404, 406 are used to evaluate each non-spoken text utterance 320 in the non-spoken text corpus 402 by determining a probability, S, of each non-spoken text utterance 320. Then, for each non-spoken text utterance 320 in the non-spoken text corpus 402, the text selection process 400 determines a score, S, in a scorer 408, based on the first probability, the second probability, and the number of words, #(w), that appear in the corresponding non-spoken text utterance 320. For example, the score S for each non-spoken text utterance 320 may be calculated as follows:

[0070]

number

[0071] After determining the scores, the non-spoken text selection process 400 selects the non-spoken text utterances 320 with the N best scores S as these non-spoken text utterances 320 best match the particular domain. The non-spoken text corpus 402 may contain billions of non-spoken text utterances 320. The non-spoken text utterances 320 selected by the text selection process 400 can include millions of utterances, far exceeding the number of untranscribed, non-synthesized speech utterances 306 spoken by human speakers. As described above, the content of the non-spoken text utterances 320 increases linguistic diversity for the particular domain that the audio encoder 210 is trained to learn, and the corresponding synthetic speech representations 332 generated from the non-spoken text utterances 320 increase acoustic / lexical diversity for the speech that the audio encoder 210 is encoding as part of the speech recognition process when the audio encoder 210 is integrated into the ASR model 200.

[0072] 5 is a flowchart of an exemplary arrangement of operations for method 500 of pre-training audio encoder 210 to jointly learn a shared representation of speech and text. Method 500 may be executed on data processing hardware 610 (FIG. 6) using instructions stored in memory hardware 620 (FIG. 6). Data processing hardware 610 and memory hardware 620 may reside on remote computer / server 201 of FIG. 1, which corresponds to computing device 600 (FIG. 6).

[0073] At operation 502, the method 500 may include receiving training data including unspoken text utterances 320 and untranscribed, unsynthesized voice utterances 306. Each unspoken text utterance 320 is not paired with any corresponding spoken utterance of unsynthesized voice. Each untranscribed, unsynthesized voice utterance 306 is not paired with a corresponding transcription.

[0074] At operation 504, the method 500 also includes generating a corresponding synthetic speech representation 332 for each non-spoken text utterance 320 in the received training data using a text-to-speech (TTS) system 330. At operation 506, the method also includes pre-training the audio encoder 210 based on the synthetic speech representation 332 generated for the non-spoken text utterance 320 and the untranscribed non-synthetic speech utterance 306 to teach the audio encoder 210 to jointly learn the shared speech and text representations. The pre-training may include pre-training the audio encoder 210 based on a contrastive loss 315 derived from each of the synthetic speech representation 332 and the untranscribed non-synthetic speech utterance 306. The pre-training may also include pre-training the audio encoder 210 based on a supervised loss 342 (e.g., an auxiliary decoder loss) derived from the synthetic speech representation 332.

[0075] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0076] Non-transitory memory may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transitory memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0077] 6 is a schematic diagram of an exemplary computing device 600 that may be used to implement the systems and methods described herein. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality are intended to be exemplary only and are not intended to limit the implementation of the invention described and / or claimed herein.

[0078] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be mounted on a common motherboard or in other manners as appropriate. Processor 610 can process instructions for execution within computing device 600, including instructions stored in memory 620 or on storage device 630, for displaying graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 680 coupled to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memory, as needed. Also, multiple computing devices 600 may be connected, each providing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0079] The memory 620 stores information non-transiently within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0080] The storage device 630 is capable of providing mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 can be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or devices in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 620, the storage device 630, or memory on the processor 610.

[0081] The high-speed controller 640 manages bandwidth-intensive operations for the computing device 600, while the low-speed controller 660 manages less bandwidth-intensive operations. Such allocation of duties is merely exemplary. In some implementations, the high-speed controller 640 is coupled to the memory 620, the display 680 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 650 that may receive various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), but may also be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, etc., or to a network device, such as a switch or router, for example, through a network adapter.

[0082] Computing device 600, as shown, may be implemented in several different forms. For example, computing device 600 may be implemented as a standard server 600a, or multiple times in a group of such servers 600a, or as a laptop computer 600b, or as part of a rack server system 600c.

[0083] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0084] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0085] The processes and logic flows described herein may be performed by one or more programmable processors, also referred to as data processing hardware, that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special-purpose logic circuitry, for example, an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operably coupled to a mass storage device to receive data from, transfer data to, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0086] To enable interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Furthermore, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0087] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0088] 100 Automated Speech Recognition (ASR) Systems 102 User Devices 104 users 106 Vocalization 107 User Interface Generator 108 Audio Subsystem 110 Acoustic Frame 111 Data Processing Hardware 113 Memory Hardware 120 Transcription 200 ASR model 200 RNN-T model 200a transducer model 201 Remote Computing Devices 210 Encoder Network 210 Audio Encoder 211 Coding features, coding representations 211m Masked coding features 212 Convolution Subsampling Block 214 Linear Layer 215 contrastive context vectors 215 encoded representation 216 Conformer Block 217 Quantizer 218 Masking Module 219 Target Context Vector 220 Prediction Network 230 Collaborative Network 240 softmax layers 300 training processes 300a Symmetric self-supervised loss part 300b Supervised loss part 300c Consistency Regularizer 302 Transcription 304 Transcribed Non-Synthesized Speech Utterances 306 Untranscribed non-synthesized speech utterances 312 First Encoded Representation 315 Contrastive Loss Module 316 Contrasting Losses 320 Non-speech Text Utterances 330 Text-to-Speech (TTS) Systems, Text-to-Speech Models 332 Synthetic speech expression 340 Supervised Loss Module 342 Supervised loss term, synthetic speech loss term 390 Auxiliary Decoder 392 First Probability Distribution 393 Second Probability Distribution 400 Contrastive Non-Speech Text Selection Process 401 Datastore 402 Non-speech text corpus 404 Domain Language Model 404 Domain-specific LM 406 Background Language Model 408 Scorer 410 Domain-Specific Language LM 500 ways 600 computing devices 600a Server 600b laptop computer 600c Rack Server System 610 Data Processing Hardware 610 processor 620 Memory Hardware 630 Storage Devices 640 High-Speed ​​Interface / Controller 650 High-Speed ​​Expansion Port 660 Low-Speed ​​Interface / Controller 670 Slow Bus 680 display 690 Low-Speed ​​Expansion Port

Claims

1. A computer-implemented method (500) that, when executed on data processing hardware (610), causes the data processing hardware (610) to perform operations, the operations comprising: non-spoken text utterances (320), each of which is not paired with any corresponding spoken utterance of the non-synthesized voice; untranscribed non-synthesized speech utterances (306), each untranscribed non-synthesized speech utterance (306) not paired with a corresponding transcription; receiving training data (306, 320) including: generating a corresponding synthetic speech representation (332) for each non-spoken text utterance (320) of the received training data (306, 320) using a text-to-speech model (330); pre-training an audio encoder (210) based on the synthesized speech representation (332) generated for the non-speech text utterance (320) and the untranscribed non-synthesized speech utterance (306) to teach the audio encoder (210) to jointly learn a shared representation between speech and text; Pre-training the audio encoder (210) For each synthesized speech representation (332), generating a corresponding coded representation (211) of said synthetic speech representation (332); A computer-implemented method (500) comprising pre-training the audio encoder (210) based on a contrastive loss (316) applied to the corresponding coded representation (211) of the synthetic speech representation (332), the contrastive loss being a loss between a contrastive context vector and a target context vector at a mask position for the corresponding coded representation (211) of the synthetic speech representation (332).

2. 10. The computer-implemented method of claim 1, wherein the audio encoder includes a stack of self-attention layers, each of which includes a multi-head self-attention mechanism.

3. Pre-training the audio encoder (210) For each untranscribed non-synthesized speech utterance (306), generating a corresponding coded representation (211) of said untranscribed, non-synthesized speech utterance (306); pre-training the audio encoder (210) based on a contrastive loss (316) applied to the corresponding coded representation (211) of the untranscribed, non-synthetic speech utterance (306), the contrastive loss being a loss between a contrastive context vector and a target context vector of a mask position for the corresponding coded representation (211) of the untranscribed, non-synthetic speech utterance (306); 3. A computer-implemented method (500) according to claim 1 or 2.

4. Pre-training the audio encoder (210) includes, at each of a plurality of time steps of each synthetic speech representation (332): generating, using an auxiliary decoder (390), a first probability distribution (392) over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation (332); determining a synthetic speech loss term (342) based on the first probability distribution (392) over possible synthetic speech recognition hypotheses and the unspoken text utterance (320) corresponding to the corresponding synthetic speech representation (332); pre-training the audio encoder (210) based on the synthetic speech loss term (342); 2. The computer-implemented method (500) of claim 1, comprising:

5. 5. The computer-implemented method of claim 4, wherein the first probability distribution over possible synthetic speech recognition hypotheses comprises one of possible phoneme labels or possible wordpiece labels.

6. Pre-training the audio encoder (210) includes, at each of the plurality of time steps of each synthetic speech representation (332), generating, using another auxiliary decoder (390), a second probability distribution (393) over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation (332), the second probability distribution (393) over possible synthetic speech recognition hypotheses including the other of the possible phoneme labels or the possible wordpiece labels; determining another synthetic speech loss term (342) based on the second probability distribution (393) over possible synthetic speech recognition hypotheses and the unspoken text utterance (320) corresponding to the corresponding synthetic speech representation (332); and pre-training the audio encoder (210) based on the other synthetic speech loss term (342); 6. The computer-implemented method (500) of claim 5, further comprising:

7. 5. The computer-implemented method of claim 4, wherein the auxiliary decoder comprises one of a Connection Temporal Classification (CTC) decoder, a Listen Attend Spell (LAS) decoder, or a Recurrent Neural Network-Transducer (RNN-T) decoder.

8. 2. The computer-implemented method of claim 1, wherein the operations further comprise augmenting one or more of the synthetic speech representations before pre-training the audio encoder based on the synthetic speech representations.

9. 9. The computer-implemented method of claim 8, wherein pre-training the audio encoder based on the synthetic speech representation comprises pre-training the audio encoder based on the one or more augmented synthetic speech representations.

10. 10. The computer-implemented method of claim 1, wherein the non-spoken text utterances are generated and / or selected using one or more language models.

11. 10. The computer-implemented method of claim 1, wherein the non-spoken text utterances are generated using a background language model and a domain language model trained on transcribed speech utterances associated with the target domain.

12. 10. The computer-implemented method of claim 1, wherein the operations further comprise pre-training the audio encoder and then fine-tuning the pre-trained audio encoder to a transcribed speech utterance.

13. A system (100), data processing hardware (610); and memory hardware (620) in communication with the data processing hardware (610), the memory hardware (620) storing instructions that, when executed on the data processing hardware (610), cause the data processing hardware (610) to perform operations, the operations including: non-spoken text utterances (320), each of which is not paired with any corresponding spoken utterance of the non-synthesized voice; untranscribed non-synthesized speech utterances (306), each untranscribed non-synthesized speech utterance (306) not paired with a corresponding transcription; receiving training data (306, 320) including: generating a corresponding synthetic speech representation (332) for each non-spoken text utterance (320) of the received training data (306, 320) using a text-to-speech model (330); pre-training an audio encoder (210) based on the synthesized speech representation (332) generated for the non-speech text utterance (320) and the untranscribed non-synthesized speech utterance (306) to teach the audio encoder (210) to jointly learn a shared representation between speech and text; Pre-training the audio encoder (210) For each synthesized speech representation (332), generating a corresponding coded representation (211) of said synthetic speech representation (332); A system (100) comprising: pre-training the audio encoder (210) based on a contrastive loss (316) applied to the corresponding coded representation (211) of the synthetic speech representation (332), the contrastive loss being a loss between a contrastive context vector and a target context vector of a mask position for the corresponding coded representation (211) of the synthetic speech representation (332).

14. 14. The system of claim 13, wherein the audio encoder includes a stack of self-attention layers, each of which includes a multi-head self-attention mechanism.

15. Pre-training the audio encoder (210) For each untranscribed non-synthesized speech utterance (306), generating a corresponding coded representation (211) of said untranscribed, non-synthesized speech utterance (306); pre-training the audio encoder (210) based on a contrastive loss (316) applied to the corresponding coded representation (211) of the untranscribed, non-synthetic speech utterance, the contrastive loss being a loss between a contrastive context vector and a target context vector of a mask position for the corresponding coded representation (211) of the untranscribed, non-synthetic speech utterance (306); 15. The system (100) of claim 13 or 14, comprising:

16. Pre-training the audio encoder (210) includes, at each of a plurality of time steps of each synthetic speech representation (332): generating, using an auxiliary decoder (390), a first probability distribution (392) over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation (332); determining a synthetic speech loss term (342) based on the first probability distribution (392) over possible synthetic speech recognition hypotheses and the unspoken text utterance (320) corresponding to the corresponding synthetic speech representation (332); pre-training the audio encoder (210) based on the synthetic speech loss term (342); 14. The system (100) of claim 13, comprising:

17. 17. The system of claim 16, wherein the first probability distribution over possible synthetic speech recognition hypotheses includes one of possible phoneme labels or possible wordpiece labels.

18. Pre-training the audio encoder (210) includes, at each of the plurality of time steps of each synthetic speech representation (332), generating, using another auxiliary decoder (390), a second probability distribution (393) over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation (332), the second probability distribution (393) over possible synthetic speech recognition hypotheses including the other of the possible phoneme labels or the possible wordpiece labels; determining another synthetic speech loss term (342) based on the second probability distribution (393) over possible synthetic speech recognition hypotheses and the unspoken text utterance (320) corresponding to the corresponding synthetic speech representation (332); and pre-training the audio encoder (210) based on the other synthetic speech loss term (342); 20. The system (100) of claim 17, further comprising:

19. 17. The system of claim 16, wherein the auxiliary decoder comprises one of a Connection Temporal Classification (CTC) decoder, a Listen Attend Spell (LAS) decoder, or a Recurrent Neural Network-Transducer (RNN-T) decoder.

20. 14. The system (100) of claim 13, wherein the operations further include augmenting one or more of the synthetic speech representations (332) before pre-training the audio encoder (210) based on the synthetic speech representations (332).

21. 21. The system (100) of claim 20, wherein pre-training the audio encoder (210) based on the synthetic speech representation (332) comprises pre-training the audio encoder (210) based on the one or more augmented synthetic speech representations (332).

22. 14. The system (100) of claim 13, wherein the non-spoken text utterances (320) are generated and / or selected using one or more language models (404, 406).

23. 14. The system of claim 13, wherein the non-spoken text utterances are generated using a background language model and a domain language model trained on transcribed speech utterances associated with the target domain.

24. 14. The system (100) of claim 13, wherein the operations further include, after pre-training the audio encoder (210), fine-tuning the pre-trained audio encoder (210) on a transcribed speech utterance.

Images