Alignment prediction for feeding text into automatic speech recognition training

Abstract

The method (900) includes receiving training data including an unspoken text utterance (320), an untranscribed non-synthesized speech utterance (306), and a transcribed non-synthesized speech utterance (304). Each unspoken text utterance is not paired with any corresponding speech utterance of the non-synthesized speech. Each untranscribed non-synthesized speech utterance is not paired with a corresponding transcription. Each transcribed non-synthesized speech utterance is paired with a corresponding transcription (302). The method also includes generating a corresponding alignment output (602) for each unspoken text utterance of the received training data using an alignment model (600). The method also includes pre-training an audio encoder (210) with the generated alignment outputs corresponding to the unspoken text utterance, the untranscribed non-synthesized speech utterance, and the transcribed non-synthesized speech utterance to teach the audio encoder to jointly learn a shared representation of speech and text.

Description

[Technical field]

[0001] The present disclosure relates to alignment prediction for inputting text into automatic speech recognition training. [Background technology]

[0002] Automatic speech recognition (ASR), the process of taking an audio input and transcribing it into text, is a very important technology used in mobile and other devices. In general, automatic speech recognition attempts to provide an accurate transcription of what a person has said by taking an audio input (e.g., a spoken utterance) and transcribing the audio input into text. Modern ASR models continue to improve both accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay time from a user's utterance to transcription) based on ongoing developments in deep neural networks. However, one challenge in developing an ASR model based on deep learning is that the parameters of the ASR model tend to overfit to the training data, which makes it difficult for the ASR model to generalize to unseen data if the training data is not extensive enough. As a result, training the ASR model with a larger training dataset improves the accuracy of the ASR model. The amount of training data used to train the ASR model can be increased by incorporating synthetic speech and / or data-augmented speech. Summary of the Invention

[0003] One aspect of the disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for pre-training an audio encoder to jointly learn a shared representation of speech and text. The operations include receiving training data including an unspoken text utterance, an untranscribed non-synthesized speech utterance, and a transcribed non-synthesized speech utterance. Each unspoken text utterance is not paired with any corresponding speech utterance of the non-synthesized speech. Each untranscribed non-synthesized speech utterance is not paired with a corresponding transcription. Each transcribed non-synthesized speech utterance is paired with a corresponding transcription. In operations, the method includes using the alignment model to generate a corresponding alignment output for each unspoken text utterance of the received training data. In operation, the method includes pre-training an audio encoder with alignment outputs generated corresponding to unspoken text utterances, untranscribed non-synthesized speech utterances, and transcribed non-synthesized speech utterances to teach the audio encoder to jointly learn a shared representation of speech and text.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the audio encoder includes a stack of self-attention layers, each including a multi-head self-attention mechanism. In some examples, pre-training the audio encoder includes: for each untranscribed non-synthetic speech utterance, generating a corresponding encoded representation of the untranscribed non-synthetic speech utterance, and pre-training the audio encoder with a contrastive loss applied to the corresponding encoded representation of the untranscribed non-synthetic speech utterance; for each alignment output, generating a corresponding encoded representation of the alignment output, and pre-training the audio encoder with a contrastive loss applied to the corresponding encoded representation of the alignment output; and for each transcribed non-synthetic speech utterance, generating a corresponding encoded representation of the transcribed non-synthetic speech utterance, and pre-training the audio encoder with a contrastive loss applied to the corresponding encoded representation of the transcribed non-synthetic speech utterance.

[0005] In some implementations, pre-training the audio encoder includes, at each of a plurality of time steps for each alignment output, generating, using an auxiliary decoder, a first probability distribution for possible synthetic speech recognition hypotheses for the corresponding alignment output, determining an alignment output loss based on the first probability distribution for the possible synthetic speech recognition hypotheses and an unspoken text utterance corresponding to the alignment output, pre-training the audio encoder based on the alignment output loss term, at each of a plurality of time steps for each transcribed non-synthetic speech utterance, generating, using the auxiliary decoder, a second probability distribution for possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance, determining a non-synthetic speech loss term based on the second probability distribution for the possible non-synthetic speech recognition hypotheses and a corresponding transcription paired with the transcribed non-synthetic speech utterance, and pre-training the audio encoder based on the non-synthetic speech loss term. In these implementations, the auxiliary decoder may include one of a connection time classification (CTC) decoder, a listen-attend-spell (LAS) decoder, or a recurrent neural network transducer (RNN-T) decoder, where the first probability distribution over the possible synthetic speech recognition hypotheses may include one of the possible phoneme labels or the possible wordpiece labels, and the second probability distribution over the possible non-synthetic speech recognition hypotheses includes one of the possible phoneme labels or the possible wordpiece labels.

[0006] In some examples, the audio encoder includes a text encoder, a speech encoder, and a shared encoder. In these examples, the operations further include, for each alignment output, determining an encoded text representation of the alignment output using the text encoder and generating a first encoded shared representation of the alignment output in a shared latent representation space using the shared encoder, and, for each transcribed non-synthetic speech utterance, determining an encoded audio representation of the transcribed non-synthetic speech utterance using the speech encoder and generating a second encoded shared representation of the transcribed non-synthetic speech utterance in the shared latent representation space using the shared encoder.

[0007] Generating a corresponding alignment output for each unspoken text utterance of the received training data may include extracting an initial text representation from the unspoken text utterance, predicting a text chunk duration for each text chunk in the unspoken text utterance, and upsampling the initial text representation using the predicted text chunk duration for each text chunk in the unspoken text utterance. In some implementations, the operations further include training an alignment model by generating an encoded audio representation for the transcribed non-synthesized speech utterance using a speech encoder, determining an alignment output for the transcription corresponding to the transcribed non-synthesized speech utterance using the alignment model, generating an encoded text representation for the alignment output, and updating parameters of the alignment model based on a comparison of the encoded audio representation for the transcribed non-synthesized speech utterance and the encoded text representation for the alignment output.

[0008] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving training data including an unspoken text utterance, an untranscribed non-synthesized speech utterance, and a transcribed non-synthesized speech utterance. Each unspoken text utterance is not paired with any corresponding speech utterance of the non-synthesized speech. Each untranscribed non-synthesized speech utterance is not paired with a corresponding transcription. Each transcribed non-synthesized speech utterance is paired with a corresponding transcription. In operations, the method includes using the alignment model to generate a corresponding alignment output for each unspoken text utterance of the received training data. In operation, the method includes pre-training an audio encoder with alignment outputs generated corresponding to unspoken text utterances, untranscribed non-synthesized speech utterances, and transcribed non-synthesized speech utterances to teach the audio encoder to jointly learn a shared representation of speech and text.

[0009] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the audio encoder includes a stack of self-attention layers, each including a multi-head self-attention mechanism. In some examples, pre-training the audio encoder includes: for each untranscribed non-synthetic speech utterance, generating a corresponding encoded representation of the untranscribed non-synthetic speech utterance, and pre-training the audio encoder with a contrastive loss applied to the corresponding encoded representation of the untranscribed non-synthetic speech utterance; for each alignment output, generating a corresponding encoded representation of the alignment output, and pre-training the audio encoder with a contrastive loss applied to the corresponding encoded representation of the alignment output; and for each transcribed non-synthetic speech utterance, generating a corresponding encoded representation of the transcribed non-synthetic speech utterance, and pre-training the audio encoder with a contrastive loss applied to the corresponding encoded representation of the transcribed non-synthetic speech utterance.

[0010] In some implementations, pre-training the audio encoder includes, at each of a plurality of time steps for each alignment output, generating, using an auxiliary decoder, a first probability distribution for possible synthetic speech recognition hypotheses for the corresponding alignment output, determining an alignment output loss based on the first probability distribution for the possible synthetic speech recognition hypotheses and an unspoken text utterance corresponding to the alignment output, pre-training the audio encoder based on the alignment output loss term, at each of a plurality of time steps for each transcribed non-synthetic speech utterance, generating, using the auxiliary decoder, a second probability distribution for possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance, determining a non-synthetic speech loss term based on the second probability distribution for the possible non-synthetic speech recognition hypotheses and a corresponding transcription paired with the transcribed non-synthetic speech utterance, and pre-training the audio encoder based on the non-synthetic speech loss term. In these implementations, the auxiliary decoder may include one of a connection time classification (CTC) decoder, a listen-attend-spell (LAS) decoder, or a recurrent neural network transducer (RNN-T) decoder, where the first probability distribution over the possible synthetic speech recognition hypotheses may include one of the possible phoneme labels or the possible wordpiece labels, and the second probability distribution over the possible non-synthetic speech recognition hypotheses includes one of the possible phoneme labels or the possible wordpiece labels.

[0011] In some examples, the audio encoder includes a text encoder, a speech encoder, and a shared encoder. In these examples, the operations further include, for each alignment output, determining an encoded text representation of the alignment output using the text encoder and generating a first encoded shared representation of the alignment output in a shared latent representation space using the shared encoder, and, for each transcribed non-synthetic speech utterance, determining an encoded audio representation of the transcribed non-synthetic speech utterance using the speech encoder and generating a second encoded shared representation of the transcribed non-synthetic speech utterance in the shared latent representation space using the shared encoder.

[0012] Generating a corresponding alignment output for each unspoken text utterance of the received training data may include extracting an initial text representation from the unspoken text utterance, predicting a text chunk duration for each text chunk in the unspoken text utterance, and upsampling the initial text representation using the predicted text chunk duration for each text chunk in the unspoken text utterance. In some implementations, the operations further include training an alignment model by generating an encoded audio representation for the transcribed non-synthesized speech utterance using a speech encoder, determining an alignment output for the transcription corresponding to the transcribed non-synthesized speech utterance using the alignment model, generating an encoded text representation for the alignment output, and updating parameters of the alignment model based on a comparison of the encoded audio representation for the transcribed non-synthesized speech utterance and the encoded text representation for the alignment output.

[0013] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the following description and drawings, and from the claims. [Brief description of the drawings]

[0014] [Figure 1] 1 is a schematic diagram of an exemplary speech recognition system. [Diagram 2] FIG. 1 is a schematic diagram of a recurrent neural network transducer (RNN-T) model architecture. [Figure 3A] FIG. 1 is a schematic diagram of an example training process for pre-training an audio encoder for a speech recognition model. [Figure 3B] FIG. 1 is a schematic diagram of an example training process for pre-training an audio encoder for a speech recognition model. [Figure 3C] FIG. 1 is a schematic diagram of an example training process for pre-training an audio encoder for a speech recognition model. [Figure 4] FIG. 2 is a schematic diagram illustrating an example of an unspoken text selection process for selecting unspoken text utterances related to a particular domain. [Diagram 5] FIG. 2 illustrates a representation of an exemplary projection space encoder for non-synthetic and synthetic speech. [Figure 6] FIG. 4 is a schematic diagram of an alignment model used during an exemplary training process for pre-training the audio encoder of the speech recognition model of FIGS. 3A-3C. [Figure 7] FIG. 1 is a schematic diagram of an exemplary training process for training a duration predictor of an alignment model. [Figure 8] FIG. 1 is a schematic diagram of an exemplary training process for training an upsampler of an alignment model. [Figure 9] 1 is a flowchart illustrating an example flow of operations for a method for pre-training an audio encoder to jointly learn a shared representation of speech and text. [Figure 10] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Like reference symbols in the various drawings indicate like elements. A major advance in automatic speech recognition has been the introduction of sequence-to-sequence (Seq2Seq) models, which map speech signals to strings of characters. At the same time, text-to-speech (TTS) or speech synthesis systems have successfully applied Seq2Seq models to deliver state-of-the-art natural and realistic sounding synthetic speech that is indistinguishable from human speech to the human ear.

[0016] One challenge in developing an ASR model based on deep learning is that the parameters of the ASR model tend to overfit to the training data, which makes it difficult for the ASR model to generalize to unseen data if the training data is not extensive enough. As a result, training an ASR model on a larger training dataset improves the accuracy of the ASR model. For example, an ASR model can be trained on a training dataset that contains more than 10,000 hours of transcribed speech using machine learning or other statistical methods. However, the performance of the ASR model degrades when the domain associated with the training data is different from the domain in which the ASR model is deployed during inference. For example, training an ASR model on transcribed speech in a domain associated with videoconferencing is less effective at recognizing speech associated with voice search queries, and vice versa.

[0017] Unpaired text data can significantly limit the amount of labeled human speech required to train an ASR model, while also providing flexibility in making the ASR model work across different domains. However, using text data (i.e., unpaired text data) in addition to speech data to train an ASR model creates problems in matching speech and text modalities of the training data. One current approach uses multi-task training to train a single model with different objectives for each modality. This approach suffers from interference and limited capacity when each modality of the training data has different properties and objectives. Other current approaches include TTS systems that synthesize unpaired text data to generate synthetic speech (i.e., modality conversion). Furthermore, it has been found that training an ASR model using synthetic speech based on text data has a different impact on ASR training than human speech, even though state-of-the-art synthetic speech examples are indistinguishable from human speech. This gap between synthetic speech and human speech is due to the mismatch of human speech data in synthetic speech data, which arises from the challenging one-to-many mapping problem that TTS systems are trying to solve. That is, while the total quality of the available synthetic speech is very high, it has much less variability and minimal choppiness than human speech. As a result, training ASR models using synthetic speech based on unpaired text data makes it difficult to generalize to real speech utterances during inference.

[0018] The embodiments described herein relate to aligning a text representation used to generate synthetic speech with a corresponding non-synthetic speech representation in a latent representation space for training an ASR model. That is, the alignment model can generate an alignment output for an unspoken text utterance when a large amount of transcribed speech (e.g., non-synthetic speech) is not available or is not very common in a target domain and / or target language for training an ASR model. More specifically, the embodiments are directed to pre-training an audio encoder of an ASR model with training data including untranscribed non-synthetic speech utterances, unspoken text utterances for generating corresponding aligned outputs, and transcribed non-synthetic speech utterances to jointly learn speech and text representations, and then fine-tuning (e.g., warm-start training) the pre-trained ASR model using the available transcribed non-synthetic speech utterances. In particular, generating an alignment output for unspoken text utterances (e.g., without converting the text utterance to speech) facilitates computationally efficient alignment of representations between the text and speech modalities of the training data. As will become apparent below, pre-training the audio encoder includes updating parameters of the audio encoder based on a combination of self-supervised control losses, supervised losses, and consistency losses derived from the training data. The ASR model may include a monolingual ASR model or a multilingual ASR model. Additionally, the learned representations between the text and speech modalities may be used for a speech translation model.

[0019] 1 illustrates an automatic speech recognition (ASR) system 100 that implements an ASR model 200 residing on a user device 102 of a user 104 and / or on a remote computing device 201 (e.g., one or more servers of a distributed system running in a cloud computing environment) that communicates with the user device 102. Although the user device 102 is illustrated as a mobile computing device (e.g., a smartphone), the user device 102 can represent any type of computing device, such as, without limitation, a tablet device, a laptop / desktop computer, a wearable device, a digital assistant device, a smart speaker / display, a smart appliance, an in-vehicle infotainment system, or an Internet of Things (IoT) device, and includes data processing hardware 111 and memory hardware 113.

[0020] The user device 102 includes an audio subsystem 108. The audio subsystem 108 is configured to receive utterances 106 spoken by the user 104 (e.g., the user device 102 may include one or more microphones that record the spoken utterances 106) and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 processable by the ASR system 100. In the illustrated example, the user makes a respective utterance 106 in the natural language of English for the phrase "What's the weather like in New York City?", and the audio subsystem 108 converts the utterance 106 into a corresponding acoustic frame 110 for input to the ASR system 100. The ASR model 200 then receives the acoustic frame 110 corresponding to the utterance 106 as input and generates / predicts a corresponding transcription 120 (e.g., a recognition result / hypothesis) of the utterance 106 as output. In the illustrated example, the user device 102 and / or the remote computing device 201 also execute a user interface generator 107 configured to present a representation of the transcription 120 of the utterance 106 to the user 104 of the user device 102. In some configurations, the transcription 120 output from the ASR system 100 is processed by a natural language understanding (NLU) module, e.g., executing on the user device 102 or the remote computing device 201, to execute user commands. Additionally or alternatively, a text-to-speech system (e.g., executing on any combination of the user device 102 or the remote computing device 201) may convert the transcription into synthetic speech for audible output by another device. For example, the original utterance 106 corresponds to a message the user 104 is sending to a friend, where the transcription 120 is converted into synthetic speech for audible output to the friend who hears the message conveyed in the original utterance 106.

[0021] Referring to FIG. 2, an exemplary frame alignment based transducer model 200 includes a recurrent neural network transducer (RNN-T) model architecture that is subject to latency constraints associated with interactive applications. The use of the RNN-T model architecture is exemplary, and the frame alignment based transducer model 200 may include other architectures, such as a transformer transducer model architecture and a conformer transducer model architecture, among others. Because the RNN-T model 200 has a small computational footprint and utilizes less memory requirements than traditional ASR architectures, the RNN-T model architecture is suitable for performing all speech recognition on the user device 102 (e.g., does not require communication with a remote server). The RNN-T model 200 includes an encoder network 210, a predictive network 220, and a joint network 230. The encoder network 210 is roughly similar to an acoustic model (AM) of a traditional ASR system and includes a stack of self-attention layers (e.g., conformer layers or transformer layers) or a recurrent network of stacked long short-term memory (LSTM) layers. For example, the encoder may take a sequence of d-dimensional feature vectors (e.g., the acoustic frame 110 (FIG. 1)) x=(x1, x2, ..., x T )(in the formula,

[0022]

number

[0023] ) and generate a high-level representation at each output step.

[0024]

number

[0025] It is expressed as: Similarly, the prediction network 220 is also an LSTM network, which, like the language model (LM), calculates the sequence of non-empty symbols, y0,...,y ui-1 , and dense representation

[0026]

number

[0027] Finally, with the RNN-T model architecture, the representations generated by the encoder network 210 and the prediction / decoder network 220 are combined by the joint network 230. The prediction network 220 can be replaced with an embedding lookup table to improve latency by outputting lookup sparse embedding values ​​instead of processing dense representations. The joint network then

[0028]

number

[0029] , which is a distribution for the next output symbol. In other words, the joint network 230 generates a probability distribution for possible speech recognition hypotheses at each output step (e.g., time step), where a "possible speech recognition hypothesis" corresponds to a set of output labels each representing a symbol / character of a particular natural language. For example, if the natural language is English, the set of output labels may include 27 symbols (e.g., one label for each of the 26 letters of the English alphabet and one label representing a space). Thus, the joint network 230 may output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels. This set of values ​​may be a vector and may indicate a probability distribution for the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation and other symbols), although the set of output labels is not limited thereto. For example, the set of output labels may include word pieces, phonemes, and / or whole words in addition to or instead of graphemes. The output distribution of the joint network 230 may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output y i can contain 100 different probability values, one for each output label. The probability distribution can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and / or words) in a beam search process (e.g., by softmax layer 240) to determine the transcription 120.

[0030] The softmax layer 240 can use any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200 at the corresponding output step. In this way, the RNN-T model 200 does not make a conditional independence assumption, but rather the prediction of each symbol is not only conditioned on the acoustics but also on the sequence of labels output so far. The RNN-T model 200 assumes that the output symbols are independent of future acoustic frames 110. This allows the RNN-T model to be used in a streaming manner.

[0031] In some examples, the encoder network (i.e., audio encoder) 210 of the RNN-T model 200 includes a stack of self-attention layers / blocks, such as conformer blocks, where each conformer block includes a series of multi-head self-attention layers, depthwise convolutional layers, and feedforward layers. The prediction network 220 can have two 2,048-dimensional LSTM layers, each of which is followed by a further 640-dimensional projection layer. Alternatively, the prediction network 220 can include a stack of transformer or conformer blocks, or an embedding lookup table, instead of an LSTM layer. Finally, the joint network 230 can also have 640 hidden units. The softmax layer 240 can be composed of a set of unified word pieces or graphemes that are generated using all unique word pieces or graphemes of multiple training datasets.

[0032] 3A-3C illustrate an example training process 300 for pre-training the audio encoder 210 of the ASR model 200 (FIG. 2). The training process 300 may use available training data to pre-train the audio encoder 210. The available training data may include unspoken text utterances (X text ) A set of 320 transcribed non-synthesized speech utterances (Xsup ) 304 set, and / or untranscribed non-synthesized speech utterances (X unsup ) 306. Each unspoken text utterance 320 includes text-only data (i.e., unpaired data), and thus, each unspoken text utterance 320 is not paired with any corresponding spoken audio representation (i.e., speech) of the utterance. An unspoken text utterance 320 may include any sequenced text chunks, including words, word pieces, phonemes, and / or graphemes. Each untranscribed non-synthesized speech utterance 306 (also referred to simply as an “untranscribed speech utterance 306”) includes audio-only data (i.e., unpaired data), and thus, each untranscribed speech utterance 306 is not paired with any corresponding transcription. On the other hand, each transcribed non-synthesized speech utterance 304 (also referred to simply as a “transcribed speech utterance 304”) includes a corresponding transcription 302 paired with a corresponding unsynthesized speech representation of the corresponding transcribed speech utterance 304.

[0033] For clarity, the training process 300 includes a self-supervised contrast loss portion 300a (FIG. 3A), a supervised loss portion 300b (FIG. 3B), and a consistency regularizer portion 300c (FIG. 3C). The training process 300 is performed by: text ) 320, transcribed non-synthesized speech (X sup ) 304 corpora, and untranscribed non-synthesized speech utterances (X unsup ) 306, the contrast loss (L w2v ) 316 and the unuttered training text utterances (X text )320 and the transcribed non-synthesized speech utterances (X sup ) 304 using the supervised loss unit 300b. aux ) 342, 344 and the consistency loss (J cons (θ)) 352 and the total loss based on (L tts4pretrain2 ) to pre-train the audio encoder 210.

[0034] 3A, the self-supervised control loss portion 300a of the training process 300 may use an alignment model 600. The alignment model 600 is configured to generate an alignment output (i.e., a text representation) 602 for each of a plurality of unspoken training text utterances 320 at each of a plurality of output steps. The unspoken text utterances 320 (also referred to simply as “unspoken text utterances 320 (singular)”) include unspoken text that is text-only data (i.e., unpaired data), and thus each unspoken text utterance (X text ) 320 is not paired with any synthetic or non-synthetic speech. Thus, for each unspoken text utterance 320, the alignment model 600 generates a corresponding alignment output 602.

[0035] 6, in some examples, the alignment model 600 includes an embedding extractor 610, a duration predictor 620, and an upsampler 630. The embedding extractor 610 receives an unspoken text utterance 320 that includes a sequence of text chunks that include words, word pieces, phonemes, and / or graphemes, and generates a corresponding initial text representation (e t) 612, which includes embedded lexical information from the unspoken text utterance 320. Additionally or alternatively, the embeddings extractor 610 may receive a transcription 302 corresponding to the transcribed non-synthesized speech utterance 304 (FIG. 3C). The duration predictor 620 receives the initial text representation 612 from the embeddings extractor 610 and predicts corresponding text chunk durations (i.e., word, wordpiece, phoneme, and / or grapheme durations) 622. The text chunk durations 622 indicate the duration for which the corresponding text chunk would be spoken if a person (or a text-to-speech system) spoke the unspoken text utterance 320. For example, the unspoken text utterance 320 may include a sequence of phonemes, and the duration predictor 620 predicts a phoneme duration 622 for each phoneme in the sequence of phonemes. In this example, the duration predictor 620 predicts the phoneme duration 622 by predicting the probability of a non-zero duration for each phoneme and predicting the probability of a continuous phoneme duration for each phoneme. Because a sequence of phonemes includes regular phonemes, silences between word boundaries, and punctuation marks, only regular phonemes are associated with non-zero durations, while silences and punctuation marks are usually associated with continuous phoneme durations. Thus, the duration predictor 620 may predict the probability of a non-zero duration using a sigmoid activation value following a first of two independent activation values, and predict the continuous text chunk duration 622 for each text chunk using a soft-plus activation value following a second of two independent projection values. The duration predictor 620 may determine whether the probability of a non-zero duration is less than a threshold for each text chunk, and if the probability of a non-zero duration is less than the threshold, a multiplier may zero out the continuous text chunk duration 622 predicted by the soft-plus activation for the corresponding text chunk. On the other hand, if the non-zero duration probability is greater than or equal to the threshold, the predicted text chunk duration 622 may be set equal to the consecutive phoneme duration predicted by the soft-plus activation.

[0036] The upsampler 630 receives, for each unspoken text utterance 320, a corresponding initial text representation 612 and a predicted text chunk duration 622, and upsamples the initial text representation 612 using the corresponding predicted text chunk duration 622 to generate an alignment output having a substantial number of frames.

[0037]

number

[0038] 602. In some examples, the alignment model 600 sends the alignment output 602 to the text encoder 202 of the audio encoder 210 (FIGS. 3B and 3C). In other examples (not shown), the alignment model 600 sends the alignment output 602 to the shared encoder 250 of the audio encoder 210 (FIGS. 3B and 3C) (e.g., bypassing the text encoder 202). In these other examples, the alignment output 602 serves as the encoded text representation 312, such that the shared encoder 250 may receive the alignment output 602 directly from the alignment model 600 (FIGS. 3B and 3C). In some additional examples, paired training data is available and the upsampler 630 generates the alignment output 602 as follows:

[0039]

number

[0040] Here, the upsampler includes a resampler layer and a refiner layer that directly aligns the initial text embedding values ​​612 with the corresponding encoded audio representations 314 (FIGS. 3B and 3C). In other examples, no paired training data is available and the upsampler 630 generates the alignment output 602 as follows:

[0041]

number

[0042] In particular, the frame numbers of the alignment output 602 indicate the predicted speech duration of the unspoken text utterance 320. In other words, the frame numbers of the alignment output 602 map (i.e., align) the sequence of text chunks of the unspoken text utterance 320 to speech frames. Here, the upsampler 630 includes a resampler layer and a refiner layer that replicate the initial text embedding values ​​612 to match the predicted text chunk durations 622 (i.e., speech durations). Thus, the alignment output 602 includes a textual representation of the unspoken text utterance 320 with timing components that are consistent with how a person would speak the unspoken text utterance 320.

[0043] In particular, in most cases, a text-to-speech (TTS) system generates an audible output to impart the timing components of human speech to the unspoken text utterance 320, whereby the training process may use the audible output (i.e., synthetic speech) to train the audio encoder 210. Thus, because the alignment model 600 generates an alignment output 602 that directly maps sequences of text chunks to audio frames, the training process 300 does not require a TTS system to train the audio encoder 210 with the unspoken text utterance 320. That is, the alignment model 600 does not transform the unspoken text utterance 320 to generate synthetic speech.

[0044] 7 illustrates an exemplary training process 700 for training an alignment model 600 using transcribed non-synthetic speech utterances 304 with corresponding transcriptions 302 (i.e., paired training data). In the illustrated example, the speech encoder 204 receives as input each transcribed non-synthetic speech utterance 304 as a sequence of features / vectors (e.g., Mel-frequency spectrograms, such as acoustic frames 110 of FIG. 1 ) and generates as output, for each of a number of time steps, an encoded audio representation (e.g., s ) 314. In parallel, the alignment model 600 receives the transcription 302 corresponding to the same non-synthesized speech utterance 304 and generates an alignment output 602 according to Equation 1. The text encoder 202 receives the alignment output 602 as input and, for each of a number of time steps, generates an encoded text representation corresponding to the transcription 302 at the corresponding time step.

[0045]

number

[0046] 312 is produced as output. The modality loss module 750 receives the encoded text representation 312 and the encoded audio representation 314 and generates a modality loss 752 based on comparing the encoded text representation 312 and the encoded audio representation as follows:

[0047]

number

[0048] Equation 3 is the encoded text representation

[0049]

number

[0050] 312 and the encoded audio representation (e s ) 314 is compared to the mean squared error (MSE) between the predicted text target and the encoded audio representation (e s )314, the modality loss (L MM ) 752, where the encoded audio representations 314 serve as ground truth labels to train the alignment model 600 to generate alignment outputs 602 that match the corresponding non-synthesized speech utterances 304. The training process 700 may use the modality loss 752 to update parameters of the alignment model 600. For example, the training process 700 may update parameters of the duration predictor 620 and / or the upsampler 630 (FIG. 6).

[0051] 8 illustrates an exemplary training process 800 for training an alignment model 600 using paired and unpaired training data. That is, the training process 800 uses transcribed non-synthesized speech utterances 304 with corresponding transcriptions 302 (i.e., paired training data) and unspoken text utterances 320 (i.e., unpaired training data) to learn a speech alignment output 602. In the illustrated example, the speech encoder 204 receives as input each transcribed non-synthesized speech utterance 304 as a sequence of features / vectors (e.g., Mel-frequency spectrograms, such as the acoustic frames 110 of FIG. 1 ) and generates, for each of a number of time steps, an encoded audio representation (e.g., a sigma-based feature vector) corresponding to the transcribed non-synthesized speech utterance 304 at the corresponding time step. s) 314 as output. In parallel, the alignment model 600 receives a transcription 302 corresponding to the same unsynthesized speech utterance 304 and generates an alignment output 602 according to Equation 1. Additionally or alternatively, the alignment model 600 can receive an unspoken text utterance 320 and generate an alignment output 602 according to Equation 2. The text encoder 202 receives the alignment output 602 as input and for each of a number of time steps generates an encoded text representation

[0052]

number

[0053] 314 is generated as output. The audio encoder 210 may include a shared encoder 250 that receives the encoded text representation 312 as an input and generates as an output a first encoded shared representation 322. The shared encoder 250 may also receive the encoded audio representation 314 as an input and generate as an output a second encoded shared representation 324. The auxiliary decoder 390 receives the first and second encoded shared representations 322, 324 as input and generates as output corresponding first and second probability distributions 392, 294 for possible speech recognition hypotheses.

[0054] The alignment masked loss module 850 receives the first probability distribution 392 corresponding to the encoded text representation 312 and the second probability distribution 394 corresponding to the encoded audio representation 314 and generates an alignment loss 852 as follows:

[0055]

number

[0056] The alignment loss 852 from Equation 4 can be applied to the masked, sampled, and encoded text representation 312 in the frequency and time domains. In particular, the alignment loss 852 can be used as a training target for both paired and unpaired training data. The training process 800 can use the alignment loss 852 to update parameters of the alignment model 600. For example, the training process 800 can update parameters of the duration predictor 620 and / or the upsampler 630 (FIG. 6).

[0057] Referring again to FIG. 3A, in some implementations, the audio encoder 210 includes a speech encoder 204 and a text encoder 202, which are described in more detail with reference to FIGs. 3B and 3C. In the illustrated example, the audio encoder 210 (or alternatively the speech encoder 204 or the text encoder 202 (FIGS. 3B and 3C)) includes a conformer encoder including a stack of conformer blocks. Each of the conformer blocks includes a series of layers that are multi-head self-attention, depthwise convolutional, and feedforward. Alternatively, the audio encoder 210 may include other types of encoders (e.g., transformer encoders) with a stack of self-attention layers / blocks. The conformer encoder 210 may be naturally divided into a feature encoder including a convolutional subsampling block 212 and a context network including a stack of linear layers 214 and conformer blocks 216. In some implementations, the convolutional subsampling block 212 has two two-dimensional convolutional layers, both with stride (2,2), thus reducing the feature sequence length by a factor of four. The convolutional subsampling block 212 receives as input a sequence of input features / vectors (e.g., Mel-frequency spectrograms, such as the acoustic frames 110 in FIG. 1 ) associated with each transcribed non-synthesized speech utterance 304 and each untranscribed non-synthesized speech utterance 306, and generates as output, for each of a plurality of output steps, encoded audio features 211 corresponding to each one of the transcribed non-synthesized speech utterances 304 or each one of the untranscribed non-synthesized speech utterances 306. The convolutional subsampling block 212 may receive as input each of the alignment outputs 602, and may generate as output, for each of a plurality of output steps, encoded text features 213 corresponding to each one of the alignment outputs 602.

[0058] The encoded audio and text features 211, 213 (i.e., interchangeably referred to as "encoded features 211, 213") output from the convolutional subsampling block 212 can be sent to a masking module 218, where a portion of the encoded features 211, 213 are randomly selected and replaced with a trained feature vector shared among all masked time steps to provide corresponding masked encoded audio features 211, 211m and masked encoded text features 213, 213m. In some examples, the masking module 218 masks the randomly selected encoded features 211, 213 for masking by randomly sampling without replacement a certain percentage p of all time steps starting from a starting index, and then masks the subsequent M consecutive time steps from all sample indexes, whereby some spans may overlap. After the masking is applied, the linear layer 214 and conformer block 216 of the context network receive the masked encoded features 211m (or the encoded features 211, 213 not selected by the masking module 218) and output the corresponding contrast context vectors (i.e., encoded representations) 215 from the masked encoded features 211m, 213m. Furthermore, the quantizer 217 receives the encoded features 211, 213 as inputs and generates a quantized vector (i.e., target context vector) 219 as output. Then, the contrast loss module 315 calculates the contrast loss (L) between the contrast context vectors (i.e., encoded representations) 215 and the target context vector 219 at the masked positions as follows: w2v )316 is derived.

[0059]

number

[0060] In the formula, c tis the contrast context vector 215 centered at the masked time step t, and q t q t and represents the target context vector 219 at time step t in a set of K+1 candidate target context vectors 219 containing K distractors. The distractors can be uniformly sampled from other masked time steps of the same utterance.

[0061] The contrastive loss 316 is optimized between the contrast context vector 215 and the target context vector 219 at the masked positions. After the audio encoder 210 has converged on the untranscribed non-synthesized speech utterance 306, the pre-training procedure is repeated on both the alignment output 602 corresponding to the unspoken text utterance 320 and the transcribed non-synthesized speech utterance 304. Thus, the contrastive loss (L w2v ) is optimized for both real / human (non-synthetic) and unspoken text utterances 320 represented by the alignment output 602, with additional auxiliary losses from the transcribed non-synthetic speech utterances 304 and the alignment output 602, as described in more detail below with reference to FIG. 3B. Thus, the training process 300a pre-trains the audio encoder 210 with the derived contrastive loss 316. The derived contrastive loss 316 is applied to the corresponding encoded features 211, 213 associated with each alignment output 602, each transcribed non-synthetic speech utterance 304, and each untranscribed non-synthetic speech utterance 306 provided as input to the audio encoder 210. Pre-training the audio encoder 210 may include updating parameters of the audio encoder 210 based on the contrastive loss 316.

[0062] Referring to FIG. 3B, the supervised loss section 300b of the training process 300 is configured to inject lexical information into the audio encoder 210 during pre-training based on supervised loss terms 342, 344 derived from the transcribed unsynthesized speech utterance 304 and the alignment output 602 corresponding to the unspoken text utterance 320 output by the alignment model 600. In particular, the supervised loss section 300b utilizes one or more auxiliary decoders 390 for generating the supervised loss terms 342, 344. The auxiliary decoders 390 may include a connectionist temporal classification (CTC) decoder, a listen-attend-spell (LAS) decoder, or an RNN-T decoder. These auxiliary decoders 390 may include at least one of a phoneme decoder configured to decode a sequence of phonemes, or a wordpiece decoder configured to decode a sequence of wordpieces. The auxiliary decoder 390 may also include a grapheme decoder configured to decode a sequence of graphemes.

[0063] In the supervised loss unit 300b, the text encoder 202 of the audio encoder 210 is configured to receive the alignment output 602 (i.e., text embedding values) from the alignment model 600, and the speech encoder 204 is configured to receive the transcribed non-synthesized speech utterance 304. That is, the text encoder 202 of the audio encoder 210 generates an encoded text representation 312 for the alignment output 602 (e.g., corresponding to the unspoken text utterance 320), and the speech encoder 204 of the audio encoder 210 generates an encoded audio representation 314 for the speech input (i.e., the transcribed non-synthesized speech utterance 304). Here, both the encoded text representation 312 and the encoded audio representation 314 may not necessarily be compatible with the auxiliary decoder 390. Therefore, the audio encoder 210 may also include a shared encoder 250. The shared encoder 250 receives the encoded text representation 312 as input and generates a first encoded shared representation 322 (e.g., the first encoded shared representation 322). text) as output. Additionally, the shared encoder 250 receives the encoded audio representation 314 as input and produces a second encoded shared representation (e sup ) 324 as output. Thus, the joint encoder 250 generates first and second encoded joint representations 322, 324 in a shared latent representation space compatible with the auxiliary decoder 390.

[0064] In particular, the shared encoder 250 receives as input each encoded text representation 312 corresponding to the alignment output 602 generated from the unspoken text utterance 320, and for each of a number of time steps, generates a first encoded shared representation (e text ) 322 as an output. An auxiliary decoder 390, including a phoneme decoder or wordpiece decoder, receives as an input each first encoded shared representation 322 output from the shared encoder 250 and generates as an output a first probability distribution 392 over possible speech recognition hypotheses for a corresponding alignment output 602 at a corresponding time step. In some examples, the first probability distribution 392 over possible speech recognition hypotheses includes one of possible phoneme labels, possible wordpiece labels, or possible grapheme labels. The supervised loss module 340 can then determine an alignment output loss term 342 based on the first probability distribution 392 over possible speech recognition hypotheses for the alignment output 602 corresponding to the unspoken text utterance 320. Here, the corresponding unspoken text utterance 320 from which the alignment output 602 is generated also serves as the ground truth transcription 302. The supervised loss unit 300b may pre-train the audio encoder 210 with the alignment output loss term 342 by updating parameters of the audio encoder 210 using the alignment output loss term 342.

[0065] Similarly, in the supervised loss unit 300b, the shared encoder 250 receives as input each transcribed encoded audio representation 314 corresponding to the non-synthesized speech utterance 304 and, for each of a number of time steps, generates a second encoded shared representation (e sup ) 324 as an output. An auxiliary decoder 390, including a phoneme or wordpiece decoder, receives as an input each second encoded shared representation 324 output from the shared encoder 250 and generates as an output a second probability distribution 394 over possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance 304 at the corresponding time step. In some examples, the second probability distribution 394 over possible non-synthetic speech recognition hypotheses includes one of possible phoneme labels, possible wordpiece labels, or possible grapheme labels. The supervised loss module 340 can then determine a non-synthetic speech loss term 344 based on the second probability distribution 394 over possible non-synthetic speech recognition hypotheses and the corresponding transcription 302 paired with the transcribed non-synthetic speech utterance 304. Here, the corresponding transcription 302 serves as a ground truth transcription and may include a sequence of target phonemes, target wordpieces, and / or target graphemes. The supervised loss unit 300 b may pre-train the audio encoder 210 with the non-synthetic speech loss term 344 by updating parameters of the audio encoder 210 using the non-synthetic speech loss term 344 .

[0066] In some implementations, the supervised loss portion 300b of the training process 300 uses another auxiliary decoder 390 to generate a first encoded shared representation (e text) 322, generates a third probability distribution 393 for possible speech recognition hypotheses, so that the supervised loss module 340 can determine the other alignment output loss term 342 based on the third probability distribution 393 and the unspoken text utterance 320 corresponding to the alignment output 602. Here, the other auxiliary decoder 390 includes the other of the phoneme decoder, the wordpiece decoder, or the grapheme decoder, and the third probability distribution 393 for possible speech recognition hypotheses includes the other of the possible phoneme labels, the possible wordpiece labels, or the possible grapheme labels. In these implementations, the other auxiliary decoder 290 also generates a fourth probability distribution 395 for possible non-synthesized speech recognition hypotheses for the corresponding second encoded shared representation 324 at the corresponding time step, so that the supervised loss module 340 can determine the other non-synthesized speech loss term 344 based on the fourth probability distribution 395 and the corresponding transcription 302 paired with the transcribed non-synthesized speech representation 304. Here, a fourth probability distribution 395 for possible non-synthetic speech recognition hypotheses includes the other of the possible phoneme labels, the possible wordpiece labels, or the possible grapheme labels. The supervised loss portion 300b of the training process 300 can similarly pre-train the audio encoder 210 with other alignment output loss terms 342 and other non-synthetic speech loss terms 344.

[0067] The untranscribed non-synthesized speech utterance 306 and the unspoken text utterance 320 each correspond to “unpaired” training data and thus the unspoken text utterance (X text )320 derived from the control loss (L w2v ) 316 into the supervised loss J associated with the alignment output loss term 342 as follows: aux Together with this, the unspoken text loss function J text can be obtained.

[0068]

number

[0069] Similarly, untranscribed non-synthesized speech (X unsup )306 derived from the control loss (L w2v ) 316 is expressed as the unsupervised speech loss function J unsup_speech can be used to represent

[0070] J unsup_speech =J w2v (x * |θ e ) (7) During pre-training of the audio encoder 210, the alignment outputs 602 and the untranscribed, non-synthesized speech 306 can be separated or mixed within each batch. To force the audio encoder 210 to learn a representation that is valid for both the alignment outputs 602 corresponding to the unspoken text utterances 320 and the non-synthesized (human / real) speech, the loss function J text Combined with Equation 5 and Equation 6, the unpaired data loss function J unpaired We apply a loss mask σ when determining:

[0071] J unpaired =σJ text +(1-σ)J speech (8) The transcribed non-synthetic speech utterances 304 correspond to the “paired” and “supervised” training data and are therefore associated with the non-synthetic speech loss term 344, a derived contrastive loss L w2v and the derived supervised loss J aux Combined, we have a paired data loss function J paired can be obtained.

[0072] J paired =L w2v (x|θ e )+L aux (y|x,θ e ,θ d ) (9) 3C, the consistency regularizer (i.e., modality matching) 300c of the training process 300 regularizes the transcribed non-synthesized speech utterance (Xsup ) 304 and the training utterance pairs 301 each containing a corresponding transcribed non-synthesized speech utterance 304 and a paired alignment output 604 of the same utterance. cons The consistency loss term 352 between the transcribed non-synthetic speech utterance 304 and the paired alignment output 604 of the same training utterance is thus associated with the same ground truth transcription. In essence, the consistency loss term 352 between the transcribed non-synthetic speech utterance 304 and the paired alignment output 604 of the same training utterance provides an aspect of unsupervised training by encouraging the audio encoder 210 to behave consistently, regardless of the supervised loss term between the ground truth transcription 302 and each of the non-synthetic speech recognition hypotheses output by the auxiliary decoder 390 and the speech recognition hypotheses output by the auxiliary decoder 390, regardless of whether the training utterance belongs to a non-synthetic speech utterance (i.e., audio training data) or an alignment output (i.e., text training data).

[0073] Similar to the alignment outputs 602 generated from the unspoken text utterance 320 of Figure 3B, the alignment model 600 may generate each paired alignment output 604 using a corresponding transcription 302 paired with a transcribed non-synthesized speech utterance 304, where the non-synthesized speech representation 304 is associated with a paired alignment output 604 generated by the alignment model 600 that maps the unspoken text utterance 320 to speech frames.

[0074] In the consistency regularizer 300c, the text encoder 202 receives as input each paired alignment output 604 and generates as output, for each of a number of time steps, an encoded text representation 313 corresponding to the paired alignment output 604 at the corresponding time step. The shared encoder 250 receives as input the encoded text representation 313 and generates a first encoded shared representation (e * sup ) 323 as output. An auxiliary decoder 390, including a phoneme decoder or wordpiece decoder, receives as input each first encoded shared representation 323 output from the shared encoder 250 and generates as output a first probability distribution 311 over possible speech recognition hypotheses for the corresponding paired alignment output 604 at the corresponding time step. In some examples, the first probability distribution 311 over possible speech recognition hypotheses includes one of the possible phoneme labels or possible wordpiece labels.

[0075] Similarly, the speech encoder 204 receives as input each transcribed non-synthesized speech utterance 304 as a sequence of features / vectors (e.g., Mel-frequency spectrograms such as acoustic frames 110 in FIG. 1 ) and generates as output, for each of a number of time steps, an encoded audio representation 314 corresponding to the transcribed non-synthesized speech utterance 304 at the corresponding time step. The shared encoder 250 receives as input the encoded audio representation 314 and generates a second encoded shared representation 324 (e.g., sup ) 324 as an output. An auxiliary decoder 390, which may include a phoneme decoder or wordpiece decoder, receives as an input each second encoded shared representation 324 output from the shared encoder 250 and generates as an output a second probability distribution 394 over possible non-synthesized speech recognition hypotheses for the corresponding transcribed non-synthesized speech utterance 304 at the corresponding time step. In some examples, the second probability distribution 394 over possible non-synthesized speech recognition hypotheses includes one of the possible phoneme labels or possible wordpiece labels.

[0076] Continuing to refer to FIG. 3C , the consistency regularizer 300 c of the training process 300 further determines, at each of a plurality of time steps for each training utterance pair 301, a consistency loss term (J) for the corresponding training utterance pair 301 based on the first probability distribution 311 for the possible speech recognition hypotheses and the second probability distribution 394 for the possible non-synthetic speech recognition hypotheses. cons For example, the training process 300 may use a consistency loss term module 350 that is configured to receive, at each time step, the corresponding unsynthesized speech and the speech recognition results 311, 394 output by the auxiliary decoder 390 and to determine a consistency loss term 352 for the corresponding training utterance pair 301 at the time step.

[0077] In some examples, the consistency regularizer portion 300c of the training process 300 may be configured to calculate the Kullback-Leibler divergence (D) between a first probability distribution 311 for possible speech recognition hypotheses and a second probability distribution 394 for possible non-synthetic speech recognition hypotheses. KL ) to determine the consistency loss term 352. D KL The consistency loss term 352 based on can be expressed as:

[0078]

number

[0079] Here, the consistency loss term 352 determined for the training utterance pair 301 at each time step provides an “unsupervised” loss term that is independent of the accuracy of the auxiliary decoder 390 (e.g., independent of the supervised loss terms 342, 344 in FIG. 3B ) and can therefore be used to update parameters of the audio encoder 210 to increase the consistency between the alignment output and the non-synthetic speech representation of the same utterance. In batch training, the consistency loss term 352 may correspond to an average loss term obtained for the batch. In other words, the consistency loss term 352 allows the audio encoder 210 to learn to behave in the same way regardless of whether the training utterance belongs to the non-synthetic speech or the alignment output, e.g., allows the audio encoder 210 to learn to make predictions of consistent encoded representations for both the non-synthetic speech (e.g., real / human speech) and the alignment output of the same training utterance.

[0080] Finally, the training process 300 calculates an unpaired data loss function (J unpaired ), paired data loss function (J paired ), and the consistency loss term (J cons ) and the total loss term J tts4pretrain2 This can be expressed as follows: J tts4pretrain2 =J unpaired +λ1J paired +λ2J cons (11) where λ1 may be equal to 1.0 and λ2 is equal to 0.1. The training process 300 includes a total loss term J tts4pretrain2The audio encoder 210 may be pre-trained using the ensemble 300, by updating its parameters to effectively teach the audio encoder 210 to learn the shared representation between speech and text. After pre-training the audio encoder 210, the training process 300 may fine-tune the pre-trained audio encoder with transcribed speech utterances, which may include supervised training samples of both alignment outputs corresponding to unspoken text utterances 320 and non-synthesized speech (e.g., human speech).

[0081] In some implementations, the training process 300 for pre-training the audio encoder 210 applies encoder consistency regularization. Unlike the decoder consistency regularization applied to the auxiliary decoder(s) in the consistency regularizer 300c, which requires assumed labels (e.g., the transcription 302 and the unspoken text utterance 320), the encoder consistency regularization has the advantage that it is applicable to all the training data 304, 306, 320, since it does not require assumed labels. The encoder consistency regularization can be applied by a Hierarchical Contrastive Consistency Regularization (HCCR) method. In the hierarchical contrastive consistency regularization, the encoder activation values ​​e, e* from the original / unaugmented and augmented speech are projected through an auxiliary network to generate z and z*. The positive and negative pairs are then constructively computed using a contrastive loss l. t,z,z* is calculated as follows:

[0082]

number

[0083] Specific to HCCR, a convolutional neural network (CNN) projection network can compute projection values ​​for increasing length segments (30, 50, 120 ms) of encoder activation values ​​e, resulting in three views (V) to provide negative examples from the same utterance in short segments and other utterances in a batch of 120 ms segments. Thus, the HCCR loss can be computed for the alignment output 602 generated from the transcribed non-synthesized speech utterance 304 (paired speech), the untranscribed non-synthesized speech utterance 306 (unpaired speech), and the unspoken text utterance 320 as follows:

[0084]

number

[0085] The HCCR loss calculated by Equation 13 is the total loss term J used for pre-training the audio encoder 210. tts4pretrain2 can be added to Equation 11 using a factor of 1e-3 as part of

[0086] The above embodiment describes the training process 300 for pre-training the audio encoder 210. However, it should be understood that the training process 300 may also be used to train / pre-train a monolingual ASR model 200 or a multilingual ASR model 200. In some examples, the training process 300 may be used to train an end-to-end ASR model with a decoder structure (i.e., not pre-trained), or to fine-tune an ASR model to perform downstream tasks such as speech translation or natural language understanding. Furthermore, the training process 300 may be used with each of the unspoken text utterances 320, the transcribed non-synthesized speech utterances 304, and the untranscribed non-synthesized speech utterances 306, or with some combination thereof.

[0087] 4, a contrastive unspoken text selection process 400 can select unspoken text utterances 320 from a large unspoken text corpus 402 for use in pre-training the audio encoder 210 such that the selected unspoken text utterances 320 are most similar to the particular domain for which the audio encoder 210 is pre-trained and learning. That is, the text selection process 400 can identify unspoken text within and close to the domain from the unspoken text corpus 402 for inclusion in the unspoken text utterances 320 for use in pre-training the audio encoder 210. In particular, the unspoken text utterances 320 selected by the selection process 400 allow for the synthesis of separate utterances on the fly during batch construction, such that new speaker embeddings z and latent variables Z can be sampled for each unspoken text utterance 320 in the batch.

[0088] The corpus of unspoken text 402 includes a large number of unspoken text utterances 320, 320a-n across a wide range of domains, with much greater linguistic diversity than the particular domain for which the audio encoder 210 is trained. As mentioned above, the set of transcribed non-synthesized speech utterances 304 may be domain-specific in that they relate to a particular domain, and each non-synthesized speech utterance 304 is paired with a corresponding transcription 302. The corpus of unspoken text 402 may be stored in the same or a different data store 401 as the spoken training utterances 304. The corpus of unspoken text 402 may change dynamically to incorporate new unspoken text utterances 320. Simply using all the unspoken text utterances 320 in the unspoken text corpus 402 is infeasible for the following reasons: Namely, i) for each sentence, the speech modality requires much more memory for encoding than text, so it is impractical to convert all of the text in the corpus 402; ii) the difference between the transcriptions 302 paired with the transcribed non-synthesized speech utterances 304 and the unspoken text utterances 320 in the unspoken text corpus 402 is enormous, requiring intelligent strategies to balance their contributions.

[0089] The text selection process 400 aims to select a subset of available unspoken text utterances 320 from the unspoken text corpus 402 as data for TTS synthesis, which results in the alignment output generated for pre-training the audio encoder 210 in the contrastive and supervised loss parts 300a, 300b of the training process 300 described above with reference to Figs. 3A and 3B. In other words, the text selection process 400 aims to improve the match between the selected subset of available unspoken text utterances 320 and the specific domain of interest, which results in a reduction in the computational resources required to utilize large amounts of non-domain specific data. Thus, the text selection process 400 reduces computational and memory costs by selecting the unspoken text utterances 320 that best match the specific domain on which the audio encoder 210 is trained.

[0090] In some examples, the text selection process 400 selects a subset of available unspoken text utterances 320 from the corpus 402 that best matches a particular domain by simply providing a domain identifier (not shown) associated with the particular domain as an input to a background LM 406 that has already been trained across the unspoken text corpus 402. As previously discussed, the unspoken text corpus 402 spans a number of different domains. In these examples, the background LM 406 may include a maximum entropy (MaxEnt LM) that can optionally accept a domain identifier as an input, as described in U.S. Patent No. 9,842,592, filed February 12, 2014, the contents of which are incorporated herein by reference in their entirety. Here, the domain identifier associated with the particular domain may enable the MaxEnt LM to output a subset of available unspoken text utterances 320 from the corpus 402 that are likely to contain words and / or phrases related to the particular domain. In some configurations, rather than evaluating the likelihood of words, the statistical language model operates in reverse mode to randomly generate text phrases that match the statistical distribution of words associated with a particular domain.

[0091] In a further example, as shown in FIG. 4, a text selection process 400 uses a transcription 302 paired with a transcribed non-synthesized speech utterance 304 spoken by a human speaker to select a subset of available unspoken text utterances 320 from the corpus 402 that best match a particular domain, where the transcribed non-synthesized speech utterance 304 includes words, phrases, and / or other terms related to the particular domain. Optionally, in addition to or instead of the transcription 304 paired with the transcribed non-synthesized speech utterance 304, a set of different transcribed utterances related to a particular domain can be used to select the unspoken text utterances 320. This provides the advantage that the transcribed non-synthesized speech utterances 304 do not all need to belong to a particular domain.

[0092] During the first stage (Stage A), the unspoken text selection process 400 builds two language models 404, 406 to enable contrastive selection of unspoken text utterances 320. Here, a domain-specific LM 410 is trained with each transcription 302 in the set of transcribed non-synthesized speech utterances 304. The set of transcribed non-synthesized speech utterances 304 is assumed to belong to the particular domain that the audio encoder 210 is trained to learn. Meanwhile, a background LM 406 is trained with each unspoken text utterance 320 in the entire unspoken text corpus 402. As mentioned above, the unspoken text corpus 402 spans many different domains. In some examples, the first stage builds the two language models 404, 406 using n-gram language model training. In some examples, the first stage builds the two language models 404, 406 using neural network language model training.

[0093] During a second state (Stage B), the unspoken text selection process 400 uses two contrasting LMs 404, 406 to generate a first probability associated with each word in the unspoken text utterance 320 that appears in the domain-specific LM 404.

[0094]

number

[0095] A second probability associated with each word in the unspoken text utterance 320 that appears in the background LM 406 is determined by determining

[0096]

number

[0097] Then, for each unspoken text utterance 320 in the unspoken text corpus 402, the process 400 determines a score S in a scorer 408 based on the first probability, the second probability, and the number of words #(w) that appear in the corresponding unspoken text utterance 320. For example, the score S for each unspoken text utterance 320 may be calculated as follows:

[0098]

number

[0099] After determining the scores, the unspoken text selection process 400 selects the unspoken text utterances 320 with the N best scores S because they best match the particular domain. The text corpus 402 may contain billions of unspoken text utterances 320. The unspoken text utterances 320 selected by the selection process 400 may contain millions of utterances and thus may far exceed the number of untranscribed, unsynthesized speech utterances 304 spoken by human speakers. As noted above, the content of the unspoken text utterances 320 increases the linguistic diversity of the particular domain that the audio encoder 210 learns by training, while the corresponding alignment output 602 generated from the unspoken text utterances 320, when the acoustic encoder 210 is integrated within the ASR model 200, increases the acoustic / linguistic diversity of the speech that the acoustic encoder 210 is encoding as part of the speech recognition process.

[0100] FIG. 5 illustrates an example projection space 500 of the alignment output and encoder representations of non-synthetic (real / human) speech utterances. After introducing consistency regularization via the consistency regularizer 300c illustrated in FIG. 3C for pre-training the audio encoder, the resulting learned speech and text encoder representations are much closer to each other compared to the speech and text encoder representations without applying consistency regularization. Thus, as is evident from the projection space 500, by using supervised training data (i.e., transcribed non-synthetic speech utterances) to pre-train the audio encoder 210, improved shared speech and text representations are effectively generated.

[0101] 9 is a flow chart illustrating an example flow of operations for a method 900 of pre-training audio encoder 210 to jointly learn a shared representation of speech and text. Method 900 can be performed on data processing hardware 1010 (FIG. 10) using instructions stored in memory hardware 1020 (FIG. 10). Data processing hardware 1010 and memory hardware 1020 can reside on a remote computer / server 201 of FIG. 1 that corresponds to computing device 1000 (FIG. 10).

[0102] At operation 902, the method 900 includes receiving training data including unspoken text utterances 320, untranscribed non-synthesized speech utterances 306, and transcribed non-synthesized speech utterances 304. Each unspoken text utterance 320 is not paired with any corresponding speech utterance of the non-synthesized speech. Each untranscribed non-synthesized speech utterance 306 is not paired with a corresponding transcription. Each transcribed non-synthesized speech utterance 304 is paired with a corresponding transcription 302. At operation 904, the method 900 includes using the alignment model 600 to generate a corresponding alignment output 602 for each unspoken text utterance 320 of the received training data. At operation 906, the method 900 includes pre-training the audio encoder 210 with the alignment outputs 602 generated to correspond to the unspoken text utterance 320, the untranscribed non-synthesized speech utterance 306, and the transcribed non-synthesized speech utterance 304 to teach the audio encoder 210 to jointly learn a shared representation of speech and text.

[0103] A software application (i.e., a software resource) can refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0104] Non-transient memory may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) in temporary or permanent primitives used by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0105] 10 is a schematic diagram of an exemplary computing device 1000 that can be used to implement the systems and methods described herein. The computing device 1000 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the scope of the embodiments and / or claims of the invention described herein.

[0106] The computing device 1000 includes a processor 1010, a memory 1020, a storage device 1030, a high-speed interface / controller 1040 that connects to the memory 1020 and a high-speed expansion port 1050, and a low-speed interface / controller 1060 that connects to a low-speed bus 1070 and the storage device 1030. Each of the components 1010, 1020, 1030, 1040, 1050, and 1060 are interconnected using various buses and may be mounted on a shared motherboard or in other manners as desired. The processor 1010 is capable of processing instructions for execution within the computing device 1000. The instructions include instructions stored in the memory 1020 or the storage device 1030 for displaying graphical information for a graphical user interface (GUI) on an external input / output device such as a display 1080 connected to the high-speed interface 1040. In other implementations, multiple processors and / or multiple buses may be used along with multiple memories and multiple types of memories as desired. Also, multiple computing devices 1000 may be connected, with each device performing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0107] The memory 1020 stores information non-transiently within the computing device 1000. The memory 1020 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 1020 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 1000. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0108] The storage device 1030 can provide mass storage for the computing device 1000. In some embodiments, the storage device 1030 is a computer-readable medium. In various different implementations, the storage device 1030 can be an array of devices, including floppy disk devices, hard disk devices, optical disk devices, or tape devices, flash memory or other similar solid-state memory devices, or devices in the form or other configuration of a storage area network. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-readable medium or machine-readable medium, such as the memory 1020, the storage device 1030, or a memory on the processor 1010.

[0109] The high-speed controller 1040 manages the bandwidth-intensive operations of the computing device 1000, while the low-speed controller 1060 manages the bandwidth-intensive operations. This allocation of roles is merely exemplary. In some implementations, the high-speed controller 1040 is connected to the memory 1020, the display 1080 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 1050 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 1060 is connected to the storage device 1030 and the low-speed expansion port 1090. The low-speed expansion port 1090, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be connected to one or more input / output devices, such as a keyboard, pointing device, scanner, etc., or may be connected to a network device, such as a switch or router, for example, via a network adapter.

[0110] The computing device 1000 may be implemented in many different forms, as shown in the figure: for example, it may be implemented as a standard server 1000a or as a repetition within a group of such servers 1000a, as a laptop computer 1000b, or as part of a rack server system 1000c.

[0111] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations include implementations in one or more computer programs executable and / or translatable into machine language on a programmable system including at least one programmable processor. The at least one programmable processor may be special purpose or general purpose and may be coupled to transmit and receive data and instructions to and from a storage system, at least one input device, and at least one output device.

[0112] These computer programs (also referred to as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to carry machine instructions and / or data to a programmable processor.

[0113] The processes and logic flows described herein may be performed by one or more programmable processors (also referred to as data processing hardware) that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or the computer is also operatively coupled to such mass storage devices to receive data from such mass storage devices, transmit data to such mass storage devices, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0114] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and a pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, such as acoustic input, speech input, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to a device used by the user (e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser).

[0115] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (900) that, when executed on data processing hardware (1010), causes the data processing hardware (1010) to perform operations, the operations comprising: receiving training data (304, 306, 320), said training data (304, 306, 320) comprising: a plurality of unspoken text utterances (320), each unspoken text utterance (320) not paired with any corresponding spoken utterance of unsynthesized speech; a plurality of untranscribed, non-synthesized speech utterances (306), each of which is not paired with a corresponding transcription; and receiving the training data (304, 306, 320) including a plurality of transcribed non-synthesized speech utterances (304), each transcribed non-synthesized speech utterance (304) paired with a corresponding transcription (302); generating, for each unspoken text utterance (320) of said received training data (304, 306, 320), a corresponding alignment output (602) using an alignment model (600); pre-training an audio encoder (210) with the alignment outputs (602) generated to correspond to the plurality of unspoken text utterances (320), the plurality of untranscribed non-synthesized speech utterances (306), and the plurality of transcribed non-synthesized speech utterances (304) to teach the audio encoder (210) to jointly learn a shared representation of speech and text; A computer-implemented method (900) comprising:

2. 2. The computer-implemented method of claim 1, wherein the audio encoder comprises a stack of self-attention layers, each of which includes a multi-head self-attention mechanism.

3. Pre-training the audio encoder (210) comprises: For each untranscribed, non-synthesized speech utterance (306), generating a corresponding encoded representation (215) of said untranscribed, non-synthesized speech utterance (306); pre-training the audio encoder (210) with a contrastive loss (316) applied to the corresponding encoded representation (215) of the untranscribed, non-synthesized speech utterance (306); For each alignment output (602), generating a corresponding encoded representation (215) of said alignment output (602); pre-training the audio encoder (210) with a contrastive loss (316) applied to the corresponding encoded representation (215) of the alignment output (602); For each transcribed non-synthesized speech utterance (304), generating a corresponding encoded representation (215) of the transcribed non-synthesized speech utterance (304); pre-training the audio encoder (210) with a contrastive loss (316) applied to the corresponding encoded representation (215) of the transcribed non-synthesized speech utterance (304); The computer-implemented method (900) of claim 1 or 2, comprising:

4. Pre-training the audio encoder (210) comprises: At each of a plurality of time steps of each alignment output (602), generating a first probability distribution (392) over possible synthetic speech recognition hypotheses for the corresponding alignment output (602) using an auxiliary decoder (390); determining an alignment output loss term (342) based on the first probability distribution (392) over possible synthetic speech recognition hypotheses and the unspoken text utterance (320) that corresponds to the alignment output (602); pre-training the audio encoder (210) based on the alignment output loss term (342); For each of a plurality of time steps of each transcribed non-synthesized speech utterance (304), generating, using the auxiliary decoder (390), a second probability distribution (394) over possible non-synthesized speech recognition hypotheses for the corresponding transcribed non-synthesized speech utterance (304); determining a non-synthesized speech loss term (344) based on the second probability distribution (394) over possible non-synthesized speech recognition hypotheses and the corresponding transcription (302) paired with the transcribed non-synthesized speech utterance (304); pre-training the audio encoder (210) based on the non-synthetic speech loss terms (344); The computer-implemented method (900) of any one of claims 1 to 3, comprising:

5. 5. The computer-implemented method of claim 4, wherein the auxiliary decoder comprises one of a connection time classification (CTC) decoder, a listen-attend-spell (LAS) decoder, or a recurrent neural network transducer (RNN-T) decoder.

6. the first probability distribution (392) for possible synthetic speech recognition hypotheses includes one of possible phoneme labels or possible wordpiece labels; the second probability distribution (394) for possible non-synthetic speech recognition hypotheses includes the one of the possible phoneme labels or the possible wordpiece labels.

6. A computer-implemented method (900) according to claim 4 or 5.

7. The computer-implemented method (900) of any one of claims 1 to 6, wherein the audio encoder (210) comprises a text encoder (202), a speech encoder (204), and a shared encoder (250).

8. The operation further comprises: For each alignment output (602), determining an encoded text representation (312) of the alignment output (202) using the text encoder (202); generating a first encoded shared representation (322) of the alignment output (602) in a shared latent representation space using the shared encoder (250); For each transcribed non-synthesized speech utterance (304), determining an encoded audio representation (314) of the transcribed non-synthesized speech utterance (304) using the speech encoder (204); generating a second encoded shared representation (324) of the transcribed non-synthesized speech utterance (304) in a shared latent representation space using the shared encoder (250); The computer-implemented method (900) of claim 7, comprising:

9. Generating the corresponding alignment output (602) for each unspoken text utterance (320) of the received training data (304, 306, 320) comprises: extracting an initial text representation (612) from the unspoken text utterance (320); predicting a text chunk duration (622) for each text chunk in the unspoken text utterance (320); upsampling the initial text representation (612) using the predicted text chunk durations (622) for each text chunk in the unspoken text utterance (320); The computer-implemented method (900) of any one of claims 1 to 8, comprising:

10. The operations further include training the alignment model (600), the training comprising: generating an encoded audio representation (314) of the transcribed non-synthesized speech utterance (304) using a speech encoder (204); determining an alignment output (602) for a transcription (302) corresponding to the transcribed non-synthesized speech utterance (304) using the alignment model (600); generating an encoded text representation (312) of the alignment output (602) using a text encoder (202); updating parameters of the alignment model (600) based on a comparison of the encoded audio representation (314) of the transcribed non-synthesized speech utterance (304) and the encoded text representation (312) of the alignment output (612); The computer-implemented method (900) of any one of claims 1 to 9, performed by:

11. Data processing hardware (1010); and memory hardware (1020) in communication with the data processing hardware (1010), the memory hardware (1020) storing instructions that, when executed by the data processing hardware (1010), cause the data processing hardware (1010) to perform an operation, the operation being: receiving training data (304, 306, 320), said training data (304, 306, 320) comprising: a plurality of unspoken text utterances (320), each unspoken text utterance (320) not paired with any corresponding spoken utterance of unsynthesized speech; a plurality of untranscribed, non-synthesized speech utterances (306), each of which is not paired with a corresponding transcription; and receiving the training data (304, 306, 320) including a plurality of transcribed non-synthesized speech utterances (304), each transcribed non-synthesized speech utterance (304) paired with a corresponding transcription (302); generating, for each unspoken text utterance (320) of said received training data (304, 306, 320), a corresponding alignment output (602) using an alignment model (600); pre-training an audio encoder (210) with the alignment outputs (602) generated to correspond to the plurality of unspoken text utterances (320), the plurality of untranscribed non-synthesized speech utterances (306), and the plurality of transcribed non-synthesized speech utterances (304) to teach the audio encoder (210) to jointly learn a shared representation of speech and text; Including, the system.

12. The system of claim 11 , wherein the audio encoder comprises a stack of self-attention layers, each of which includes a multi-head self-attention mechanism.

13. Pre-training the audio encoder (210) comprises: For each untranscribed, non-synthesized speech utterance (306), generating a corresponding encoded representation (215) of said untranscribed, non-synthesized speech utterance (306); pre-training the audio encoder (210) with a contrastive loss (316) applied to the corresponding encoded representation (215) of the untranscribed, non-synthesized speech utterance (306); For each alignment output (602), generating a corresponding encoded representation (215) of said alignment output (602); pre-training the audio encoder (210) with a contrastive loss (316) applied to the corresponding encoded representation (215) of the alignment output (602); For each transcribed non-synthesized speech utterance (304), generating a corresponding encoded representation (215) of the transcribed non-synthesized speech utterance (304); pre-training the audio encoder (210) with a contrastive loss (316) applied to the corresponding encoded representation (215) of the transcribed non-synthesized speech utterance (304); The system (100) of claim 1 or 2, comprising:

14. Pre-training the audio encoder (210) comprises: At each of a plurality of time steps of each alignment output (602), generating a first probability distribution (392) over possible synthetic speech recognition hypotheses for the corresponding alignment output (602) using an auxiliary decoder (390); determining an alignment output loss term (342) based on the first probability distribution (392) over possible synthetic speech recognition hypotheses and the unspoken text utterance (320) that corresponds to the alignment output (602); pre-training the audio encoder (210) based on the alignment output loss term (342); For each of a plurality of time steps of each transcribed non-synthesized speech utterance (304), generating, using the auxiliary decoder (390), a second probability distribution (394) over possible non-synthesized speech recognition hypotheses for the corresponding transcribed non-synthesized speech utterance (304); determining a non-synthesized speech loss term (344) based on the second probability distribution (394) over possible non-synthesized speech recognition hypotheses and the corresponding transcription (302) paired with the transcribed non-synthesized speech utterance (304); pre-training the audio encoder (210) based on the non-synthetic speech loss terms (344); The system (100) according to any one of claims 11 to 13, comprising:

15. 15. The system of claim 14, wherein the auxiliary decoder comprises one of a connection time classification (CTC) decoder, a listen-attend-spell (LAS) decoder, or a recurrent neural network transducer (RNN-T) decoder.

16. the first probability distribution (392) for possible synthetic speech recognition hypotheses includes one of possible phoneme labels or possible wordpiece labels; the second probability distribution (394) for possible non-synthetic speech recognition hypotheses includes the one of the possible phoneme labels or the possible wordpiece labels. A system (100) according to claim 14 or 15.

17. The system (100) of any one of claims 11 to 16, wherein the audio encoder (210) comprises a text encoder (202), a speech encoder (204), and a shared encoder (250).

18. The operation further comprises: For each alignment output (602), determining an encoded text representation (312) of the alignment output (202) using the text encoder (202); generating a first encoded shared representation (322) of the alignment output (602) in a shared latent representation space using the shared encoder (250); For each transcribed non-synthesized speech utterance (304), determining an encoded audio representation (314) of the transcribed non-synthesized speech utterance (304) using the speech encoder (204); generating a second encoded shared representation (324) of the transcribed non-synthesized speech utterance (304) in a shared latent representation space using the shared encoder (250); The system (100) of claim 17, comprising:

19. Generating the corresponding alignment output (602) for each unspoken text utterance (320) of the received training data (304, 306, 320) comprises: extracting an initial text representation (612) from the unspoken text utterance (320); predicting a text chunk duration (622) for each text chunk in the unspoken text utterance (320); upsampling the initial text representation (612) using the predicted text chunk durations (622) for each text chunk in the unspoken text utterance (320); The system (100) according to any one of claims 11 to 18, comprising:

20. The operations further include training the alignment model (600), the training comprising: generating an encoded audio representation (314) of the transcribed non-synthesized speech utterance (304) using a speech encoder (204); determining an alignment output (602) for a transcription (302) corresponding to the transcribed non-synthesized speech utterance (304) using the alignment model (600); generating an encoded text representation (312) of the alignment output (602) using a text encoder (202); updating parameters of the alignment model (600) based on a comparison of the encoded audio representation (314) of the transcribed non-synthesized speech utterance (304) and the encoded text representation (312) of the alignment output (612); The system (100) according to any one of claims 11 to 19, wherein the system (100) is executed by:

Images