Training automatic speech recognition model using aligned text and speech representations without transcribed speech data

Abstract

A method (700) includes receiving training data including a non-spoken text utterance (320) in a target language. Each non-spoken text utterance is not paired with any corresponding spoken utterance of the non-synthetic speech. The method also includes generating a corresponding alignment output (402) for each non-spoken text utterance using an alignment model (400) trained on transcribed speech utterances (304) in one or more training languages each different from the target language. The method further includes generating, using a text encoder (202), a corresponding encoded text representation (312) for each aligned output, and training a speech recognition model (200) on the encoded text representations generated for the aligned outputs. Training the speech recognition model teaches the speech recognition model to learn how to recognize speech in a target language.

Description

Technical Field

[0001] The present disclosure relates to training automatic speech recognition models using aligned text and speech representations without transcribed speech data. Background Art

[0002] Automatic speech recognition (ASR), i.e., the process of taking audio input and transcribing it into text, has largely become an important technology used in mobile devices and other devices. Generally speaking, automatic speech recognition attempts to provide an accurate transcription of what people say by taking audio input (e.g., voice utterances) and transcribing the audio input into text. Based on the continuous development of deep neural networks, modern ASR models continue to improve in terms of accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay between the user speaking and the transcription). However, one challenge in developing deep learning-based ASR models is the need for a large amount of transcribed speech during training. In some instances, non-verbal text data is used to train ASR models to supplement a small set of transcribed speech training data. However, this challenge becomes more complicated when training ASR models using resource-poor languages ​​that include zero available transcribed speech training data. Summary of the invention

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for training an automatic speech recognition model using aligned text and speech representations in the absence of transcribed speech data. These operations include receiving training data including non-verbal text utterances in a target language. Each non-verbal text utterance is not paired with any corresponding spoken utterance of non-synthesized speech. These operations also include using an alignment model to generate a corresponding alignment output for each non-verbal text utterance of the received training data, the alignment model being trained on transcribed speech utterances in one or more training languages ​​that are each different from the target language. These operations also include using a text encoder to generate a corresponding encoded text representation for each alignment output. These operations also include training a speech recognition model on the encoded text representation generated for the alignment output corresponding to the non-verbal text utterance in the target language to teach the speech recognition model to learn how to recognize speech in the target language.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, training a speech recognition model includes training the speech recognition model without using any transcribed speech utterances in a target language for supervised learning. In some examples, the speech recognition model includes an audio encoder and a decoder. The decoder may include a recurrent neural network transducer (RNN-T) architecture. In these examples, the audio encoder includes a text encoder, a speech encoder, and a shared encoder. Here, the encoder may include multiple multi-head self-attention layers. The audio encoder may include a Conformer (conformer) encoder. In these examples, the operations further include adjusting at least one of the audio encoder or decoder according to a language identifier that uniquely identifies the target language. Adjusting at least one of the audio encoder or decoder includes using a residual adapter layer to adjust at least one of the audio encoder or decoder according to the language identifier. The alignment model can be trained on transcribed speech utterances in one or more training languages ​​different from the target language.

[0005] In some implementations, training a speech recognition model includes: using a shared encoder to generate a first encoded shared representation of the aligned output for each aligned output in a shared latent representation space; for each transcribed speech utterance in one or more training languages, using a speech encoder to determine an encoded audio representation of the transcribed speech utterance; and generating a second encoded shared representation of the transcribed speech utterance in the shared latent representation space. Here, training the speech recognition model includes training the speech recognition model on a first encoded shared representation generated for the aligned output corresponding to the non-verbal text utterance in the target language and a second encoded shared representation generated for the transcribed speech utterance in one or more training languages. Each non-verbal text utterance can include a sequence of words, word fragments, phonemes, and / or graphemes. In some examples, these operations further include using a pronunciation model to convert a script of a non-verbal text utterance in the target language into a phonetic representation shared across multiple languages.

[0006] Another aspect of the present disclosure provides a system, which includes data processing hardware and memory hardware, and the memory hardware stores instructions, which, when executed on the data processing hardware, cause the data processing hardware to perform operations. These operations include receiving training data including non-verbal text utterances in a target language. Each non-verbal text utterance is not paired with any corresponding spoken utterance of non-synthesized speech. These operations also include using an alignment model to generate a corresponding alignment output for each non-verbal text utterance of the received training data, and the alignment model is trained on speech utterances of transcriptions of one or more training languages ​​that are different from the target language. These operations also include using a text encoder to generate a corresponding encoded text representation for each alignment output. These operations also include training a speech recognition model on the encoded text representation generated for the alignment output corresponding to the non-verbal text utterance in the target language to teach the speech recognition model to learn how to recognize speech in the target language.

[0007] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, training a speech recognition model includes training the speech recognition model without using any transcribed speech utterances in a target language for supervised learning. In some examples, the speech recognition model includes an audio encoder and a decoder. The decoder may include a recurrent neural network transducer (RNN-T) architecture. In these examples, the audio encoder includes a text encoder, a speech encoder, and a shared encoder. Here, the encoder may include multiple multi-head self-attention layers. The audio encoder may include a Conformer encoder. In these examples, the operations further include adjusting at least one of the audio encoder or decoder according to a language identifier that uniquely identifies the target language. Adjusting at least one of the audio encoder or decoder includes using a residual adapter layer to adjust at least one of the audio encoder or decoder according to the language identifier. The alignment model can be trained on transcribed speech utterances in one or more training languages ​​different from the target language.

[0008] In some implementations, training a speech recognition model includes: using a shared encoder to generate a first encoded shared representation of the aligned output for each aligned output in a shared latent representation space; for each transcribed speech utterance in one or more training languages, using a speech encoder to determine an encoded audio representation of the transcribed speech utterance; and generating a second encoded shared representation of the transcribed speech utterance in the shared latent representation space. Here, training the speech recognition model includes training the speech recognition model on a first encoded shared representation generated for the aligned output corresponding to the non-verbal text utterance in the target language and a second encoded shared representation generated for the transcribed speech utterance in one or more training languages. Each non-verbal text utterance can include a sequence of words, word fragments, phonemes, and / or graphemes. In some examples, these operations further include using a pronunciation model to convert a script of a non-verbal text utterance in the target language into a phonetic representation shared across multiple languages.

[0009] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram of an example speech recognition system.

[0011] Figure 2 is a schematic diagram of an example speech recognition model.

[0012] FIG. 3A to FIG. 3C It is used for training Figure 2 Schematic diagram of an example training process for an audio encoder of a speech recognition model.

[0013] Figure 4 is FIG. 3A to FIG. 3C Schematic diagram of the alignment model used during the example training process of an audio encoder for training a speech recognition model in .

[0014] Figure 5 is a schematic diagram of an example training process for training a duration predictor for an alignment model.

[0015] Figure 6 is a diagram of an example training process for training an upsampler for an alignment model.

[0016] Figure 7 is a flow diagram of an example arrangement of operations of a method for training a speech recognition model using aligned text and speech representations without transcribed speech data.

[0017] Figure 8 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0018] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION

[0019] Training state-of-the-art automatic speech recognition (ASR) models typically requires a large amount of labeled training data, including speech utterances that are each paired with a corresponding transcription (i.e., a true value label). In particular, for resource-poor languages, the cost of obtaining a large amount of labeled training data can be very high. Therefore, recent methods for training ASR models include using a large amount of unlabeled training data (i.e., speech that is not paired with any corresponding transcription and / or non-verbal text that is not paired with any corresponding speech) to supplement a relatively small amount of labeled training data for self-supervised training. However, these methods still require a small amount of labeled training data in the specific language that the ASR model is trained to recognize. However, in some instances, there is zero available labeled training data for certain resource-poor languages. Therefore, in these instances, there is no labeled training data to supplement the large amount of unlabeled training data.

[0020] Therefore, the implementation of this article relates to a method and system for training an ASR model using aligned text and speech representations in the absence of transcribed speech training data. Specifically, training an ASR model includes receiving training data, which includes non-verbal text utterances in a target language, wherein each non-verbal text utterance is not paired with any corresponding utterance of non-synthesized (or synthesized) speech. The alignment model generates a corresponding alignment output (i.e., aligned text representation) for each non-verbal text utterance of the received training data. It is worth noting that the alignment model is trained on transcribed (i.e., labeled) speech utterances of one or more training languages ​​each of which is different from the target language that the ASR model is trained to recognize. In addition, a text encoder generates a corresponding encoded text representation for each alignment output. Thereafter, the ASR model is trained on the encoded text representation generated for the alignment output corresponding to the non-verbal text utterance in the target language to teach the ASR model to learn how to recognize speech in the target language when there is no labeled training data in the target language available. That is, the ASR model is trained without using any transcribed speech utterance in the target language for supervised learning.

[0021] Figure 1An ASR system 100 is shown implementing an ASR model 200 that resides on a user device 102 of a user 104 and / or a remote computing device 201 (e.g., one or more servers of a distributed system executing in a cloud computing environment) in communication with the user device 102. Although the user device 102 is depicted as a mobile computing device (e.g., a smartphone), the user device 102 may correspond to any type of computing device, such as, but not limited to, a tablet device, a laptop / desktop computer, a wearable device, a digital assistant device, a smart speaker / display, a smart home appliance, an automotive infotainment system, or an Internet of Things (IoT) device, and is equipped with data processing hardware and memory hardware 113.

[0022] The user device 102 includes an audio subsystem 108 configured to receive an utterance 106 spoken by a user 104 (e.g., the user device 102 may include one or more microphones for recording the spoken utterance 106) and convert the utterance 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by the ASR system 100. In the example shown, the user speaks a corresponding utterance 106 of the phrase "What is the weather in New York City?" in natural language English, and the audio subsystem 108 converts the utterance 106 into a corresponding acoustic frame 110 for input to the ASR system 100. Thereafter, the ASR model 200 receives the acoustic frame 110 corresponding to the utterance 106 as input and generates / predicts a corresponding transcription 120 (e.g., a recognition result / hypothesis) of the utterance 106 as output. In the example shown, the user device 102 and / or the remote computing device 201 also executes a user interface generator 107 that is configured to present a representation of a transcription 120 of the utterance 106 to the user 104 of the user device 102. In some configurations, the transcription 120 output from the ASR system 100 is processed by a natural language understanding (NLU) module, for example, executed on the user device 102 or the remote computing device 201, to execute the user command. Additionally or alternatively, a text-to-speech system (e.g., executed on any combination of the user device 102 or the remote computing device 201) can convert the transcription into synthesized speech for audible output by another device. For example, the original utterance 106 can correspond to a message that the user 104 is sending to a friend, where the transcription 120 is converted into synthesized speech for audible output to the friend to hear the message conveyed in the original utterance 106.

[0023] The ASR model 200 can operate in a streaming mode, a non-streaming mode, or some combination thereof. The ASR model 200 operates in a streaming mode by encoding a sequence of acoustic frames 110 while receiving a sequence of acoustic frames 110 and then decoding the encoded sequence of acoustic frames 110 into an initial transcription (e.g., speech recognition results / hypotheses) 120. Therefore, the initial transcription 120 can correspond to words, word fragments, and / or individual characters generated by the ASR model 200 once spoken. On the other hand, the ASR model 200 operates in a non-streaming mode by receiving and processing additional right-context to improve the initial transcription 120 and thereby generate a final transcription 120. That is, the ASR model 200 processes additional input audio data or encoded acoustic frames (e.g., right-context) to improve the transcription 120 output by the ASR model 200, but the delay increases.

[0024] refer to Figure 2 , the example ASR model 200 may include a recurrent neural network transducer (RNN-T) model architecture that complies with the latency constraints of interactive applications. The use of the RNN-T model architecture is exemplary only, as the ASR model 200 may include other architectures such as transformer-transducer and conformer-transducer model architectures. The RNN-T model 200 provides a small computational footprint and uses less memory requirements than conventional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device 102 (e.g., without the need to communicate with a remote server). The RNN-T model 200 includes an encoder network 210, a prediction network 220, and a joint network 230. The encoder network 210, which is roughly similar to the acoustic model (AM) in a traditional ASR system, includes a stack of self-attention layers (e.g., Conformer layers or Transformer layers) or a recurrent network of stacked long short-term memory (LSTM) layers. For example, the encoder network (eg, audio encoder) 210 reads a d-dimensional feature vector (eg, acoustic frame 110 ( Figure 1 ))Sequence x=(x1,x2,···,x T ),in And a higher-order feature representation is generated at each output step. This higher-order feature representation is represented as In some examples, encoder network 210 includes a dual encoder framework ( Figure 3B and Figure 3C ).

[0025] Similarly, the prediction network 220 is also an LSTM network, which, like a language model (LM), converts the non-empty symbol sequence y0, ..., y currently output by the final Softmax layer 240 into ui-1 Processing as dense representation The prediction network 220 and the joint network 230 may be collectively referred to as a decoder comprising an RNN-T architecture. Finally, using the RNN-T model architecture, the representations produced by the encoder network 210 and the prediction / decoder network 220 are combined by the joint network 230. The prediction network 220 may be replaced by an embedding lookup table to improve latency by outputting the sparse embeddings looked up instead of processing dense representations. The joint network then predicts It is the distribution about the next output symbol. In other words, the joint network 230 generates a probability distribution about possible speech recognition hypotheses at each output step (e.g., time step). Here, "possible speech recognition hypotheses" correspond to a set of output labels, each of which represents a symbol / character in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, for example, one label for each of the 26 letters in the English alphabet, and one label specifies a space. Therefore, the joint network 230 can output a set of values, which indicates the possibility of each of the predetermined set of output labels occurring. This set of values ​​can be a vector and can indicate a probability distribution about the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possible punctuation marks and other symbols), but the set of output labels is not limited to this. For example, as a supplement or alternative to graphemes, the set of output labels may include word pieces, phonemes and / or entire words. The output distribution of the joint network 230 may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output y of the joint network 230 is i 100 different probability values ​​may be included, one for each output label. The probability distribution may then be used (e.g., by a Softmax layer 240) to select candidate orthographic elements (e.g., graphemes, word fragments, and / or words) and assign scores thereto in a beam search process for use in determining the transcription 120.

[0026] The Softmax layer 240 may employ any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200 at the corresponding output step. In this manner, the RNN-T model 200 does not make conditional independence assumptions, but instead the prediction of each symbol is conditioned not only on the acoustics, but also on the sequence of labels output so far. The RNN-T model 200 does assume that the output symbols are independent of future acoustic frames 110, which allows the RNN-T model to be employed in a streaming manner, a non-streaming manner, or some combination thereof.

[0027] In some examples, the audio encoder 210 of the RNN-T model includes multiple multi-head (e.g., 8-head) self-attention layers. For example, the multiple multi-head self-attention layers may include a Conformer layer (e.g., a Conformer encoder), a Transformer layer, a performer layer, a convolutional layer (including a lightweight convolutional layer), or any other type of multi-head self-attention layer. The multiple multi-head self-attention layers may include any number of layers, such as 16 layers. In addition, the audio encoder 210 may operate in a streaming manner (e.g., once an initial higher-order feature representation is generated, the audio encoder 210 outputs the initial higher-order feature representation), a non-streaming manner (e.g., the audio encoder 210 outputs subsequent higher-order feature representations by processing additional right context to improve the initial higher-order feature representation), or a combination of a streaming manner and a non-streaming manner.

[0028] FIG. 3A to FIG. 3C The method for training the ASR model 200 ( Figure 2 ). The training process 300 described herein describes training the audio encoder 210 of the ASR model 200, however, it should be understood that the training process 300 may also include pre-training and / or fine-tuning training of the audio encoder 210. In addition, the implementation described herein considers that the training process 300 trains the audio encoder 210 of the ASR model 200 without training the decoder (e.g., the prediction network 220 and the joint network 230) of the ASR model 200. However, it should be understood that the training process 300 may additionally or alternatively train other components of the ASR model 200 (e.g., the prediction network 220 and / or the joint network 230) in conjunction with the audio encoder 210.

[0029] The training process 300 trains the audio encoder 210 using available training data, which includes non-verbal text utterances (X 文本 )320 collections of transcribed non-synthetic speech utterances (X sup )304 and / or untranscribed non-synthesized speech utterances (X unsup) 306. Notably, each non-verbal text utterance 320 in the set of non-verbal text utterances 320 includes pure text data (i.e., unpaired data) in the target language, such that each non-verbal text utterance 320 is not paired with any corresponding spoken audio representation (i.e., speech) of the utterance. Here, the target language is any language used by the training process 300 to train the audio encoder 210 to recognize locations where zero-transcribed (i.e., labeled) training data is used during training. The non-verbal text utterances 320 may include any sequence of text blocks, including words, word fragments, phonemes, and / or graphemes. Optionally, the available training data may also include untranscribed non-synthesized speech utterances 306 (also referred to as "untranscribed speech utterances 306"), each of which includes pure audio data (i.e., unpaired data) in the target language, such that the untranscribed speech utterance 306 is not paired with any corresponding transcription. It is worth noting that when the training data includes untranscribed speech utterances 306 in addition to non-verbal text utterances 306, the untranscribed speech utterances 306 represent training utterances that are different from the non-verbal text utterances 320, so that the training process 300 cannot simply pair the non-verbal text utterances 320 with the untranscribed speech utterances 306 to generate labeled training data.

[0030] On the other hand, each transcribed non-synthesized speech utterance 304 (also referred to simply as a "transcribed speech utterance 304") includes a corresponding transcription 302 paired with a corresponding non-synthesized speech representation of the corresponding transcribed speech utterance 304 in one or more training languages. Each of the one or more training languages ​​is different from the target language. For example, the one or more training languages ​​may include 52 languages ​​with transcribed speech utterances, and the target language may include another 50 languages ​​with plain text training data (e.g., each language is different from the 52 training languages). As will become apparent, the transcribed speech utterance 304 is used to train the alignment model 400 to generate an alignment output 402 for the transcribed speech utterance 304 in the one or more training languages. Thereafter, the trained alignment model 400 is configured to receive as input a non-verbal text utterance 320 in a target language (e.g., different from each of the one or more training languages ​​used to train the alignment model 400) and generate as output an alignment output 402 in the target language for training the audio encoder 210. Thus, training the alignment model 400 using transcribed speech utterances 304 in one or more training languages ​​enables the alignment model 400 to generate aligned output 402 in a target language, even though the alignment model 400 has never been trained with any training data in the target language.

[0031] For simplicity, the training process 300 includes a contrastive self-supervised loss part 300a ( Figure 3A), supervision loss part 300b( Figure 3B ) and the consistency regularization part 300c ( Figure 3C The training process 300 targets the total loss (L tts4pretrain2 ) Training the audio encoder 210: Using the contrastive self-supervised loss part 300a from the non-verbal training text utterance (X 文本 )320 (e.g., in the target language), transcribed non-synthesized speech utterances (X sup ) 304 (e.g., in one or more training languages) and a corpus of untranscribed non-synthesized speech utterances (X unsup ) 306 (e.g., in the target language) derived contrastive loss (L w2v )316; Using the supervised loss portion 300b from the non-verbal training text utterance (X 文本 )320 and transcribed non-synthesized speech utterances (X sup ) The supervised loss (L aux )342, 344; and the consistency loss derived using the consistency regularization portion 300c 352.

[0032] refer to Figure 3A , the contrastive self-supervised loss portion 300a of the training process 300 can employ an alignment model 400 configured to generate an aligned output (i.e., text representation) 402 for each of a plurality of non-verbal training text utterances 320 at each of a plurality of output steps. The non-verbal text utterances 320 include non-verbal text that is plain text data (i.e., unpaired data) in a target language, such that each non-verbal text utterance (X 文本 ) 320 are not paired with any synthesized or non-synthesized speech. Therefore, the alignment model 400 generates a corresponding alignment output 402 for each of the non-verbal text utterances 320.

[0033] Reference now Figure 4 In some examples, the alignment model 400 includes an embedding extractor 410, a duration predictor 420, and an upsampler 430. The embedding extractor 410 receives a non-verbal text utterance 320 including a sequence of text blocks (including words, word fragments, phonemes, and / or graphemes) and extracts a corresponding initial text representation (e t ) 412. The initial text representation 412 includes embedded vocabulary information from the non-verbal text utterance 320. Additionally or alternatively, the embedding extractor 410 may receive a transcription 302 corresponding to the transcribed non-synthesized speech utterance 304 ( Figure 3C). The duration predictor 420 receives the initial text representation 412 from the embedding extractor 410 and predicts the corresponding text block duration (i.e., word, word fragment, phoneme and / or grapheme duration) 422. The text block duration 422 indicates the duration that the corresponding text block will be spoken when a human (or a text-to-speech system) speaks the non-verbal text utterance 320. For example, the non-verbal text utterance 320 may include a phoneme sequence, and the duration predictor 420 predicts the phoneme duration 422 of each phoneme in the phoneme sequence. In this example, the duration predictor 420 predicts the phoneme duration 422 by predicting the probability of a non-zero duration for each phoneme and predicting the probability of a continuous phoneme duration for each phoneme. Since the phoneme sequence includes regular phones, silence between word boundaries, and punctuation marks, only regular phones are associated with non-zero durations, while silence and punctuation marks are typically associated with continuous phoneme durations. Therefore, the duration predictor 420 can use the sigmoid activation after the first of the two independent activations to predict the probability of non-zero duration, and use the soft plus activation after the second of the two independent projections to predict the continuous text block duration 422 for each text block. The duration predictor 420 determines whether the probability of non-zero duration is less than a threshold for each text block, and when the probability of non-zero duration is less than the threshold, the multiplier can zero the continuous text block duration 422 predicted by the soft plus activation for the corresponding text block. Otherwise, when the probability of non-zero duration is not less than the threshold, the predicted text block duration 422 can be set equal to the continuous phoneme duration predicted by the soft plus activation.

[0034] For each non-verbal text utterance 320, the upsampler 430 receives the corresponding initial text representation 412 and the predicted text block duration 422, and generates an aligned output having multiple frames by upsampling the initial text representation 412 using the corresponding predicted text block duration 422. 402. Here, the alignment output 402 represents the aligned speech-text representation. In some examples, the alignment model 400 sends the alignment output 402 to the text encoder 202 ( Figure 3B and Figure 3C In other examples (not shown), the alignment model 400 sends the alignment output 402 to the shared encoder 250 of the audio encoder 210 (e.g., bypassing the text encoder 202) ( Figure 3B and Figure 3C). In these other examples, the aligned output 402 is used as the encoded text representation 312, so that the shared encoder 250 can receive the aligned output 402 directly from the alignment model 400. In still other examples, paired training data is available, and the upsampler 430 generates the aligned output 402 as follows.

[0035]

[0036] Here, the upsampler 430 includes a resampler and refiner layer that aligns the initial text embedding 412 to directly align with the corresponding encoded audio representation 314 ( Figure 3B and 3C ) alignment. However, when paired training data is not available, upsampler 430 generates aligned output 402 as follows.

[0037]

[0038] Specifically, the frame number of the alignment output 402 indicates the predicted speech duration of the non-verbal text utterance 320. In other words, the frame number of the alignment output 402 maps (i.e., aligns) a sequence of text blocks of the non-verbal text utterance 320 to speech frames. Here, the upsampler 430 includes a resampler and refiner layer that copies the initial text embedding 412 to match the predicted text block duration 422 (i.e., speech duration). Therefore, the alignment output 402 includes a text representation of the non-verbal text utterance 320 having a timing component that is aligned with how a human would speak the non-verbal text utterance 320. In some examples, the embedding extractor 410 receives a language identifier 321 that uniquely identifies a language in one or more training languages ​​and / or a target language to adjust the alignment model 400.

[0039] In some implementations, training the alignment model 400 using training data in one or more training languages ​​and then generating an alignment output 402 in a target language (e.g., different from each of the training languages) produces a low-quality alignment output 402 because the script of the target language (e.g., Brahimic) does not overlap with the script of the one or more training languages. To this end, in some examples, the alignment model 400 includes a pronunciation model that converts the script of the non-verbal text utterance 320 in the target language into a representation (e.g., a phonetic representation) shared across multiple languages. In other examples, the alignment model 400 can transliterate the script of the non-verbal text utterance 320 into a different script. That is, the different script can be aligned with the script of the transcribed speech utterance 304.

[0040] It is noteworthy that in most instances, a text-to-speech (TTS) system generates an audible output to give the non-verbal text utterance 320 the timing components of human speech, so that the training process can use the audible output from the TTS system (i.e., synthesized speech) to train the audio encoder 210. However, the alignment model 400 advantageously generates an alignment output 402, thereby directly mapping a sequence of text blocks to speech frames. Therefore, the training process 300 does not require any TTS system to generate synthesized speech from the non-verbal text utterance 320 to train the audio encoder 210. That is, neither the training process 300 nor the alignment model 400 converts the non-verbal text utterance 320 into synthesized speech, but rather generates an alignment output 402 (i.e., text alignment).

[0041] Figure 5 An example training process 500 is shown for training the alignment model 400 using transcribed non-synthesized speech utterances 304 in one or more training languages. That is, each of the one or more training languages ​​is a language different from the target language that the audio encoder 210 ( FIG. 3 ) is trained to recognize. In addition, each transcribed non-synthesized speech utterance 304 has a corresponding transcription 302. In the example shown, the speech encoder 204 receives the speech utterances 304 as a sequence of features / vectors (e.g., a mel-spectrogram, such as Figure 1 110) as input, and for each of a plurality of output steps, generating an encoded audio representation (e) corresponding to the transcribed non-synthesized speech utterance 304 at the corresponding output step. s ) 314 as output. At the same time, the alignment model 400 receives the transcription 302 corresponding to the same non-synthesized speech utterance 304 and generates the aligned output 402 according to Equation 1. The text encoder 202 receives the aligned output 402 as input and generates, for each of the plurality of output steps, an encoded text representation corresponding to the transcription 302 at the corresponding output step 312 as output.

[0042] The modality loss module 550 receives the encoded text representation 312 and the encoded audio representation 314 and generates a modality loss 552 based on comparing the encoded text representation 3412 and the encoded audio representation as follows.

[0043]

[0044] Equation 3 represents the encoded text 312 and coded audio representation (e s )314 is added to the mean square error (MSE) between the predicted text target and the encoded audio representation (e s )314 RNN-T model alignment between them to determine the modality loss 552. Here, the encoded audio representation 314 is used as a true value label to train the alignment model 400 to generate an alignment output 402 that is aligned with the corresponding non-synthesized speech utterance 304. The training process 700 can use the modality loss 552 to update the parameters of the alignment model 400. For example, the training process 700 can update the duration predictor 420 and / or the upsampler 430 ( Figure 4 ) parameters.

[0045] Figure 6 An example training process 600 is shown for training an alignment model 400 using paired training data (i.e., transcribed speech utterances 304 in one or more training languages) and unpaired training data (i.e., non-verbal text utterances 320 in a target language). That is, the training process 600 uses transcribed non-synthesized speech utterances 304 with corresponding transcriptions 302 (i.e., paired training data) and non-verbal text utterances 320 (i.e., unpaired training data) to learn an alignment output 402 for speech alignment. In the example shown, the speech encoder 204 receives the speech utterances as a sequence of features / vectors (e.g., mel-spectrograms, such as Figure 1 110) as input, and for each of a plurality of output steps, generating an encoded audio representation (e) corresponding to the transcribed non-synthesized speech utterance 304 at the corresponding output step. s ) 314 as output. At the same time, the alignment model 400 receives the transcription 302 corresponding to the same non-synthesized speech utterance 304 and generates an aligned output according to Equation 1. Additionally or alternatively, the alignment model 400 may receive the non-verbal text utterance 320 and generate the aligned output 402 according to Equation 2. The text encoder 202 receives the aligned output 402 as input and generates an encoded text representation for each of the plurality of output steps 314 as output.

[0046] The audio encoder 210 may include a shared encoder 250 that receives an encoded text representation 312 as input and generates a first encoded shared representation 322 as output. The shared encoder 250 may also receive an encoded audio representation 314 as input and generate a second encoded shared representation 324 as output. An auxiliary decoder 390 receives the first encoded shared representation 322 and the second encoded shared representation 324 as input and generates a corresponding first probability distribution 392 and a second probability distribution 294 about possible speech recognition hypotheses as output.

[0047] The alignment masking loss module 650 receives the first probability distribution 392 corresponding to the encoded text representation 312 and the second probability distribution 394 corresponding to the encoded audio representation 314 and generates an alignment loss 652 as follows.

[0048]

[0049] The alignment loss 652 in Equation 4 can be applied to the masked, sampled encoded text representation 312 in the frequency domain and the time domain. It is worth noting that the alignment loss 652 can be used as a training target for both paired training data and unpaired training data. The training process 600 can use the alignment loss 652 to update the parameters of the alignment model 400. For example, the training process 800 can update the duration predictor 420 and / or the upsampler 430 ( Figure 4 ) parameters.

[0050] Return to reference Figure 3A In some implementations, the audio encoder 210 includes a reference Figure 3B and Figure 3C The speech encoder 204 and the text encoder 202 are described in more detail. In the example shown, the audio encoder 210 (alternatively, the speech encoder 204 or the text encoder 202 ( Figure 3B and Figure 3C )) comprises a Conformer encoder comprising a stack of Conformer blocks, each of which comprises a stack of multi-headed self-attention layers, depth-wise convolutional layers, and feed-forward layers. Alternatively, the audio encoder 210 may comprise another type of encoder, such as a transformer or performer encoder, having a stack of multi-headed self-attention layers / blocks. The Conformer encoder 210 can be naturally divided into a feature encoder (comprising a convolutional subsampling block 212) and a context network (comprising a stack of linear layers 214 and Conformer blocks 216). In some implementations, the convolutional subsampling block 212 has two two-dimensional convolutional layers, both of which have a stride of (2,2), resulting in a reduction in the feature sequence length by 1 / 4. The convolutional subsampling block 212 receives an input feature / vector sequence (e.g., a Mel-spectrogram, such as Figure 1 The convolutional subsampling block 212 may receive as input each alignment output 402 generated by the alignment model 400 from the non-verbal text utterance 320, and may generate as output an encoded text feature 213 corresponding to the corresponding one of the aligned outputs 402 for each of the plurality of output steps.

[0051] The encoded audio features 211 and encoded text features 213 (i.e., interchangeably referred to as "encoded features 211, 213") output from the convolutional subsampling block 212 can be fed to a masking module 218, in which some of the encoded features 211, 213 are randomly selected and replaced with trained feature vectors shared between all masked time steps to provide corresponding masked encoded audio features 211, 211m and masked encoded text features 213, 213m. In some examples, the masking module 218 masks the randomly selected encoded features 211, 213 to be masked by randomly sampling a certain proportion p of all time steps as a starting index without replacement and then masks the subsequent M consecutive time steps from each sample index, whereby some spans may overlap. After applying masking, the linear layer 214 and the Conformer block 216 of the context network receive the masked encoded features 211m, 213m (or the encoded features 211, 213 not selected by the masking module 218) and output the corresponding contrast context vector (i.e., encoded representation) 215 from the masked encoded features 211m, 213m. In addition, the quantizer 217 receives the encoded features 211, 213 as input and generates a quantized vector (i.e., target context vector) 219 as output. Thereafter, the contrast loss module 315 derives the contrast loss (L) between the contrast context vector 215 at the masked position and the target context vector 219 as follows w2v )316.

[0052]

[0053] where c t is the contrast context vector 215 centered at the masked output step (ie, time step t), and q t Including q t The target context vector 219 at output step t is selected from the set of K+1 candidate target context vectors 219 of the K distractors. The distractors may be uniformly sampled from other masked output steps of the same utterance.

[0054] The contrastive loss 316 is optimized between the contrastive context vector 215 at the masked position and the target context vector 219. After the audio encoder 210 converges on the untranscribed non-synthesized speech utterance 306, the training process is repeated for both the aligned output 402 corresponding to the non-verbal text utterance 320 and the transcribed non-synthesized speech utterance 304. Thus, the contrastive loss 316 (L w2v ) is optimized for both real / human (non-synthetic) and non-verbal text utterances 320 represented by the aligned output 402, where an additional auxiliary loss is derived from the transcribed non-synthetic speech utterances 304 and the aligned output 402, as described below with reference to Figure 3B, described in more detail. Thus, the contrastive self-supervised loss portion 300a of the training process 300 trains the audio encoder 210 using a contrastive loss 316 derived from the corresponding encoded features 211, 213 associated with each aligned output 402, each transcribed non-synthetic speech utterance 304, and each untranscribed non-synthetic speech utterance 306 provided as input to the audio encoder 210. Training the audio encoder 210 may include updating parameters of the audio encoder 210 based on the contrastive loss 316.

[0055] refer to Figure 3B , the supervised loss portion 300b of the training process 300 is configured to inject vocabulary information into the audio encoder 210 during training based on the supervised loss terms 342, 344 derived from the transcribed non-synthesized speech utterance and the alignment output 402 corresponding to the non-verbal text utterance 320 output by the alignment model 400. It is worth noting that the supervised loss portion 300b utilizes one or more auxiliary decoders 390 to generate the supervised loss terms 342, 344. The auxiliary decoders 390 may include a connection time classification (CTC) decoder, a listening attention spelling (LAS) decoder, or an RNN-T decoder. These auxiliary decoders 390 may include at least one of a phoneme decoder configured to decode a phoneme sequence or a word piece decoder configured to decode a word piece sequence. The auxiliary decoder 390 may also include a grapheme decoder configured to decode a grapheme sequence.

[0056] During the supervised loss portion 300b, the text encoder 202 of the audio encoder is configured to receive an alignment output 402 (i.e., a text embedding) from the alignment model, and the speech encoder is configured to receive a transcribed non-synthesized speech utterance 304. That is, the text encoder 202 of the audio encoder generates an encoded text representation 3123 for the alignment output 402 (e.g., corresponding to the non-verbal text utterance 320), and the speech encoder 204 of the audio encoder 210 generates an encoded audio representation 314 for the speech input (i.e., the transcribed non-synthesized speech utterance 304). Here, both the encoded text representation 312 and the encoded audio representation 314 may not be compatible with the auxiliary decoder 390. Therefore, the audio encoder 210 may also include a shared encoder 250 that receives the encoded text representation 312 as input and generates a first encoded shared representation 322 (e.g., corresponding to the non-verbal text utterance 320). 文本 ) as output. In addition, the shared encoder 250 receives the encoded audio representation 314 as input and generates a second encoded shared representation (e sup ) 324 as output. Therefore, the shared encoder 250 generates the first encoded shared representation 322 and the second encoded shared representation 324 into a shared latent representation space that is compatible with the auxiliary decoder 390.

[0057] Specifically, the shared encoder 250 receives as input each encoded text representation 312 corresponding to the aligned output 402 generated from the non-verbal text utterance 320, and generates, for each of the plurality of output steps, a first encoded shared representation (e ) corresponding to the aligned output 402 at the corresponding output step. 文本 ) 322 as output. An auxiliary decoder 390 including a phoneme decoder, a word piece decoder or a byte decoder receives each first encoded shared representation 322 output from the shared encoder 250 as input, and generates a first probability distribution 392 of possible speech recognition hypotheses for the corresponding alignment output 402 at the corresponding time step as output. In some examples, the first probability distribution 392 of possible speech recognition hypotheses includes one of possible phoneme tags, possible word piece tags or possible grapheme tags. Thereafter, the supervised loss module 340 can determine the alignment output loss term 342 based on the first probability distribution 392 of possible speech recognition hypotheses for the alignment output 402 corresponding to the non-verbal text utterance 320. Here, the corresponding non-verbal text utterance 320 from which the alignment output 402 is generated is also used as a true value transcription. The supervised loss portion 300b can train the audio encoder 210 for the alignment output loss term 342 by updating the parameters of the audio encoder 210 based on the alignment output loss term 342.

[0058] Similarly, during the supervised loss portion 300b, the shared encoder 250 receives as input each transcribed encoded audio representation 314 corresponding to the non-synthesized speech utterance 304 and, for each of a plurality of output steps, generates a second encoded shared representation (e ) corresponding to the transcribed non-synthesized speech utterance 304 at the corresponding output step. sup) 324 as output. An auxiliary decoder 390 including a phoneme decoder, a word piece decoder, or a byte decoder receives each second encoded shared representation 324 output from the shared encoder 250 as input and generates a second probability distribution 394 about possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance 304 at the corresponding output step as output. In some examples, the second probability distribution 394 about possible non-synthetic speech recognition hypotheses includes one of possible phoneme labels, possible word piece labels, or possible grapheme labels. Thereafter, the supervised loss module 340 can determine a non-synthetic speech loss term 344 based on the second probability distribution 394 about possible non-synthetic speech recognition hypotheses and the corresponding transcription 302 paired with the transcribed non-synthetic speech utterance 304. Here, the corresponding transcription 302 is used as a true value transcription and can include a sequence of target phonemes, target word pieces, and / or target graphemes. The supervised loss portion 300b can train the audio encoder 210 for the non-synthetic speech loss term 344 by updating the parameters of the audio encoder 210 based on the non-synthetic speech loss term 344.

[0059] In some implementations, the supervised loss portion 300b of the training process 300 uses another auxiliary decoder 390 to decode the first encoded shared representation (e) based on the aligned output 402 at the corresponding output step. 文本 ) 322 to generate a third probability distribution 393 about possible speech recognition hypotheses, whereby the supervisory loss module 340 can determine another alignment output loss term 342 based on the third probability distribution 393 and the non-verbal text utterance 320 corresponding to the alignment output 402. Here, the other auxiliary decoder 390 includes another of a phoneme decoder, a word piece decoder, or a grapheme decoder, and the third probability distribution 393 about possible speech recognition hypotheses includes another of possible phoneme labels, possible word piece labels, or possible grapheme labels. In these implementations, the other auxiliary decoder 390 also generates a fourth probability distribution 395 about possible non-synthetic speech recognition hypotheses for the corresponding second encoded shared representation 324 at the corresponding output step, whereby the supervisory loss module 340 can determine another non-synthetic speech loss term 344 based on the fourth probability distribution 395 and the corresponding transcription 302 paired with the transcribed non-synthetic speech representation 304. Here, the fourth probability distribution 395 about possible non-synthetic speech recognition hypotheses includes another of possible phoneme labels, possible word piece labels, or possible grapheme labels. The supervised loss portion 300 b of the training process 300 may similarly train the audio encoder 210 for another alignment output loss term 342 and another non-synthesized speech loss term 344 .

[0060] Untranscribed non-synthesized speech utterance 306 and non-verbal text utterance 320 each correspond to “unpaired” training data, whereby the non-verbal text utterance (X 文本 ) The contrast loss (L w2v ) 316 and the supervision loss associated with the alignment output loss term 342 Combining to obtain the non-verbal text loss function as follows.

[0061]

[0062] Likewise, the untranscribed non-synthetic speech utterances (X unsup ) The contrast loss (L w2v )316 can be used to express the unsupervised speech loss function as follows.

[0063]

[0064] During the training of the audio encoder 210, the aligned output 402 and the untranscribed non-synthesized speech 306 may be separated or mixed within each batch. In order to force the audio encoder 210 to learn a valid representation for both the aligned output 402 corresponding to the non-verbal text speech 320 and the non-synthesized (human / real) speech, in the loss function combining Equation 5 and Equation 6 Apply the loss mask σ to obtain the unpaired data loss function as follows:

[0065]

[0066] The transcribed non-synthesized speech utterances 304 correspond to “paired” and “supervised” training data, from which the derived contrastive loss L w2v and the derived supervision loss associated with the non-synthesized speech loss term 344 Combine to get the paired data loss function as follows:

[0067]

[0068] In some scenarios, after training the audio encoder 210, the ASR model recognizes audio from the target language during inference using graphemes from the training language. Thus, in some implementations, the supervised portion 300b of the training process 300 employs residual adapter layers 330 that adjust the audio encoder 210 or decoder (e.g., the prediction network 220 and the joint network 230 ( Figure 2)). Each residual adapter layer 330 includes a small feed-forward network including a self-attention layer with a bottleneck dimension (e.g., 2 self-attention layers). Thus, the residual identifier output 332 is fed to the shared encoder 250 so that the audio encoder 210 is conditioned according to the language identifier 321 and does not recognize audio from the target language during inference using graphemes from the training language.

[0069] refer to Figure 3C The consistency regularization part (i.e., the modality matching part) 300c of the training process 300 is configured to generate a consistency loss term between the training utterance pairs 301 352 to facilitate the audio encoder 210 to learn consistent predictions between the aligned outputs 402 corresponding to the non-synthesized speech (e.g., real / human speech) and the non-verbal text utterances 320, each training utterance pair comprising a transcribed non-synthesized speech utterance (X sup ) 304 and the paired aligned output 404 of the same utterance as the corresponding transcribed non-synthetic speech utterance 304. Therefore, the non-synthetic speech utterance 304 and the paired aligned output 404 in each training utterance pair 301 are associated with the same true value transcription. In short, the consistency loss term 352 between the transcribed non-synthetic speech utterance 304 and the paired aligned output 404 of the same training utterance provides an unsupervised training aspect by encouraging the audio encoder 210 to behave consistently regardless of whether the training utterance belongs to non-synthetic speech (i.e., speech training data) or the aligned output (i.e., text training data), and this unsupervised training aspect is independent of the supervised loss term between the true value transcription 302 and each of the following: the non-synthetic speech recognition hypothesis output by the auxiliary decoder 390; and the speech recognition hypothesis output by the auxiliary decoder 390.

[0070] Similar to Figure 3B In the example of the alignment output 402 generated from the non-verbal text utterance 320, the alignment model 400 can generate each paired alignment output 404 using the corresponding transcription 302 paired with the transcribed non-synthetic speech utterance 304. Here, the non-synthetic speech representation 304 is associated with the paired alignment output 404 generated by the alignment model 400 mapping the non-verbal text utterance 320 into the speech frame.

[0071] During the consistency regularization portion 300c, the text encoder 202 receives each paired alignment output 404 as input and generates, for each of a plurality of output steps, an encoded text representation 313 corresponding to the paired alignment output 404 at the corresponding output step as output. The shared encoder 250 receives the encoded text representation 313 as input and generates a first encoded shared representation (e * sup) 323 as output. The auxiliary decoder 390 including a phoneme decoder or a word fragment decoder receives each first encoded shared representation 323 output from the shared encoder 250 as input and generates a first probability distribution 311 about possible speech recognition hypotheses for the corresponding paired alignment output 404 at the corresponding output step as output. In some examples, the first probability distribution 311 about possible speech recognition hypotheses includes one of possible phoneme labels or possible word fragment labels.

[0072] Similarly, the speech encoder 204 receives the speech data as a sequence of features / vectors (e.g., a Mel-spectrogram such as Figure 1 The shared encoder 250 receives the encoded audio representation 314 as input and generates a second encoded shared representation (e ) corresponding to the transcribed non-synthesized speech utterance 304 at the corresponding output step. sup ) 324 as output. An auxiliary decoder 390 including a phoneme decoder or a word fragment decoder receives each second encoded shared representation 324 output from the shared encoder 250 as input and generates a second probability distribution 394 about possible non-synthetic speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance 304 at the corresponding time step as output. In some examples, the second probability distribution 394 about possible non-synthetic speech recognition hypotheses includes one of possible phoneme labels or possible word fragment labels.

[0073] Continue to refer Figure 3C The consistency regularization part 300c of the training process 300 further determines the consistency loss term of the corresponding training utterance pair 301 at each time step of the multiple time steps of each training utterance pair 301 based on the first probability distribution 311 about the possible speech recognition hypothesis and the second probability distribution 394 about the possible non-synthetic speech recognition hypothesis. 352. For example, the training process 300 can employ a consistency loss term module 350 that is configured to receive the corresponding non-synthesized speech and speech recognition results 311, 394 output by the auxiliary decoder 390 at each time step and determine a consistency loss term 352 for the corresponding training utterance pair 301 at the time step.

[0074] In some examples, the consistency regularization portion 300c of the training process 300 is based on the Kullback-Leibler divergence (D ) between the first probability distribution 311 about possible speech recognition hypotheses and the second probability distribution 394 about possible non-synthesized speech recognition hypotheses. KL ) to determine the consistent loss term 352. Based on DKL The consistent loss term 352 of can be expressed by the following equation:

[0075]

[0076] Here, the consistent loss term 352 determined for each training utterance pair 301 at each time step provides an accuracy independent of the auxiliary decoder 390 (e.g., independent of Figure 3B The "unsupervised" loss term 352 is a supervised loss term 342, 344 of the training speech, and therefore, can be used to update the parameters of the audio encoder 210 to promote consistency between the non-synthesized speech representation and the aligned output of the same speech. In batch training, the consistent loss term 352 can correspond to the average loss term obtained for the batch. In other words, the consistent loss term 352 allows the audio encoder 210 to learn to behave the same, for example, making consistent encoded representation predictions for both the non-synthesized speech (e.g., real / human speech) and the aligned output of the same training speech, regardless of whether the training speech belongs to the non-synthesized speech or the aligned output.

[0077] Finally, the training process 300 may combine the unpaired data loss function Paired Data Loss Function and the consistent loss term To obtain the overall loss term which can be expressed as

[0078]

[0079] Where λ1 may be equal to 1.0 and λ2 may be equal to 0.1. The training process 300 may use an overall loss term The audio encoder 210 is pre-trained by updating the parameters of the audio encoder 210 to effectively teach the audio encoder 210 to learn a shared representation between speech and text in a target language, even if no labeled training data in the target language is available. After training the audio encoder 210, the training process 300 can fine-tune the pre-trained audio encoder on transcribed speech utterances, which can include both aligned outputs corresponding to non-verbal text utterances 320 and supervised training samples of non-synthesized speech (e.g., human speech).

[0080] In some implementations, the training process 300 for training the audio encoder 210 applies encoder consistency regularization. Unlike the decoder consistency regularization applied to the auxiliary decoder during the consistency regularization portion 300c, which requires hypothesized labels (e.g., transcription 302 and non-verbal text utterances 320), encoder consistency regularization does not require hypothesized labels and therefore has the advantage of allowing application to all training data 304, 306, 320. Encoder consistency regularization can be applied via a hierarchical contrastive consistency regularization (HCCR) technique, where encoder activations e, e* from original / non-enhanced and enhanced speech are projected through the auxiliary network to generate z and z*. Thereafter, the positive and negative pairs are constructive, and the contrastive loss l t,z,z* The calculation is as follows:

[0081]

[0082] Specifically for HCCR, a convolutional neural network (CNN) projection network can compute projections on segments of increasing duration (30, 50, 120ms) of the encoder activation e to produce 3 views (V) and extract negative examples from the same utterance in short segments and from other utterances in the batch with 120ms segments. Thus, the HCCR loss can be calculated for the transcribed non-synthetic speech utterance 304 (paired speech), the untranscribed non-synthetic speech utterance 306 (unpaired speech), and the alignment output 402 generated from the non-verbal text utterance 320 as follows:

[0083]

[0084] The HCCR loss calculated by Equation 13 can be added to Equation 11 with a coefficient of 1e-3 as the overall loss term part of for pre-training the audio encoder 210.

[0085] The above implementation describes that the training process 300 trains the audio encoder 210 for the target language, however, it should be understood that the training process 300 can also be used to train the audio encoder for multiple target languages ​​that are different from one or more training languages. Therefore, the audio encoder 210 is used for the multilingual ASR model 200. In some instances, the training process 300 can be used to train an end-to-end ASR model (i.e., non-pre-training) with a decoder structure or fine-tune the ASR model to perform downstream tasks, such as speech conversion or natural language understanding. In addition, the implementation described above uses each part 300a-c of the training process 300 to describe the training process. However, it should be understood that any combination of the training parts 300a-c can be used to independently train the audio encoder 210 using any combination of non-verbal text utterances 320, transcribed non-synthetic speech utterances 304 and / or untranscribed non-synthetic speech utterances 306.

[0086] For example, transcribed non-synthesized speech utterances 304 in one or more training languages ​​may be initially used to train the alignment model 400 ( Figure 5 and Figure 6 ). Thereafter, using the alignment model 400 trained with labeled training data in one or more training languages, the training process 300 can train the audio encoder using the alignment output loss term 342 derived from the non-verbal text utterance 320 in the target language during the supervised loss portion 300b. Advantageously, the training process 300 can initially train the alignment model 400 with a resource-rich language (e.g., where a large amount of labeled training data already exists), and use the trained alignment model 400 to train the audio encoder 210 on a resource-poor (e.g., where little or zero labeled training data exists) target language. In short, by training the alignment model with one or more resource-rich languages ​​using a large amount of labeled training data, the alignment model 400 learns to generate an aligned output in the target language, even though the alignment model 400 has never been trained on any data in the target language (or zero labeled training data).

[0087] In some scenarios, after training the audio encoder 210, the ASR model recognizes audio from a target language during inference using graphemes from the training language. Thus, in some implementations, the consistency regularization portion 300c of the training process 300 employs residual adapter layers 330 that adjust the audio encoder 210 or decoder (e.g., the prediction network 220 and the joint network 230 ( Figure 2)). Each residual adapter layer 330 includes a small feed-forward network including a self-attention layer with a bottleneck dimension (e.g., 2 self-attention layers). Thus, the residual identifier output 332 is fed to the shared encoder 250 so that the audio encoder 210 is conditioned according to the language identifier 321 and does not recognize audio from the target language during inference using graphemes from the training language.

[0088] Figure 7 8 is a flowchart of an example arrangement of operations of a computer-implemented method 700 for training an automatic speech recognition model using aligned text and speech representations without transcribed speech data. The method 700 may use data stored in memory hardware 820 ( Figure 8 ) on the data processing hardware 810 ( Figure 8 ) is executed on the computing device 800 ( Figure 8 ) corresponding to Figure 1 on the user device 102 and / or the remote computer / server 201.

[0089] At operation 702, method 700 includes receiving training data including non-verbal text utterances 320 in a target language. Each non-verbal text utterance 320 is not paired with any corresponding spoken utterance of non-synthesized speech (or synthesized speech). At operation 704, method 700 includes using alignment model 400 to generate corresponding alignment output 402 for each non-verbal text utterance 320 of the received training data. Alignment model 400 is trained on speech utterances 304 transcribed in one or more training languages ​​that are different from the target language. That is, alignment model 400 is trained on training languages, and alignment output 402 is generated for non-verbal text utterances 320 in the target language that alignment model 400 has not seen during training. At operation 706, method 700 includes using text encoder 202 to generate corresponding encoded text representation 312 for each alignment output 402. At operation 708, the method 700 includes training the speech recognition model 200 on the encoded text representation 312 generated for the alignment output 402 corresponding to the non-verbal text utterance 320 in the target language to teach the speech recognition model 200 to learn how to recognize speech in the target language. Notably, the only transcribed (i.e., paired) training data used to train the speech recognition model 200 to learn how to recognize speech in the target language is the transcribed speech utterance 304 in one or more training languages, each of which is different from the target language, used to train the alignment model 400.

[0090] Figure 8800 is a schematic diagram of an example computing device 800 that can be used to implement the systems and methods described in this document. Computing device 800 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only, and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0091] The computing device 800 includes a processor 810, a memory 820, a storage device 830, a high-speed interface / controller 840 connected to the memory 820 and a high-speed expansion port 850, and a low-speed interface / controller 860 connected to a low-speed bus 870 and the storage device 830. Each of the components 810, 820, 830, 840, 850, and 860 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 810 can process instructions for execution within the computing device 800, including instructions stored in the memory 820 or on the storage device 830, to display graphical information for a graphical user interface (GUI) on an external input / output device (such as a display 880 coupled to the high-speed interface 840). In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. Moreover, multiple computing devices 800 can be connected, each of which provides a portion of the necessary operations (for example, as a server group, a blade server group, or a multi-processor system).

[0092] The memory 820 stores information non-temporarily within the computing device 800. The memory 820 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-temporary memory 820 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 800. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0093] The storage device 830 is capable of providing mass storage for the computing device 800. In some implementations, the storage device 830 is a computer-readable medium. In various implementations, the storage device 830 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as a memory 820, a storage device 830, or a memory on a processor 810.

[0094] The high-speed controller 840 manages bandwidth-intensive operations of the computing device 800, while the low-speed controller 860 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 840 is coupled to a memory 820, a display 880 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 850 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 860 is coupled to a storage device 830 and a low-speed expansion port 890. The low-speed expansion port 890, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, through a network adapter.

[0095] As shown, computing device 800 can be implemented in a variety of different forms. For example, computing device can be implemented as a standard server 800a or multiple implementations in a group of such servers 800a, implemented as a laptop computer 800b, or implemented as part of a rack server system 800c.

[0096] Various implementations of the systems and techniques described herein may be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that may be executed and / or interpreted on a programmable system that includes at least one programmable processor that may be special purpose or general purpose and that may be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0097] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0098] The processes and logic flows described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from the one or more mass storage devices or transfer data to the one or more mass storage devices or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0099] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0100] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the following claims.

Claims

1. A computer-implemented method (700) which, when executed on data processing hardware (810), causes the data processing hardware (810) to perform operations comprising: receiving training data comprising non-verbal text utterances (320) in a target language, each non-verbal text utterance (320) being unpaired with any corresponding spoken utterance of non-synthesized speech; generating a corresponding alignment output (402) for each non-verbal text utterance (320) of the received training data using an alignment model (400) trained on transcribed speech utterances (304) in one or more training languages ​​each different from the target language; Using the text encoder (202) to generate a corresponding encoded text representation (312) for each alignment output (402); and A speech recognition model (200) is trained on the encoded text representation (312) generated for the aligned output (402) corresponding to the non-verbal text utterance (320) in the target language to teach the speech recognition model (400) to learn how to recognize speech in the target language.

2. The computer-implemented method (700) of claim 1, wherein training the speech recognition model (200) comprises training the speech recognition model (200) without supervised learning using any transcribed speech utterances in the target language.

3. The computer-implemented method (700) of claim 1 or 2, wherein the speech recognition model (200) comprises an audio encoder (210) and a decoder (220, 230).

4. The computer-implemented method (700) of claim 3, wherein the decoder (220, 230) comprises a recurrent neural network transducer (RNN-T) architecture.

5. The computer-implemented method (700) of claim 3 or 4, wherein the audio encoder (210) comprises: The text encoder (202); Speech encoder (204); as well as Shared encoder (250).

6. The computer-implemented method (700) of any one of claims 3 to 5, wherein the audio encoder (210) comprises a plurality of multi-head self-attention layers.

7. The computer-implemented method (700) of any one of claims 3 to 6, wherein the audio encoder (210) comprises a Conformer encoder.

8. The computer-implemented method (700) of any one of claims 3 to 7, wherein the operations further comprise adjusting at least one of the audio encoder (210) or the decoder (220, 230) based on a language identifier (321) that uniquely identifies the target language.

9. The computer-implemented method (700) of claim 8, wherein adjusting the at least one of the audio encoder (210) or the decoder (220, 230) comprises using a residual adapter layer (330) to adjust the at least one of the audio encoder (210) or the decoder (220, 230) based on the language identifier (321).

10. The computer-implemented method (700) of any one of claims 1 to 9, wherein training the speech recognition model (200) comprises: For each aligned output (402), using a shared encoder (250) to generate a first encoded shared representation (322) of the aligned output (402) in a shared latent representation space; For each transcribed speech utterance (304) in the one or more training languages: determining an encoded audio representation (314) of the transcribed speech utterance (304) using a speech encoder (204); and generating, using the shared encoder (250), a second encoded shared representation (324) of the transcribed speech utterance (304) in the shared latent representation space, Wherein training the speech recognition model (200) includes training the speech recognition model (200) on the first encoded shared representation (322) generated for the aligned output (402) corresponding to the non-verbal text utterance (320) in the target language and the second encoded shared representation (324) generated for the transcribed speech utterance (304) in the one or more training languages.

11. The computer-implemented method (700) of any one of claims 1 to 10, wherein each non-verbal text utterance (320) comprises a sequence of words, word fragments, graphemes, and / or phonemes.

12. The computer-implemented method (700) of any one of claims 1 to 11, wherein the operations further comprise using a pronunciation model to convert a transcript of the non-verbal text utterance (320) in the target language into a phonetic representation shared across multiple languages.

13. A system (100), comprising: Data processing hardware (810); as well as Memory hardware (820) in communication with the data processing hardware (810), the memory hardware (820) storing instructions that, when executed on the data processing hardware (810), cause the data processing hardware (810) to perform operations, the operations comprising: receiving training data comprising non-verbal text utterances (320) in a target language, each non-verbal text utterance (320) being unpaired with any corresponding spoken utterance of non-synthesized speech; generating a corresponding alignment output (402) for each non-verbal text utterance (320) of the received training data using an alignment model (400) trained on transcribed speech utterances (304) in one or more training languages ​​each different from the target language; Using the text encoder (202) to generate a corresponding encoded text representation (312) for each alignment output (402); and A speech recognition model (200) is trained on the encoded text representation (312) generated for the aligned output (402) corresponding to the non-verbal text utterance (320) in the target language to teach the speech recognition model (400) to learn how to recognize speech in the target language.

14. The system (100) of claim 13, wherein training the speech recognition model (200) comprises training the speech recognition model (200) without supervised learning using any transcribed speech utterances in the target language.

15. The system (100) of claim 13 or 14, wherein the speech recognition model (200) comprises an audio encoder (210) and a decoder (220, 230).

16. The system (100) of claim 15, wherein the decoder (220, 230) comprises a recurrent neural network transducer (RNN-T) architecture.

17. The system (100) of claim 15 or 16, wherein the audio encoder (210) comprises: The text encoder (202); Speech encoder (204); as well as Shared encoder (250).

18. The system (100) of any one of claims 15 to 17, wherein the audio encoder (210) comprises a plurality of multi-head self-attention layers.

19. The system (100) of any one of claims 15 to 18, wherein the audio encoder (210) comprises a Conformer encoder.

20. The system (100) of any one of claims 15 to 19, wherein the operations further comprise adjusting at least one of the audio encoder (210) or the decoder (220, 230) based on a language identifier (321) that uniquely identifies the target language.

21. The system (100) of claim 20, wherein adjusting the at least one of the audio encoder (210) or the decoder (220, 230) comprises using a residual adapter layer (330) to adjust the at least one of the audio encoder (210) or the decoder (220, 230) based on the language identifier (321).

22. The system (100) of any one of claims 13 to 21, wherein training the speech recognition model (200) comprises: For each aligned output (402), using a shared encoder (250) to generate a first encoded shared representation (322) of the aligned output (402) in a shared latent representation space; For each transcribed speech utterance (304) in the one or more training languages: determining an encoded audio representation (314) of the transcribed speech utterance (304) using a speech encoder (204); and generating, using the shared encoder (250), a second encoded shared representation (324) of the transcribed speech utterance (304) in the shared latent representation space, Wherein training the speech recognition model (200) includes training the speech recognition model (200) on the first encoded shared representation (322) generated for the aligned output (402) corresponding to the non-verbal text utterance (320) in the target language and the second encoded shared representation (324) generated for the transcribed speech utterance (304) in the one or more training languages.

23. The system (100) of any one of claims 13 to 22, wherein each non-verbal text utterance (320) comprises a sequence of words, word fragments, graphemes and / or phonemes.

24. The system (100) of any one of claims 13 to 23, wherein the operations further comprise using a pronunciation model to convert a transcript of the non-verbal text utterance (320) in the target language into a phonetic representation shared across multiple languages.

Images