Joint voice and text streaming model for ASR

Abstract

A method (500) includes receiving training data including a set of non-spoken textual utterances (320). For each respective non-spoken text utterance, the method comprises: lexical meta-processing the respective text utterance into a sequence of sub-word units (402); generating a first higher-order textual feature representation for a corresponding sub-word unit lexical meta from the corresponding non-verbal text utterance (203); receiving the first higher-order textual feature representation generated by a textual encoder (202); and generating a first probability distribution of the possible text units (253). The method further includes training an encoder (210) based on the first probability distribution of possible text units generated by the first decoder (250) for each respective non-verbal text utterance in the set of non-verbal text utterances.

Description

Technical Field

[0001] The present disclosure relates to a joint speech and text streaming model for ASR. Background Art

[0002] Automatic speech recognition (ASR) (i.e., the process of taking audio input and transcribing it into text) has largely become an important technology used in mobile devices and other devices. In general, automatic speech recognition attempts to provide an accurate transcription of what people say by taking audio input (e.g., voice utterances) and transcribing the audio input into text. Based on the continuous development of deep neural networks, modern ASR models continue to improve in terms of accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay between the client speaking and the transcription). However, one challenge in developing ASR models based on deep learning is that the parameters of the ASR model tend to overfit the training data, which makes it difficult for the ASR model to generalize unknown data when the training data is not extensive enough. Therefore, training the ASR model for a larger training data set improves the accuracy of the ASR model. Synthetic speech and / or data-enhanced speech can be incorporated to increase the amount of training data used to train the ASR model. Summary of the invention

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations of a joint speech and text streaming model for automatic speech recognition. The operations include receiving training data including a set of non-verbal text utterances. Here, each corresponding non-verbal text utterance in the set of non-verbal text utterances is not paired with any corresponding spoken utterance of speech. For each corresponding non-verbal text utterance in the set of non-verbal text utterances, the operations include: tokenizing the corresponding non-verbal text utterance into a sequence of sub-word units; generating a first high-order text feature representation for a corresponding sub-word unit in the sequence of sub-word units tokenized from the corresponding non-verbal text utterance at each output step in a plurality of output steps by a text encoder of an encoder; receiving the first high-order text feature representation generated by the text encoder at each output step in a plurality of output steps as input to a first pass decoder; and generating a first probability distribution of possible text units by the first pass decoder at each output step in a plurality of output steps. The operations also include training the encoder based on a first probability distribution of possible text units generated by the first-pass decoder at each of a plurality of output steps for each respective non-verbal text utterance in the set of non-verbal text utterances.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, for each corresponding non-verbal text utterance in a set of non-verbal text utterances, the operation further includes: receiving a first high-order text feature representation generated by a text encoder at each output step in a plurality of output steps as an input to a non-causal audio text encoder of the encoder; generating a second high-order text feature representation by the non-causal audio text encoder for the corresponding first high-order text feature representation; receiving a second high-order text feature representation generated by a non-causal audio text encoder at each output step in a plurality of output steps as an input to a second pass decoder; and generating a second probability distribution of possible text units by the second decoder at each output step in a plurality of output steps. Here, the training encoder model is further based on a second probability distribution of possible text units generated by the second pass decoder at each output step in a plurality of output steps for each corresponding non-verbal text utterance in a set of non-verbal text utterances. In these implementations, the first pass decoder and the second pass decoder may include the same decoder. The non-causal audio-text encoder may include one of a plurality of unidirectional long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers.

[0005] In some examples, the training data further includes a set of transcribed speech utterances, wherein each transcribed speech utterance in the set of transcribed speech utterances is paired with a corresponding transcription and represented by a sequence of corresponding acoustic frames. Here, for each corresponding transcribed speech utterance in the set of transcribed speech utterances, the operation further includes: generating a first high-order audio feature representation for a corresponding acoustic frame in a sequence of acoustic frames representing the transcribed speech utterance at each of a plurality of output steps by a causal speech encoder of the encoder; receiving the first high-order audio feature representation generated by the causal speech encoder at each of a plurality of output steps as input to a first pass decoder; and generating a first probability distribution of possible speech recognition hypotheses at each of a plurality of output steps by the first pass decoder. In these examples, the training encoder is further based on the first probability distribution of possible speech recognition hypotheses generated by the first pass decoder at each of a plurality of output steps for each corresponding transcribed speech utterance in the set of transcribed speech utterances. The causal speech encoder may include one of a plurality of unidirectional long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers. Alternatively, the causal speech encoder may include an initial conformer layer stack and the non-causal audiotext encoder includes a final conformer layer stack overlaid on the initial conformer layer stack.

[0006] In some implementations, the causal speech encoder and the non-causal audio-text encoder of the encoder are trained using a hybrid autoregressive transformer decomposition. Here, for each corresponding transcribed speech utterance in the set of transcribed speech utterances, the operation may further include: receiving a first high-order audio feature representation generated by the causal speech encoder at each output step in a plurality of output steps as an input to the non-causal audio-text encoder; generating a second high-order audio feature representation by the non-causal audio-text encoder for the corresponding first high-order audio feature representation; receiving a second high-order audio feature representation generated by the non-causal audio-text encoder at each output step in a plurality of output steps as an input to the second pass decoder; and generating a second probability distribution of possible speech recognition hypotheses at each output step in a plurality of output steps by the second pass decoder. Here, training the encoder is further based on a second probability distribution of possible speech recognition hypotheses generated by the second pass decoder at each output step in a plurality of output steps for each corresponding transcribed speech utterance in the set of transcribed speech utterances. In these implementations, training the encoder model may include training the encoder using a minimum word error loss function.

[0007] Each sub-word unit in the sequence of sub-word units may include a phoneme or a word fragment, and each text unit in the first probability distribution of possible text units includes a word fragment. In some implementations, for each corresponding non-verbal text utterance in the set of non-verbal text utterances, the operations further include: upsampling a distribution of a sequence of sub-word units tokenized from the corresponding non-verbal text utterance using a parameter-free duration model; and randomly masking a portion of the upsampled distribution of the sequence of sub-word units.

[0008] Another aspect of the present disclosure provides a system comprising data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving training data comprising a set of non-verbal text utterances. Here, each corresponding non-verbal text utterance in the set of non-verbal text utterances is not paired with any corresponding spoken utterance of speech. For each corresponding non-verbal text utterance in the set of non-verbal text utterances, the operations include: tokenizing the corresponding non-verbal text utterance into a sequence of sub-word units; generating, by a text encoder of an encoder at each of a plurality of output steps, a first high-order text feature representation for a corresponding sub-word unit in a sequence of sub-word units tokenized from the corresponding non-verbal text utterance; receiving the first high-order text feature representation generated by the text encoder at each of a plurality of output steps as input to a first pass decoder; and generating, by the first pass decoder at each of a plurality of output steps, a first probability distribution of possible text units. The operations also include training the encoder based on a first probability distribution of possible text units generated by the first-pass decoder at each of a plurality of output steps for each respective non-verbal text utterance in the set of non-verbal text utterances.

[0009] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, for each corresponding non-verbal text utterance in a set of non-verbal text utterances, the operation further includes: receiving a first high-order text feature representation generated by a text encoder at each output step in a plurality of output steps as an input to a non-causal audio text encoder of the encoder; generating a second high-order text feature representation by the non-causal audio text encoder for the corresponding first high-order text feature representation; receiving a second high-order text feature representation generated by the non-causal audio text encoder at each output step in a plurality of output steps as an input to a second pass decoder; and generating a second probability distribution of possible text units by the second decoder at each output step in a plurality of output steps. Here, the training encoder model is further based on a second probability distribution of possible text units generated by the second pass decoder at each output step in a plurality of output steps for each corresponding non-verbal text utterance in a set of non-verbal text utterances. In these implementations, the first pass decoder and the second pass decoder may include the same decoder. The non-causal audio text encoder may include one of a plurality of unidirectional long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers.

[0010] In some examples, the training data further includes a set of transcribed speech utterances, wherein each transcribed speech utterance in the set of transcribed speech utterances is paired with a corresponding transcription and represented by a sequence of corresponding acoustic frames. Here, for each corresponding transcribed speech utterance in the set of transcribed speech utterances, the operation further includes: generating a first high-order audio feature representation for a corresponding acoustic frame in a sequence of acoustic frames representing the transcribed speech utterance at each of a plurality of output steps by a causal speech encoder of the encoder; receiving the first high-order audio feature representation generated by the causal speech encoder at each of a plurality of output steps as input to a first pass decoder; and generating a first probability distribution of possible speech recognition hypotheses at each of a plurality of output steps by the first pass decoder. In these examples, the training encoder is further based on the first probability distribution of possible speech recognition hypotheses generated by the first pass decoder at each of a plurality of output steps for each corresponding transcribed speech utterance in the set of transcribed speech utterances. The causal speech encoder may include one of a plurality of unidirectional long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers. Alternatively, the causal speech encoder may include an initial conformer layer stack and the non-causal audiotext encoder includes a final conformer layer stack overlaid on the initial conformer layer stack.

[0011] In some implementations, the causal speech encoder and the non-causal audio-text encoder of the encoder are trained using a hybrid autoregressive transformer decomposition. Here, for each corresponding transcribed speech utterance in the set of transcribed speech utterances, the operation may further include: receiving a first high-order audio feature representation generated by the causal speech encoder at each output step in a plurality of output steps as an input to the non-causal audio-text encoder; generating a second high-order audio feature representation by the non-causal audio-text encoder for the corresponding first high-order audio feature representation; receiving a second high-order audio feature representation generated by the non-causal audio-text encoder at each output step in a plurality of output steps as an input to the second pass decoder; and generating a second probability distribution of possible speech recognition hypotheses at each output step in a plurality of output steps by the second pass decoder. Here, training the encoder is further based on a second probability distribution of possible speech recognition hypotheses generated by the second pass decoder at each output step in a plurality of output steps for each corresponding transcribed speech utterance in the set of transcribed speech utterances. In these implementations, training the encoder model may include training the encoder using a minimum word error loss function.

[0012] Each sub-word unit in the sequence of sub-word units may include a phoneme or a word fragment, and each text unit in the first probability distribution of possible text units includes a word fragment. In some implementations, for each corresponding non-verbal text utterance in the set of non-verbal text utterances, the operations further include: upsampling a distribution of a sequence of sub-word units tokenized from the corresponding non-verbal text utterance using a parameter-free duration model; and randomly masking a portion of the upsampled distribution of the sequence of sub-word units.

[0013] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a schematic diagram of an example speech recognition system.

[0015] Figure 2 is a schematic diagram of an example speech recognition model.

[0016] Figure 3A and Figure 3B is a diagram of an example training process for an encoder for training a speech recognition model.

[0017] Figure 4 is a schematic diagram of an example alignment model.

[0018] Figure 5 is a flow diagram of an example arrangement of operations for a computer-implemented method of training an encoder of a speech recognition model to jointly learn a shared representation of speech and text.

[0019] Figure 6 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0020] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0021] Automatic speech recognition has made great progress by introducing the Sequence-to-Sequence (Seq2Seq) model that maps from audio sequences to character sequences. Meanwhile, Text-to-Speech (TTS) or speech synthesis systems have successfully applied Seq2Seq models to obtain state-of-the-art natural and realistic synthesized speech that is indistinguishable from human speech by the human ear.

[0022] One challenge in developing deep learning-based ASR models is that the parameters of the ASR model tend to overfit the training data, which results in difficulty for the ASR model to generalize unknown data when the training data is not extensive enough. Therefore, training the ASR model for a larger training data set improves the accuracy of the ASR model. For example, an ASR model can be trained for a training data set that includes more than 10,000 hours of transcribed speech using machine learning or other statistical methods. However, the performance of the ASR model is affected when the domain associated with the training data is different from the domain in which the ASR model will be deployed during inference. For example, an ASR model trained for transcribed speech in a domain associated with video conferencing is inefficient in recognizing speech associated with voice search queries, and vice versa.

[0023] Unpaired text data potentially significantly limits the amount of labeled human speech required to train an ASR model, while also providing flexibility for moving ASR models across different domains. However, using text data (i.e., unpaired text data) in addition to speech data to train an ASR model poses a challenge for combining the speech and text modalities of the training data. One current approach uses multi-task training to train a single model under different objectives for each modality. Since the nature and objectives of each modality of the training data are different, this approach is subject to interference and capacity limitations. Another current approach includes a TTS system that synthesizes unpaired text data to generate synthetic speech (i.e., modality conversion). However, despite the existence of examples where the most advanced synthetic speech is indistinguishable from human speech, using synthetic speech based on text data to train an ASR model still shows that it affects ASR training differently than human speech. This gap between synthetic speech and human speech is attributed to the mismatch between synthetic speech data and human speech data, and these mismatches are caused by the difficult one-to-many mapping problem that the TTS system attempts to solve. That is, while the overall quality of the available synthesized speech is very high, the synthesized speech exhibits much less variation than human speech and has minimal speech disfluencies. As a result, ASR models trained using synthesized speech based on unpaired text data have difficulty generalizing to real speech utterances during inference. Furthermore, the use of synthesized speech on unpaired text data also fails to account for ASR models that are trained to operate in a streaming manner, i.e., output transcriptions immediately upon receiving input audio data.

[0024] Therefore, the implementation of this article relates to a method and system for a joint speech and text streaming model for ASR. Specifically, a training process obtains training data, including a collection of non-verbal text utterances that are not paired with any corresponding spoken utterance of speech (e.g., human speech or synthetic speech). For each corresponding non-verbal text utterance, the training process tokenizes the corresponding non-verbal text utterance into a sequence of sub-word units (e.g., alignment output), generates a first high-order text feature representation for the corresponding sub-word unit using a text encoder of the encoder of the speech recognition model, and generates a first probability distribution of possible text units using a first pass decoder. The training process trains an encoder (e.g., an encoder of a speech recognition model) based on the first probability distribution of possible text units generated by the first pass decoder. It is obvious that the encoder of the speech recognition model operates in a streaming and non-streaming mode during the training process, so that the encoder is trained for the loss obtained during the streaming and non-streaming period. In addition, the training process can train the encoder using training data, which includes a collection of transcribed speech utterances that are each paired with a corresponding transcription. Therefore, using non-verbal text utterances and transcribed speech utterances, the training process is able to train the encoder of the speech recognition model using the shared latent representations of the speech and text modalities.

[0025] Figure 1 is an example of a speech environment 100. In the speech environment 100, the way in which a user 104 interacts with a computing device such as a user device 10 may be through voice input. The user device 10 (also generally referred to as a device 10) is configured to capture sound (e.g., streamed audio data) from one or more users 104 within the speech environment 100. Here, the streamed audio data may refer to the spoken utterance 106 of the user 104, which serves as an audible query, a command to the user device 10, or an audible communication captured by the device 10. The voice-enabled system of the user device 10 may process the query or command by answering the query and / or causing one or more downstream applications to execute / fulfill the command.

[0026] The user device 10 may correspond to any computing device associated with the user 104 and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, mobile devices (e.g., smart watches), smart home appliances, Internet of Things (IoT) devices, in-vehicle infotainment systems, smart displays, smart speakers, and the like. The user device 10 includes data processing hardware 12 and memory hardware 14 that communicates with the data processing hardware 12 and stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 having: an audio capture device (e.g., a microphone) 16, 16a for capturing spoken utterances 106 within the speech environment 100 and converting the spoken utterances into electrical signals; and a speech output device (e.g., a speaker) 16, 16b for transmitting audible audio signals (e.g., as output data from the user device 10). Although the user device 10 may implement an array of audio capture devices 16 a without departing from the scope of the present disclosure, one or more of the capture devices 16 a in the array may not physically reside on the user device 10 but may be in communication with the audio system 16 .

[0027] In the speech environment 100, an automatic speech recognition (ASR) system 118 implements an ASR model 200 and resides on a user device 10 of a user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system executed in a cloud computing environment) that communicates with the user device 10 via a network 40. In some examples, the ASR model 200 may be a recurrent neural network transducer (RNN-T) model. The user device 10 and / or the remote computing device 60 also includes an audio subsystem 108 that is configured to receive an utterance 106 spoken by the user 104 and captured by the audio capture device 16a, and convert the utterance 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by the ASR system 118. In the example shown, the user speaks a corresponding utterance 106, and the audio subsystem 108 converts the utterance 106 into corresponding audio data (e.g., a sequence of acoustic frames) 110 for input to the ASR system 118. Thereafter, the ASR model 200 receives as input a sequence of acoustic frames 110 corresponding to the utterance 106, and generates / predicts a corresponding transcription 120 (e.g., speech recognition results / hypotheses) of the utterance 106 at each output step as the ASR model receives (e.g., processes) each acoustic frame 110 in the sequence of acoustic frames 110.

[0028] In the example shown, the ASR model 200 can perform streaming speech recognition to produce initial speech recognition results 120, 120b, and generate final speech recognition results 120, 120a by improving the initial speech recognition results 120b. The speech recognition result 120 can correspond to a portion of the speech recognition result or the entire speech recognition result. In other words, the speech recognition result 120 can correspond to a portion of the speech 106 or the entire speech 106. For example, a partial speech recognition result can correspond to a portion of a spoken speech or even a portion of a spoken word. However, it will be apparent that the ASR model 200 performs additional processing on the final speech recognition result 120a, whereby the final speech recognition result 120a may be delayed compared to the initial speech recognition result 120b.

[0029] The user device 10 and / or the remote computing device 60 also executes a user interface generator 107 that is configured to present a representation of a transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 can display the initial speech recognition result 120b in a streaming manner during time 1, and then display the final speech recognition result 120a in a streaming manner during time 2. It is noteworthy that the ASR model 200 outputs the final speech recognition result 120a in a streaming manner, even if the final speech recognition result 120a improves on the initial speech recognition result 120b. In some configurations, the transcription 120 output from the ASR system 118 is processed, for example, by a natural language understanding (NLU) module executed on the user device 10 or the remote computing device 60 to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of user device 10 or remote computing device 60) may convert transcription 120 into synthesized speech for audible output by user device 10 and / or another device.

[0030] In the example shown, the user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) of the user device 10 using the ASR system 118. For example, Figure 1A user 104 is depicted communicating with a digital assistant application 50 and the digital assistant application 50 displays a digital assistant interface 18 on a screen of a user device 10 that depicts a conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, "What time is the concert tonight?" This question from the user 104 is a spoken utterance 106 that is captured by an audio capture device 16a and processed by an audio system 16 of the user device 10. In this example, the audio system 16 receives the spoken utterance 106 and converts the spoken utterance into a sequence of acoustic frames 110 for input to an ASR system 118.

[0031] Continuing with this example, the ASR model 200, upon receiving a sequence of acoustic frames 110 corresponding to an utterance 106 spoken by the user 104, encodes the sequence of acoustic frames 110 and then decodes the encoded sequence of acoustic frames 110 into an initial speech recognition result 120b. During time 1, the user interface generator 107 presents a representation of the initial speech recognition result 120b of the utterance 106 to the user 104 of the user device 10 in a streaming manner via the digital assistant interface 18, such that words, word fragments, and / or individual characters appear on the screen immediately as they are spoken. In some examples, the first look-ahead audio scene is equal to zero.

[0032] During time 2, the user interface generator 107 presents a representation of the final speech recognition result 120a of the utterance 106 to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming manner, so that the words, word fragments and / or individual characters appear on the screen immediately when generated by the ASR model 200. In some implementations, the user interface generator 107 replaces the representation of the initial speech recognition result 120b presented at time 1 with the representation of the final speech recognition result 120a presented at time 2. Here, time 1 and time 2 may include timestamps corresponding to the times when the user interface generator 107 presented the corresponding speech recognition results 120. In this example, the timestamp of time 1 indicates that the user interface generator 107 presented the initial speech recognition result 120b at an earlier time than the final speech recognition result 120a. For example, because the final speech recognition result 120a is considered more accurate than the initial speech recognition result 120b, the final speech recognition result 120a that is ultimately displayed as the transcription 120 can repair any terms that may have been misrecognized in the initial speech recognition result 120b. In this example, the streamed initial speech recognition result 120b output by the ASR model 200 is displayed on the screen of the user device 10 at time 1 and is associated with low latency, and provides the user 104 with a response that his query is being processed, while the final speech recognition result 120a output by the ASR model 200 and displayed on the screen at time 2 utilizes additional speech recognition models and / or language models to improve the quality of speech recognition in terms of accuracy, but increases latency. However, since the initial speech recognition result 120b is displayed while the user is speaking the utterance 106, the higher latency associated with generating and ultimately displaying the final speech recognition result 120a is not noticeable to the user 104.

[0033] exist Figure 1 In the example shown, the digital assistant application 50 can use natural language processing to respond to questions asked by the user 104. Natural language processing generally refers to the process of interpreting written language (e.g., initial speech recognition results 120b and / or final speech recognition results 120a) and determining whether the written language prompts any action. In this example, the digital assistant application 50 uses natural language processing to recognize that the question from the user 104 involves the user's schedule, and more specifically, the concert on the user's schedule. By using natural language processing to recognize these details, the automated assistant returns a response 19 to the user's query, where the response 19 states "Venue doors open at 6:30 PM and concert starts at 8pm". In some configurations, natural language processing occurs on a remote server 60 that communicates with the data processing hardware 12 of the user device 10.

[0034] refer to Figure 2 , the example ASR model 200 includes a recurrent neural network transducer (RNN-T) model architecture that complies with latency constraints associated with interactive applications. The use of the RNN-T model architecture is exemplary, and the ASR model 200 may include other architectures such as transformer-transducer and conformer-transducer model architectures. The RNN-T model 200 provides a small computational footprint and uses less memory requirements than conventional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device 102 (e.g., without the need to communicate with a remote server). The RNN-T model 200 includes an encoder network 210, a prediction network 220, and a joint network 230. The encoder network 210 (which is roughly similar to the acoustic model (AM) in a traditional ASR system) includes a stack of self-attention layers (e.g., Conformer or Transformer layers) or a recurrent network of stacked long short-term memory (LSTM) layers. For example, the encoder reads a sequence of d-dimensional feature vectors (e.g., acoustic frames 110 ( Figure 1 ))x = (x1,x2,···,x T ), where x t ∈ℝ d , and a higher-order feature representation is produced at each output step. The higher-order feature representation is represented as .

[0035] Similarly, the prediction network 220 is also an LSTM network, which, like a language model (LM), converts the non-empty symbol sequence y0, ..., y currently output by the final Softmax layer 240 into ui−1 Processing as dense representation Finally, using the RNN-T model architecture, the representations produced by the encoder network 210 and the prediction / decoder network 220 are combined by the joint network 230. The prediction network 220 can be replaced by an embedding lookup table to improve latency by outputting the sparse embeddings looked up instead of processing dense representations. The joint network then predicts , which is the distribution of the next output symbol. In other words, the joint network 230 generates a probability distribution of possible speech recognition hypotheses at each output step (e.g., time step). Here, "possible speech recognition hypotheses" correspond to a set of output labels, each of which represents a symbol / character in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, for example, one label for each of the 26 letters in the English alphabet, and one label specifies a space. Therefore, the joint network 230 can output a set of values ​​that indicates the probability of each of the predetermined set of output labels occurring. This set of values ​​can be a vector and can indicate a probability distribution of the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possible punctuation marks and other symbols), but the output label set is not limited to this. For example, as a supplement or alternative to graphemes, the set of output labels may include word fragments, phonemes and / or entire words. The output distribution of the joint network 230 may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output y of the joint network 230 is i 100 different probability values ​​may be included, one for each output label. The probability distribution may then be used (e.g., by a Softmax layer 240) to select candidate orthographic elements (e.g., graphemes, word fragments, and / or words) and assign scores thereto in a beam search process for use in determining the transcription 120.

[0036] The Softmax layer 240 may employ any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200 at the corresponding output step. In this manner, the RNN-T model 200 does not make conditional independence assumptions, but instead the prediction of each symbol is conditioned not only on the acoustics, but also on the sequence of labels output so far. The RNN-T model 200 does assume that the output symbol is independent of future acoustic frames 110, which allows the RNN-T model to be employed in a streaming manner.

[0037] In some examples, the encoder network (i.e., encoder) 210 of the RNN-T model 200 includes a stack of self-attention layers / blocks (such as conformer blocks). Here, each conformer block includes a series of multi-head self-attention layers, depth-wise convolutional layers, and feed-forward layers. The prediction network 220 can have two 2,048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. Alternatively, the prediction network 220 can include a stack of transformer or conformer blocks, or an embedding lookup table instead of an LSTM layer. Finally, the joint network 230 can also have 640 hidden units. The Softmax layer 240 can be composed of a unified set of word fragments or graphemes, which is generated using all unique word fragments or graphemes in multiple training data sets.

[0038] Figure 3A and Figure 3B An encoder 210 ( Figure 2 ). The example training process 300 may be used to pre-train, train, and / or fine-tune the encoder 210. The training process 300 trains the encoder 210 using available training data, which includes non-verbal text utterances (X text ) 320 collections and / or transcriptions of non-synthesized speech utterances (X sup ) 304. Each non-verbal text utterance 320 includes pure text data (i.e., unpaired data), such that each non-verbal text utterance 320 is not paired with any corresponding spoken audio representation (i.e., speech) of the utterance. The non-verbal text utterance 320 may include any sequence of text blocks, including words, word fragments, phonemes, and / or graphemes. Each transcribed non-synthesized speech utterance 304 (also referred to as "transcribed speech utterance 304") is paired with a corresponding transcription 302 and is represented by a corresponding acoustic frame 110 ( Figure 1 ) is represented by the corresponding sequence.

[0039] For simplicity, the training process 300 includes a semi-supervised loss part 300, 300a ( Figure 3A ) and supervision loss portion 300, 300b ( Figure 3B). It is apparent that the training process 300 trains the encoder 210 based on a total loss that includes unpaired loss terms 312, 314 derived by the semi-supervised loss portion 300a using the non-verbal text utterance 320 and paired loss terms 322, 324 derived by the supervised loss portion 300b using the transcribed speech utterance 304. It is apparent that both the semi-supervised loss portion 300a and the supervised loss portion 300b derive losses when the encoder 210 operates in a streaming mode (e.g., unpaired causal loss terms 312 and paired causal loss terms 322) and a non-streaming mode (e.g., unpaired non-causal loss terms 314 and paired non-causal loss terms 324). The training process 300 can jointly train the encoder 210 using any combination of unpaired loss terms 312, 314 and paired loss terms 322, 324.

[0040] Reference now Figure 3A , the semi-supervised loss portion 300a of the training process 300 can employ an alignment model 400 configured to generate an aligned output (i.e., text representation) 402 for each of the plurality of non-verbal training text utterances 320 at each of the plurality of output steps. The non-verbal text utterances 320 (also referred to as “non-verbal text utterances 320” for short) include non-verbal text that is pure text data (i.e., unpaired data), such that each non-verbal text utterance (X text ) 320 is not paired with any synthesized or non-synthesized speech. Therefore, the alignment model 400 generates a corresponding alignment output 402 for each non-verbal text utterance in the non-verbal text utterance 320.

[0041] Reference now Figure 4 In some examples, the alignment model 400 is configured to tokenize the non-verbal text utterance 320 and / or the transcription 302 into a sequence of sub-word units (e.g., alignment output) 402. The alignment model 400 may include an embedding extractor 410, a duration predictor 420, and an upsampler 430. The embedding extractor 410 receives the non-verbal text utterance 320 including a sequence of text blocks (including words, word fragments, phonemes, and / or graphemes) and extracts a corresponding initial text representation (e.g., an alignment output). t ) 412. The initial text representation 412 includes embedded vocabulary information from the non-verbal text utterance 320. Additionally or alternatively, the embedding extractor 410 may receive the transcribed speech utterance 304 ( Figure 3B) corresponding to the transcription 302. Duration predictor 420 receives initial text representation 412 from embedding extractor 410 and predicts corresponding text block duration (i.e., word, word fragment, phoneme and / or grapheme duration) 422. Text block duration 422 indicates the duration that the corresponding text block will be spoken when a human (or text-to-speech system) speaks the non-verbal text utterance 320 (or transcription 302). For example, the non-verbal text utterance 320 may include a phoneme sequence, and duration predictor 420 predicts a phoneme duration 422 for each phoneme in the phoneme sequence. In this example, duration predictor 420 predicts the phoneme duration 422 by predicting a probability of non-zero duration for each phoneme and predicting a probability of continuous phoneme duration for each phoneme. Since the phoneme sequence includes regular phonemes, silence between word boundaries, and punctuation marks, only regular phonemes are associated with non-zero durations, while silence and punctuation marks are generally associated with continuous phoneme durations. Therefore, the duration predictor 420 can use the sigmoid activation to predict the probability of non-zero duration after the first of the two independent activations, and use the soft plus activation to predict the continuous text block duration 422 of each text block after the second of the two independent projections. The duration predictor 420 determines whether the probability of non-zero duration is less than a threshold for each text block, and when the probability of non-zero duration is less than the threshold, the multiplier can zero the continuous text block duration 422 predicted by the soft plus activation for the corresponding text block. Otherwise, when the probability of non-zero duration is not less than the threshold, the predicted text block duration 422 can be set equal to the continuous phoneme duration predicted by the soft plus activation.

[0042] For each non-verbal text utterance 320, an upsampler 430 receives a corresponding initial text representation 412 and a predicted text block duration 422, and generates an aligned output having a plurality of frames by upsampling the initial text representation 412 using the corresponding predicted text block duration 422. ) 402. In some examples, the alignment model 400 sends the alignment output 402 to the encoder 210 ( Figure 3A ) of the text encoder 202. In other examples (not shown), the alignment model 400 sends the alignment output 402 directly to the encoder 210 ( Figure 3A ) of the non-causal audio-text encoder 206. In these other examples, the alignment output 402 is used as the first high-order text feature representation 203, so that the non-causal audio-text encoder 206 can directly receive the alignment output 402 from the alignment model 400. In some additional examples, paired training data is available, and the upsampler 430 generates the alignment output 402 as follows.

[0043]

[0044] Here, the upsampler includes a resampler and a refiner layer that aligns the initial text embedding 412 to align directly with the corresponding encoded audio representation. In yet other examples, paired training data is not available, and the upsampler 430 generates the aligned output 402 as follows.

[0045]

[0046] Specifically, the frame number of the aligned output 402 indicates the predicted speech duration of the non-verbal text utterance 320. In other words, the frame number of the aligned output 402 maps (i.e., aligns) the sequence of text blocks of the non-verbal text utterance 320 to speech frames. Here, the upsampler 430 includes a resampler and a refiner layer that replicate the initial text embedding 412 to match the predicted text block duration 422 (i.e., speech duration). Thus, the aligned output 402 includes a text representation of the non-verbal text utterance 320 having a timing component that is aligned with how a human speaks the non-verbal text utterance 320.

[0047] Advantageously, the alignment model 400 includes a parameter-free duration model, thereby greatly simplifying the training process 300 ( Figure 3A and Figure 3B). That is, the alignment model 400 is a parameter-free duration model that saves memory and computing resources, without parameters (e.g., neural network parameters) that are trained during training and consume memory and computing resources. In some implementations, the parameter-free alignment model 400 includes a fixed repetition model. Here, the fixed repetition model replicates each alignment output (e.g., subword unit) 402 a predetermined number of times. For example, each alignment output 402 may include a predetermined repetition length of three (3), which corresponds to 180 milliseconds of each alignment output 402. Therefore, in these examples, each alignment output 402 will have a speech duration corresponding to the predetermined repetition length, such as 180 milliseconds. In other implementations, the parameter-free alignment model 400 includes a random repetition model. That is, the random repetition model replicates each alignment output 402 a random number of times. That is, the alignment model 400 can sample a corresponding repetition length from the repetition length distribution of each corresponding alignment output 402, and apply the sampled repetition length to the corresponding alignment output 402. For example, the repetition length distribution may include one repetition length (e.g., 60 milliseconds), two repetition lengths (e.g., 120 milliseconds), or three repetition lengths (e.g., 180 milliseconds). Continuing with this example, the alignment model 400 may sample a repetition length of one for the first alignment output 402 so that the first alignment output 402 has a speech duration of 60 milliseconds, and sample a repetition length of three for the second alignment output 402 so that the second alignment output has a speech duration of 180 milliseconds. Using the repetition length distribution, the alignment model 400 is able to better simulate the variation of spoken utterances in the alignment output 402.

[0048] In some examples, the parameter-free alignment model 400 includes a subword distribution model. In the subword distribution model, the alignment model 400 determines the distribution of each subword unit (e.g., alignment output 402). That is, for each transcribed speech utterance 304, the alignment model 400 generates a forced alignment using a baseline alignment model to estimate the phoneme and word alignment of each word in the transcript from the transcribed speech utterance 304. The alignment is used to determine the statistics of the number of frames corresponding to each phoneme or word in the transcribed speech utterance 304. In this way, the alignment model 400 decomposes each word into its constituent word fragments and evenly distributes the total number of frames between its constituent word fragments. By accumulating the statistics of all transcribed speech utterances 304, the alignment model 400 determines a sufficient Gaussian distribution (e.g., including a mean and a standard deviation) for each subword unit. In this way, the duration of each subword unit can be derived by sampling from a corresponding Gaussian distribution or a Gaussian distribution that is sufficiently similar to the subword unit. Therefore, the subword distribution model samples from the Gaussian distribution model during inference to determine the alignment output 402. It is worth noting that since the subword distribution model is a parameter-free model, each unit is sampled independently and is independent of the scene effect.

[0049] In other examples, the parameter-free alignment model 400 includes an alignment subword distribution model. Here, the alignment model uses text from the transcribed speech utterance 304 to enhance the unpaired text data. In fact, this enhancement method treats the text from the transcribed speech utterance 304 as unpaired text. In addition, the alignment model upsamples the non-verbal text utterance 320 based on the real number of frames obtained using forced alignment. Specifically, the alignment model 400 divides the total number of frames of the word among the constituent word fragments. On the other hand, for unpaired text data (e.g., non-verbal text utterance 320), the alignment model 400 upsamples the text using a subword distribution model.

[0050] Thus, using any of the parameter-free duration models described above, the alignment model 400 upsamples the distribution of sequences of sub-word units tokenized from the corresponding non-verbal text utterance 320 (or transcribed speech utterance 304) and randomly masks a portion of the upsampled distribution of the sequence of sub-word units. Here, masking the upsampled distribution (e.g., setting portions of the sub-word units to null values) masks the alignment output 402, making the alignment output 402 sufficiently difficult to use in the training process 300. Thus, in any parameter-free duration model employed by the alignment model 400, not using parameters greatly simplifies the training process 300 ( Figure 3A and Figure 3B ).

[0051] It is worth noting that in most cases, the text-to-speech (TTS) system generates an audible output to give the non-verbal text utterances 320 a timing component of human speech, so that the training process can use the audible output (i.e., synthesized speech) to train the encoder 210. Therefore, because the alignment model 400 generates an alignment output 402 that directly maps a sequence of text blocks to speech frames, the training process 300 does not require any TTS system to train the encoder 210 using the non-verbal text utterances 320. That is, the alignment model 400 does not convert the non-verbal text utterances 320 to generate synthesized speech, which, as described above, may not accurately simulate human speech for use in training a speech recognition model.

[0052] Return to reference Figure 3A , the semi-supervised loss portion 300a of the training process 300 is configured to inject vocabulary information into the encoder 210 during training based on an unpaired causal loss term 312 and an unpaired non-causal loss term 314, each of which is derived from an alignment output 402 corresponding to a non-verbal text utterance 320 output by the alignment model 400. To this end, in some examples, the ASR model 200 ( Figure 2) includes a text encoder 202 and a non-causal audio-text encoder 206. Optionally, the text encoder 202 can be used only during the training process 300, and not during the reasoning of the ASR model 200. The text encoder 202 can be a causal text encoder that does not receive any additional right scenes (e.g., there is no additional frame of the alignment output 402). Specifically, the text encoder 202 is configured to receive the alignment output 402 (i.e., text embedding) from the alignment model 400, and generate a first high-order text feature representation 203 for the corresponding alignment output 402 (e.g., corresponding to the non-verbal text utterance 320) at each output step in a plurality of output steps. That is, the text encoder 202 operates in a streaming manner, so that at each output step, the text encoder 202 outputs the first high-order text feature representation 203 immediately after it is generated. Therefore, the first high-order text feature representation 203 can correspond to a part of the alignment output 402 or all of the alignment output 402.

[0053] The semi-supervised loss part 300a of the training process 300 uses the ASR model 200 ( Figure 2 ), the first pass decoder 250 being configured to receive as input the first high-order text feature representation 203 output from the text encoder 202 at each output step in a plurality of output steps, and to generate as output a first probability distribution 253 of possible text units for the corresponding first high-order text feature representation 203. Here, each text unit from the first probability distribution 253 may include a word fragment. In some implementations, the first pass decoder 250 includes an RNN-T architecture. The first pass decoder 250 may include a phoneme decoder configured to decode a phoneme sequence, a word fragment decoder configured to decode a word fragment sequence, and / or a grapheme decoder configured to decode a grapheme sequence. In some examples, the first probability distribution 253 of possible text units includes one of possible text labels, possible phoneme labels, possible word fragment labels, or possible grapheme labels. The unpaired loss module 310 is configured to determine an unpaired causal loss term 312 based on the first probability distribution 253 of possible text units and the corresponding non-verbal text utterance 320. The unpaired causal loss term 312 can be represented by Indicates that, y t represents the first probability distribution 253 of possible text units, and x t represents a non-verbal text utterance 320. Here, in determining the unpaired causal loss term 312 for the corresponding non-verbal text utterance 320, the corresponding non-verbal text utterance 320 in which the first probability distribution 253 of possible text units is generated serves as the true transcription.

[0054] Continue to refer Figure 3A , the encoder 210 includes a non-causal audio-text encoder 206, which is configured to generate a second high-order text feature representation 207 for the corresponding first high-order text feature representation 203. It is obvious that the non-causal audio-text encoder 206 generates high-order feature representations for text and audio encoding, so that the training process 300 uses a shared potential representation including speech and text modalities to train the encoder 210. The non-causal audio-text encoder 206 may include one of a plurality of unidirectional long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers. It is worth noting that the non-causal audio-text encoder 206 operates in a non-streaming manner, so that the non-causal audio-text encoder 206 processes the additional right scene to generate the second high-order text feature representation 207. That is, in contrast to the text encoder 202, the non-causal audio-text encoder 206 receives the additional right scene (e.g., the additional frame of the alignment output 402) and generates the second high-order text feature representation 207 by processing the additional right scene. In some examples, the non-causal audio-text encoder 206 generates a second high-order text feature representation 207 without receiving any alignment output 402 or audio data as input. In these examples, the non-causal audio-text encoder 206 only receives the first high-order text feature representation 203 generated by the text encoder 202 at each output step in multiple output steps, whereby the first high-order text feature representation 203 represents the additional right scene (e.g., an additional right scene frame of 900 ms). Therefore, by processing the first high-order text feature representation 203 corresponding to the additional right scene, the non-causal audio-text encoder 206 generates a second high-order text feature representation 207 with higher accuracy, but at the expense of increased latency.

[0055] The semi-supervised loss portion 300a of the training process 300 includes the ASR model 200 ( Figure 2), the second pass decoder 260 being configured to receive the second high-order text feature representation 207 output by the non-causal audio-text encoder 206 as input, and to generate a second probability distribution 263 of possible text units for the corresponding second high-order text feature representation 207 as output. Here, each text unit from the first probability distribution 253 may include a word fragment. In some examples, the first pass decoder 250 and the second pass decoder are the same decoder. In some implementations, the second pass decoder 260 includes an RNN-T architecture. The second pass decoder 260 may include a phoneme decoder configured to decode a phoneme sequence, a word fragment decoder configured to decode a word fragment sequence, and / or a grapheme decoder configured to decode a grapheme sequence. In some examples, the first probability distribution 253 of possible text units includes one of possible text labels, possible phoneme labels, possible word fragment labels, or possible grapheme labels. Therefore, the unpaired loss module 310 is further configured to determine an unpaired non-causal loss term 314 based on the second probability distribution 263 of possible text units and the corresponding non-verbal text utterance 320. The unpaired causal loss term 312 can be composed of Indicates that, y t represents the second probability distribution 263 of possible text units, and x t represents a non-verbal text utterance 320. Here, in order to determine the unpaired non-causal loss term 314 corresponding to the non-verbal text utterance 320, the corresponding non-verbal text utterance 320 in which the second probability distribution 263 of possible text units was generated serves as the true transcription.

[0056] Thus, the semi-supervised loss portion 300a of the training process 300 trains the ASR model 200 based on unpaired loss terms 312, 314 derived from the non-verbal text utterance 320 ( Figure 2 ) of the encoder 210. Training the encoder 210 may include updating parameters of the text encoder 202 and / or the non-causal audio-text encoder 206 based on the unpaired loss terms 312, 314. Notably, the unpaired causal loss terms 312 indicate the loss when the encoder 210 operates on the non-verbal text utterance 320 in a streaming manner, and the unpaired non-causal loss terms 314 indicate the loss when the encoder 210 operates on the non-verbal text utterance 320 in a non-streaming manner. In this way, the encoder 210 jointly trains the unpaired losses 312, 314 when the encoder 210 operates in a streaming mode and a non-streaming mode.

[0057] Reference now Figure 3B, the supervised loss portion 300b of the training process 300 is configured to inject vocabulary information into the encoder 210 during training based on the paired causal loss terms 322 and the paired non-causal loss terms 324, each of which is derived from the corresponding transcribed speech utterance 304. In some examples, the encoder 210 may be used in addition to or instead of the text encoder 202 ( Figure 3A ) includes a causal speech encoder 204 and a non-causal audio-text encoder 206. In some examples, the causal speech encoder 204 includes one of a plurality of unidirectional long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers. In these examples, the causal speech encoder 204 may include an initial conformer layer stack, and the non-causal audio-text encoder includes a final conformer layer stack covered on the initial conformer layer stack. The causal speech encoder 204 does not receive any additional right scene (e.g., there is no additional frame of the transcribed speech utterance 304). Specifically, the causal speech encoder 204 is configured to receive the transcribed speech utterance 304 and generate a first high-order audio feature representation 205 at each output step in a plurality of output steps. That is, the causal speech encoder 204 operates in a streaming manner so that at each output step, the causal speech encoder 204 outputs the first high-order audio feature representation 205 immediately after it is generated. Therefore, the first high-order audio feature representation 205 can correspond to a part of the transcribed speech utterance 304 or all of the transcribed speech utterance 304.

[0058] The supervised loss portion 300b of the training process 300 employs a first pass decoder 250 and a second pass decoder 260. The first pass decoder 250 is configured to receive as input the first high-order audio feature representation 205 output from the causal speech encoder 204 at each output step in a plurality of output steps, and to generate as output a first probability distribution 255 of possible speech recognition hypotheses. In some implementations, the first pass decoder 250 comprises an RNN-T architecture. The first pass decoder 250 may comprise a phoneme decoder configured to decode a phoneme sequence, a word fragment decoder configured to decode a word fragment sequence, and / or a grapheme decoder configured to decode a grapheme sequence. In some examples, the first probability distribution 255 of possible speech recognition hypotheses comprises one of possible phoneme labels, possible word fragment labels, or possible grapheme labels. Thereafter, the paired loss module 315 is configured to determine a paired causal loss term 322 based on the first probability distribution 255 of possible speech recognition hypotheses and the transcription 302 of the corresponding transcribed speech utterance 304. The paired causal loss term 322 can be represented by Indicates that, y srepresents a first probability distribution 255 of possible speech recognition hypotheses, and x s Represents the transcribed speech utterance 304. Here, in determining the paired causal loss term 322 for the corresponding transcribed speech utterance 304, the transcription 302 paired with the corresponding transcribed speech utterance 304, in which the first probability distribution 255 of possible speech recognition hypotheses is generated, serves as the true transcription.

[0059] Continue to refer Figure 3B , the encoder 210 includes a non-causal audio-text encoder 206, which is configured to generate a second high-order audio feature representation 208 for the corresponding first high-order audio feature representation 205. That is, in contrast to the causal speech encoder 204, the non-causal audio-text encoder 206 receives an additional right scene (e.g., an additional acoustic frame corresponding to the transcribed speech utterance 304) and generates a second high-order text feature representation 207 by processing the additional right scene. In some examples, the non-causal audio-text encoder 206 generates the second high-order audio feature representation 208 without receiving any additional transcribed speech utterance 304 or future acoustic frames. In these examples, the non-causal audio-text encoder 206 only receives the first high-order audio feature representation 205 generated by the causal speech encoder 204 at each output step in a plurality of output steps, whereby the first high-order audio feature representation 205 represents the additional right scene (e.g., an additional right scene frame of 900 ms). Therefore, by processing the first high-order audio feature representation 205 corresponding to the additional right scene, the non-causal audiotext encoder 206 generates the second high-order audio feature representation 208 with higher accuracy, but at the expense of increased latency.

[0060] The supervised loss portion 300b of the training process 300 includes the ASR model 200 ( Figure 2) is configured to receive as input a second high-order audio feature representation 208 output by the non-causal audio-text encoder 206, and to generate as output a second probability distribution 265 of possible speech recognition hypotheses corresponding to the second high-order audio feature representation 208. In some implementations, the second-pass decoder 260 includes an RNN-T architecture. The second-pass decoder 260 may include a phoneme decoder configured to decode a phoneme sequence, a word fragment decoder configured to decode a word fragment sequence, and / or a grapheme decoder configured to decode a grapheme sequence. In some examples, the second probability distribution 265 of possible speech recognition hypotheses includes one of possible phoneme labels, possible word fragment labels, or possible grapheme labels. Therefore, the paired loss module 310 is further configured to determine a paired non-causal loss term 324 based on the second probability distribution 265 of possible speech recognition hypotheses and the transcription 302 of the corresponding transcribed speech utterance 304. The paired non-causal loss term 324 may be composed of Indicates that, y s represents a second probability distribution 265 of possible speech recognition hypotheses, and x t Represents the transcribed speech utterance 304. Here, in determining the paired non-causal loss term 324 for the corresponding transcribed speech utterance 304, the transcription 302 for the corresponding transcribed speech utterance 304, in which the second probability distribution 265 of possible speech recognition hypotheses was generated, serves as the true transcription.

[0061] Thus, the supervised loss portion 300b of the training process 300 trains the ASR model 200 based on the paired loss terms 322, 324 derived from the transcribed speech utterance 304 ( Figure 2 ) encoder 210. Training the encoder 210 can include updating parameters of the causal speech encoder 204 and / or the non-causal audio-text encoder 206 based on the paired loss terms 322, 324. In some examples, the training process 300 uses a hybrid autoregressive transformer decomposition to train the causal speech encoder 204 and the non-causal audio-text encoder 206. Notably, the paired causal loss terms 322 indicate the loss when the encoder 210 operates on the transcribed speech utterance 304 in a streaming manner, and the paired non-causal loss terms 324 indicate the loss when the encoder 210 operates on the transcribed speech utterance 304 in a non-streaming manner. In this way, when the encoder 210 operates in a streaming mode and a non-streaming mode, the encoder 210 jointly trains the paired losses 322, 324.

[0062] Reference again Figure 3A and Figure 3BThe training process 300 trains the ASR model 200 by updating the parameters of the ASR model 200 based on the unpaired losses 312, 314 and the paired losses 322, 324 ( Figure 2 ). More specifically, the training process 300 can jointly train the encoder 210 of the ASR model 200 based on the unpaired losses 312, 314 and the paired losses 322, 324. Advantageously, the joint training method enables the training process 300 to train the encoder 210 based on the unpaired causal loss terms 312 and the paired causal loss terms 322 when the encoder 210 operates in a streaming mode, and to train the encoder 210 based on the unpaired non-causal loss terms 312 and the paired non-causal loss terms 324 when the encoder 210 operates in a non-streaming mode. Notably, when the training process 300 determines the unpaired causal loss terms 312 and the paired causal loss terms 322, the encoder 210 does not consider the additional right scene. On the other hand, when the training process 300 determines the unpaired non-causal loss terms 314 and the paired non-causal loss terms 324, the encoder 210 does process the additional right scene. In addition, the training process 300 uses the non-verbal text utterances 320 and the transcribed speech utterances 304 to train the encoder 210, thereby making the encoder 210 compatible with encoding in both text and speech modalities. In short, the training process 300 uses a shared latent space representation between text and speech to enable training in both text and speech modalities, and to enable training in both streaming and non-streaming modes. That is, the training process 300 can train the encoder 210 using an overall training loss according to the following formula:

[0063]

[0064] In equation 3, represents the overall training loss, represents the weight corresponding to the paired loss terms 322, 324, and represents the weight corresponding to the unpaired loss terms 312, 314. In addition, in Equation 3, It can be expressed as ,and Expressed as In some implementations, the overall loss is determined on a mini-batch of training samples, where each mini-batch includes half of the transcribed speech utterances 304 and half of the non-verbal text utterances 320 .

[0065] In some implementations, the training process 300 uses a minimum word error rate (MWER) loss to train the encoder 210, aiming to minimize the number of word errors. Specifically, given a speech utterance (x), a corresponding true transcription (y *) and a set of N best hypotheses (y i ), the MWER loss can be obtained according to the following formula:

[0066]

[0067] In equation 4, represents the difference between hypothesis (y) and true transcription (y * ). The training process 300 may incorporate the non-verbal text utterance 320 into the MWER loss according to the following formula:

[0068]

[0069] In equation 5, represents the MWER loss based on the output of the first pass decoder 250, represents the MWER loss based on the output of the second pass decoder 260, represents the interpolation weight, and represents the overall loss from Eq. 3.

[0070] The above implementation describes that the training process 300 trains the encoder 210 of the ASR model 200, however, it should be understood that the training process 300 can also be used to train / pre-train a single language ASR model 200 or a multi-language ASR model 200. In some instances, the training process 300 can be used to train an end-to-end ASR model (i.e., non-pre-trained) with a decoder structure or fine-tune the ASR model to perform downstream tasks such as speech translation or natural language understanding. In addition, the training process 300 can be used independently or using some combination of them with any training data source, including non-verbal text utterances 320 and transcribed speech utterances 304.

[0071] Figure 5 1 is a flow chart of an example operational arrangement of a method 500 for pre-training an encoder 210 to jointly learn a shared representation of speech and text. The method 500 may use the data stored in the memory hardware 620 ( Figure 6 ) on the data processing hardware 610 ( Figure 6 ) is executed on the . Data processing hardware 610 and memory hardware 620 may reside on Figure 1 and computing device 600( Figure 6 ) on the corresponding remote computer / server 60.

[0072] At operation 502, the method 500 includes receiving training data including a set of non-verbal text utterances 320. Here, each non-verbal text utterance 320 in the set of non-verbal text utterances is not paired with any corresponding spoken utterance of speech (e.g., synthetic or non-synthetic / human speech). For each corresponding non-verbal text utterance 320 in the set of non-verbal text utterances 320, the method 500 performs operations 504-510. At operation 504, the method 500 includes tokenizing the corresponding non-verbal text utterance into a sequence of sub-word units (e.g., alignment output) 402. At operation 506, the method 500 includes, at each of a plurality of output steps, generating a first high-order text feature representation 203 for a corresponding sub-word unit 402 in the sequence of sub-word units 402 tokenized from the corresponding non-verbal text utterance 320. Here, the text encoder 202 of the encoder 210 of the ASR model 200 generates the first high-order text feature representation 203. At operation 508, the method 500 includes receiving the first high-order text feature representation 203 generated by the text encoder 202 at each of the plurality of output steps as input to the first pass decoder 250. At operation 510, the method 500 includes generating, by the first pass decoder 250, a first probability distribution 253 of possible text units. At operation 512, the method 500 includes training the encoder 210 of the ASR model 200 based on the first probability distribution 253 of possible text units generated by the first pass decoder 250 at each of the plurality of output steps for each respective non-verbal text utterance 320 in the set of non-verbal text utterances 320.

[0073] Figure 6 600 that can be used to implement the systems and methods described in this document. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0074] The computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 connected to the memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 connected to a low-speed bus 670 and the storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 610 can process instructions for execution within the computing device 600, including instructions stored in the memory 620 or on the storage device 630, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 680 coupled to the high-speed interface 640). In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. In addition, multiple computing devices 600 can be connected, each of which provides a portion of the necessary operations (for example, as a server group, a blade server group, or a multi-processor system).

[0075] Memory 620 stores information non-temporarily within computing device 600. Memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporary memory 620 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0076] The storage device 630 can provide mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various implementations, the storage device 630 can be a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state memory device, or a device array (including a device in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0077] The high-speed controller 640 manages bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 640 is coupled to a memory 620, a display 680 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to a storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, through a network adapter.

[0078] The computing device 600 can be implemented in a variety of different forms, as shown. For example, it can be implemented as a standard server 600a or multiple times as a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0079] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be either special purpose or general purpose and can be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0080] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0081] The process and logic flow described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating output. The process and logic flow can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, a processor suitable for executing a computer program includes both a general-purpose microprocessor and a special-purpose microprocessor, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0082] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and possibly a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0083] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the following claims.

Claims

1. A computer-implemented method (500) which, when executed on data processing hardware (610), causes the data processing hardware (610) to perform operations comprising: receiving training data comprising a collection of non-verbal text utterances (320), each respective non-verbal text utterance (320) in the collection of non-verbal text utterances (320) being unpaired with any corresponding spoken utterance of speech; For each respective non-verbal text utterance (320) in said set of non-verbal text utterances (320): lemmatizing the corresponding non-verbal text utterance (320) into a sequence of sub-word units (402); generating, by a text encoder (202) of the encoder (210), at each of a plurality of output steps, a first high-order text feature representation (203) for a corresponding sub-word unit (402) in said sequence of sub-word units (402) tokenized from said corresponding non-verbal text utterance (320); Receiving the first high-order text feature representation (203) generated by the text encoder (202) at each of the plurality of output steps as input to a first pass decoder (250); as well as Generating, by the first pass decoder (250), a first probability distribution (253) of possible text units at each of the plurality of output steps; and The encoder (210) is trained based on the first probability distribution (253) of possible text units generated by the first pass decoder (250) at each of the plurality of output steps for each respective non-verbal text utterance (320) in the set of non-verbal text utterances (320).

2. The computer-implemented method (500) of claim 1, wherein for each respective non-verbal text utterance (320) in the set of non-verbal text utterances (320), the operations further comprise: receiving the first high-order text feature representation (203) generated by the text encoder (202) at each of the plurality of output steps as input to a non-causal audio-text encoder (206) of the encoder (210); Generating, by the non-causal audio-text encoder (206), a second high-order text feature representation (207) for the corresponding first high-order text feature representation (203) at each output step in the plurality of output steps; receiving the second high-order text feature representation (207) generated by the non-causal audio-text encoder (206) at each of the plurality of output steps as input to a second pass decoder (260); as well as generating, by the second decoder (260), a second probability distribution (263) of possible text units at each output step in the plurality of output steps, wherein training the encoder (210) is further based on the second probability distribution (263) of possible text units generated by the second decoder (260) at each of the plurality of output steps for each corresponding non-verbal text utterance (320) in the set of non-verbal text utterances (320).

3. The computer-implemented method (500) of claim 2, wherein the first pass decoder (250) and the second pass decoder (260) comprise the same decoder.

4. The computer-implemented method (500) of claim 2 or 3, wherein the non-causal audio-text encoder (206) comprises one of: Multiple unidirectional long short-term memory (LSTM) layers; multiple conformer layers; or Multiple transformer layers.

5. The computer-implemented method (500) of any one of claims 2 to 4, wherein: The training data further comprises a set of transcribed speech utterances (304), each transcribed speech utterance (304) in the set of transcribed speech utterances (304) being paired with a corresponding transcription (302) and represented by a sequence of corresponding acoustic frames (110); and For each respective transcribed speech utterance (304) in the set of transcribed speech utterances (304), the operations further include: generating, by a causal speech encoder (204) of the encoder (210), at each of the plurality of output steps, a first high-order audio feature representation (205) for a corresponding acoustic frame (110) in the sequence of acoustic frames (110) representing the transcribed speech utterance (304); receiving the first high-order audio feature representation (205) generated by the causal speech encoder (204) at each of the plurality of output steps as input to the first pass decoder (250); as well as generating, by the first pass decoder (250), at each of the plurality of output steps, a first probability distribution (255) of possible speech recognition hypotheses, Wherein training the encoder (210) is further based on the first probability distribution (255) of possible speech recognition hypotheses generated by the first decoder (250) at each of the plurality of output steps for each corresponding transcribed speech utterance (304) in the set of transcribed speech utterances (304).

6. The computer-implemented method (500) of claim 5, wherein the causal speech encoder (204) comprises one of: Multiple unidirectional long short-term memory (LSTM) layers; multiple conformer layers; or Multiple transformer layers.

7. The computer-implemented method (500) of claim 5 or 6, wherein: The causal speech encoder (204) comprises an initial conformer layer stack; and The non-causal audiotext encoder (206) includes a final conformer layer stack overlying the initial conformer layer stack.

8. A computer-implemented method (500) as described in any one of claims 5 to 7, wherein the causal speech encoder (204) and the non-causal audio-text encoder (206) of the encoder (210) are trained using hybrid autoregressive transformer decomposition.

9. The computer-implemented method (500) of claim 8, wherein for each respective transcribed speech utterance (304) in the set of transcribed speech utterances (304), the operations further comprise: receiving the first high-order audio feature representation (205) generated by the causal speech encoder (204) at each of the plurality of output steps as input to the non-causal audio-text encoder (206); Generating, by the non-causal audiotext encoder (206), a second high-order audio feature representation (208) for the corresponding first high-order audio feature representation (205) at each of the plurality of output steps; receiving the second high-order audio feature representation (208) generated by the non-causal audiotext encoder (206) at each of the plurality of output steps as input to the second pass decoder (260); as well as generating, by the second pass decoder (260), a second probability distribution (265) of possible speech recognition hypotheses at each of the plurality of output steps, Wherein training the encoder (210) is further based on the second probability distribution (265) of possible speech recognition hypotheses generated by the second decoder (260) at each of the plurality of output steps for each corresponding transcribed speech utterance (304) in the set of transcribed speech utterances (304).

10. The computer-implemented method (500) of claim 9, wherein training the encoder (210) comprises training the encoder (210) using a minimum word error loss function.

11. The computer-implemented method (500) of any one of claims 1 to 10, wherein: Each sub-word unit (402) in the sequence of sub-word units (402) comprises one of a phoneme or a word fragment; and Each text unit in said first probability distribution (203) of possible text units comprises a word fragment.

12. The computer-implemented method (500) of any one of claims 1 to 11, wherein for each respective non-verbal text utterance (320) in the set of non-verbal text utterances (320), the operations further comprise: upsampling the distribution of the sequence of subword units (402) tokenized from the corresponding non-verbal text utterance (320) using a parameter-free duration model; as well as A portion of the upsampled distribution of the sequence of sub-word units (402) is randomly masked.

13. A system (118), comprising: Data processing hardware (610); as well as Memory hardware (620) in communication with the data processing hardware (610), the memory hardware (620) storing instructions that, when executed on the data processing hardware (610), cause the data processing hardware (610) to perform operations, the operations comprising: receiving training data comprising a collection of non-verbal text utterances (320), each respective non-verbal text utterance (320) in the collection of non-verbal text utterances (320) being unpaired with any corresponding spoken utterance of speech; For each respective non-verbal text utterance (320) in said set of non-verbal text utterances (320): lemmatizing the corresponding non-verbal text utterance (320) into a sequence of sub-word units (402); generating, by a text encoder (202) of the encoder (210), at each of a plurality of output steps, a first high-order text feature representation (203) for a corresponding sub-word unit (402) in said sequence of sub-word units (402) tokenized from said corresponding non-verbal text utterance (320); Receiving the first high-order text feature representation (203) generated by the text encoder (202) at each of the plurality of output steps as input to a first pass decoder (250); as well as Generating, by the first pass decoder (250), a first probability distribution (253) of possible text units at each of the plurality of output steps; and The encoder (210) is trained based on the first probability distribution (253) of possible text units generated by the first pass decoder (250) at each of the plurality of output steps for each respective non-verbal text utterance (320) in the set of non-verbal text utterances (320).

14. The system (118) of claim 13, wherein for each respective non-verbal text utterance (320) in the set of non-verbal text utterances (320), the operations further comprise: receiving the first high-order text feature representation (203) generated by the text encoder (202) at each of the plurality of output steps as input to a non-causal audio-text encoder (206) of the encoder (210); Generating, by the non-causal audio-text encoder (206), a second high-order text feature representation (207) for the corresponding first high-order text feature representation (203) at each output step in the plurality of output steps; receiving the second high-order text feature representation (207) generated by the non-causal audio-text encoder (206) at each of the plurality of output steps as input to a second pass decoder (260); as well as generating, by the second decoder (260), a second probability distribution (263) of possible text units at each output step in the plurality of output steps, wherein training the encoder (210) is further based on the second probability distribution (263) of possible text units generated by the second decoder (260) at each of the plurality of output steps for each corresponding non-verbal text utterance (320) in the set of non-verbal text utterances (320).

15. The system (118) of claim 14, wherein the first pass decoder (250) and the second pass decoder (260) comprise the same decoder.

16. The system (118) of claim 14 or 15, wherein the non-causal audiotext encoder (206) comprises one of: Multiple unidirectional long short-term memory (LSTM) layers; multiple conformer layers; or Multiple transformer layers.

17. The system (118) of any one of claims 14 to 16, wherein: The training data further comprises a set of transcribed speech utterances (304), each transcribed speech utterance (304) in the set of transcribed speech utterances (304) being paired with a corresponding transcription (302) and represented by a sequence of corresponding acoustic frames (110); and For each respective transcribed speech utterance (304) in the set of transcribed speech utterances (304), the operations further include: generating, by a causal speech encoder (204) of the encoder (210), at each of the plurality of output steps, a first high-order audio feature representation (205) for a corresponding acoustic frame (110) in the sequence of acoustic frames (110) representing the transcribed speech utterance (304); receiving the first high-order audio feature representation (205) generated by the causal speech encoder (204) at each of the plurality of output steps as input to the first pass decoder (250); as well as generating, by the first pass decoder (250), at each of the plurality of output steps, a first probability distribution (255) of possible speech recognition hypotheses, Wherein training the encoder (210) is further based on the first probability distribution (255) of possible speech recognition hypotheses generated by the first decoder (250) at each of the plurality of output steps for each corresponding transcribed speech utterance (304) in the set of transcribed speech utterances (304).

18. The system (118) of claim 17, wherein the causal speech encoder (204) comprises one of: Multiple unidirectional long short-term memory (LSTM) layers; multiple conformer layers; or Multiple transformer layers.

19. The system (118) of claim 17 or 18, wherein: The causal speech encoder (204) comprises an initial conformer layer stack; and The non-causal audiotext encoder (206) includes a final conformer layer stack overlying the initial conformer layer stack.

20. The system (118) of any one of claims 17 to 19, wherein the causal speech encoder (204) and the non-causal audio-text encoder (206) of the encoder (210) are trained using hybrid autoregressive transformer decomposition.

21. The system (118) of claim 20, wherein for each respective transcribed speech utterance (304) in the set of transcribed speech utterances (304), the operations further comprise: receiving the first high-order audio feature representation (205) generated by the causal speech encoder (204) at each of the plurality of output steps as input to the non-causal audio-text encoder (206); Generating, by the non-causal audiotext encoder (206), a second high-order audio feature representation (208) for the corresponding first high-order audio feature representation (205) at each of the plurality of output steps; receiving the second high-order audio feature representation (208) generated by the non-causal audiotext encoder (206) at each of the plurality of output steps as input to the second pass decoder (260); as well as generating, by the second pass decoder (260), a second probability distribution (265) of possible speech recognition hypotheses at each of the plurality of output steps, Wherein training the encoder (210) is further based on the second probability distribution (265) of possible speech recognition hypotheses generated by the second decoder (260) at each of the plurality of output steps for each corresponding transcribed speech utterance (304) in the set of transcribed speech utterances (304).

22. The system (118) of claim 21, wherein training the encoder (210) comprises training the encoder (210) using a minimum word error loss function.

23. The system (118) of any one of claims 13 to 22, wherein: Each sub-word unit (402) in the sequence of sub-word units (402) comprises one of a phoneme or a word fragment; and Each text unit in said first probability distribution (203) of possible text units comprises a word fragment.

24. The system (118) of any one of claims 13 to 23, wherein for each respective non-verbal text utterance (320) in the set of non-verbal text utterances (320), the operations further comprise: upsampling the distribution of the sequence of subword units (402) tokenized from the corresponding non-verbal text utterance (320) using a parameter-free duration model; as well as A portion of the upsampled distribution of the sequence of sub-word units (402) is randomly masked.

Images