Divelation via text-only and semi-supervised training

Abstract

A method (500) of text-only and semi-supervised training for diliberation includes receiving training data (320) including a plurality of non-spoken text utterances (330), each unpaired with any corresponding spoken utterance of non-synthetic speech, and training a diliberation model (240) including a text encoder (242) and a diliberation decoder (250) on the non-spoken text utterances. The method also includes receiving, in the trained diliberation model, first pass hypotheses (232) and acausal acoustic embeddings (222). The first pass hypotheses are generated by a recurrent neural network transducer (RNN-T) decoder (230) for the acausal acoustic embeddings encoded by the acausal encoder. The method also includes encoding the first pass hypotheses generated by the RNN-T decoder using a text encoder, and generating a second pass hypothesis (248) using a divelation decoder that attends to both the first pass hypotheses and the non-causal acoustic embeddings.

Description

[Technical field]

[0001] This disclosure relates to divestiture via text-only and semi-supervised training. [Background technology]

[0002] Modern automatic speech recognition (ASR) systems are focused on providing not only good quality (e.g., low word error rate (WER)) but also low latency (e.g., short delay between when a user speaks and when a transcription appears). Furthermore, when using ASR systems today, there is a demand that the ASR system decodes speech in a streaming format that responds in real time or even faster than real time. To explain, when an ASR system is deployed on a mobile phone that undergoes direct user interactivity, the mobile phone application using the ASR system may require speech recognition such that words are streamed to be displayed on the screen as soon as they are spoken. Here, mobile phone users may also have a low tolerance for latency. This low tolerance leads to speech recognition attempting to run on the mobile device in a manner that minimizes the impact of latency and inaccuracies that may adversely affect the user's experience. Summary of the Invention

[0003] One aspect of the disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations, including receiving training data including a plurality of non-spoken text utterances, each unpaired with any corresponding spoken utterance of non-synthetic speech, and training a diliberation model including a text encoder and a diliberation decoder for the plurality of non-spoken text utterances. The operations also include receiving, in the trained diliberation model, a first pass hypothesis and a non-causal acoustic embedding. The first pass hypothesis is generated by a recurrent neural network transducer (RNN-T) decoder for the non-causal acoustic embedding encoded by the non-causal encoder. The operations also include encoding the first pass hypothesis generated by the RNN-T decoder using the text encoder, and generating a second pass hypothesis using a diliberation decoder that attends to both the first pass hypothesis and the non-causal acoustic embedding.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operations further include receiving a sequence of audio frames and encoding each audio frame of the sequence of audio frames into a corresponding causal audio embedding using a causal encoder. The operations also include generating a non-causal audio embedding using a non-causal encoder configured to receive as input the encoded causal audio embedding, and decoding the non-causal audio embedding using an RNN-T decoder to generate a first pass hypothesis. In some examples, the dilliberation decoder includes a long short-term memory (LSTM) network followed by a softmax layer, where the LSTM network may include at least two layers. In some implementations, the text encoder includes a stack of multiple self-attention blocks, each having a multi-head self-attention mechanism. In these implementations, each self-attention block may include one of a conformer block or a transformer block.

[0005] In some examples, training the diliberation model includes pre-training a text encoder for each non-spoken text utterance by tokenizing the corresponding non-spoken text utterance into a sequence of sub-word units. This example also includes replacing each tokenized sub-word unit in a first portion of the tokenized sequence of sub-word units with a mask token and replacing each tokenized sub-word unit in a second portion of the tokenized sequence of sub-word units with a random token. Additionally or alternatively, training the diliberation model includes generating a corresponding synthetic speech representation for each non-spoken text utterance of the received training data using the text-to-speech model, and training the diliberation model using the non-spoken text utterance and the corresponding synthetic speech representation.

[0006] In some implementations, the training data further includes a plurality of untranscribed non-synthesized speech utterances, each untranscribed non-synthesized speech utterance not paired with a corresponding transcription. In these implementations, the operations further include predicting a corresponding transcription of each untranscribed non-synthesized speech utterance using the trained speech recognition model. Here, training the diliberation model further includes training a diliberation decoder using the untranscribed non-synthesized speech utterances and the corresponding predicted transcriptions as semi-supervised data. In some examples, generating the second pass hypotheses includes generating a first context vector using a first attention mechanism that attends to the encoded first pass hypotheses, generating a second context vector using a second attention mechanism that attends to the non-causal acoustic embeddings, and decoding the first context vector and the second context vector in the diliberation decoder to form the second pass hypotheses.

[0007] Another aspect of the disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including receiving training data including a plurality of non-spoken text utterances, each unpaired with any corresponding spoken utterance of non-synthetic speech, and training a diliberation model including a text encoder and a diliberation decoder for the non-spoken text utterances. The operations also include receiving a first pass hypothesis and a non-causal acoustic embedding in the trained diliberation model. The first pass hypothesis is generated by a recurrent neural network transducer (RNN-T) decoder for the non-causal acoustic embedding encoded by the non-causal encoder. The operations also include encoding the first pass hypothesis generated by the RNN-T decoder using the text encoder, and generating a second pass hypothesis using a diliberation decoder that attends to both the first pass hypothesis and the non-causal acoustic embedding.

[0008] This aspect may include one or more of the following optional features. In some implementations, the operations further include receiving a sequence of audio frames and encoding each audio frame of the sequence of audio frames into a corresponding causal audio embedding using a causal encoder. The operations also include generating a non-causal audio embedding using a non-causal encoder configured to receive as input the encoded causal audio embedding, and decoding the non-causal audio embedding using an RNN-T decoder to generate a first pass hypothesis. In some examples, the dilliberation decoder includes a long short-term memory (LSTM) network followed by a softmax layer, where the LSTM network may include at least two layers. In some implementations, the text encoder includes a stack of multiple self-attention blocks, each having a multi-head self-attention mechanism. In these implementations, each self-attention block may include one of a conformer block or a transformer block.

[0009] In some examples, training the diliberation model includes pre-training a text encoder for each non-spoken text utterance by tokenizing the corresponding non-spoken text utterance into a sequence of sub-word units. This example also includes replacing each tokenized sub-word unit in a first portion of the tokenized sequence of sub-word units with a mask token and replacing each tokenized sub-word unit in a second portion of the tokenized sequence of sub-word units with a random token. Additionally or alternatively, training the diliberation model includes generating a corresponding synthetic speech representation for each non-spoken text utterance of the received training data using the text-to-speech model, and training the diliberation model using the non-spoken text utterance and the corresponding synthetic speech representation.

[0010] In some implementations, the training data further includes a plurality of untranscribed non-synthesized speech utterances, each untranscribed non-synthesized speech utterance not paired with a corresponding transcription. In these implementations, the operations further include predicting a corresponding transcription of each untranscribed non-synthesized speech utterance using the trained speech recognition model. Here, training the diliberation model further includes training a diliberation decoder using the untranscribed non-synthesized speech utterances and the corresponding predicted transcriptions as semi-supervised data. In some examples, generating the second pass hypotheses includes generating a first context vector using a first attention mechanism that attends to the encoded first pass hypotheses, generating a second context vector using a second attention mechanism that attends to the non-causal acoustic embeddings, and decoding the first context vector and the second context vector in the diliberation decoder to form the second pass hypotheses.

[0011] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0012] [Figure 1A] FIG. 1 is a schematic diagram of an example audio environment using a pre-trained diliveration two-pass architecture. [Figure 1B] FIG. 1 is a schematic diagram of an example audio environment using a pre-trained diliveration two-pass architecture. [Diagram 2] FIG. 1 is a schematic diagram of an exemplary direvelation two-pass architecture for speech recognition. [Figure 3A] FIG. 1 is a schematic diagram of an exemplary training process for training a diliberation model of a diliberation two-pass architecture to learn consistent predictions. [Figure 3B]FIG. 1 is a schematic diagram of an exemplary training process for training a diliberation model of a diliberation two-pass architecture to learn consistent predictions. [Figure 3C] FIG. 1 is a schematic diagram of an exemplary training process for training a diliberation model of a diliberation two-pass architecture to learn consistent predictions. [Figure 4] FIG. 2 is a schematic diagram of an example conformer block. [Diagram 5] 4 is a flow chart of an exemplary operational configuration of a method for performing speech recognition. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Like reference symbols in the various drawings indicate like elements.

[0014] Speech recognition continues to evolve to meet the untethered and rapid demands of the mobile environment. In an effort to improve the quality of automatic speech recognition systems (ASR), new speech recognition architectures or improvements to existing architectures continue to be developed. To explain, in the early days of speech recognition, multiple models were used, with each model having a dedicated purpose. For example, ASR systems included an acoustic model (AM), a pronunciation model (PM), and a language model (LM). The acoustic model mapped segments of audio (i.e., frames of audio) to phonemes. The pronunciation model connected these phonemes together to form words, while the language model was used to represent the likelihood of a given phrase (i.e., the probability of it being a sequence of words). Although these individual models worked in tandem, each model was trained independently and often with different datasets manually designed.

[0015] The separate model approach allows speech recognition systems to be fairly accurate, especially when the training corpus (i.e., body of training data) of a given model meets the model's validity. However, the need to train separate models independently presents its own complexities, leading to the development of unified model architectures. These unified models sought to directly map audio waveforms (i.e., input sequences) to output sentences (i.e., output sequences) using a single neural network. This resulted in a sequence-to-sequence conversion approach, where a sequence of words (or graphemes) is generated given a sequence of audio features. Examples of sequence-to-sequence conversion models include the "attention-based" model and the "listen-attend-spell" (LAS) model. The LAS model transcribes spoken utterances into text using a listener component, an attendant component, and a speller component. Here, the listener is a recurrent neural network (RNN) encoder that receives an audio input (e.g., a time-frequency representation of a speech input) and maps the audio input to a higher-level feature representation. The attender attends to the higher-level features to learn alignments between input features and predictive subword units (e.g., graphemes or word pieces). The speller is an attention-based RNN decoder that generates strings from the input by generating probability distributions over a set of virtualized words. With the unified structure, all components of the model can be trained jointly as a single end-to-end (E2E) neural network. Here, an E2E model refers to a model whose architecture is built only with neural networks. The complete neural network works without any external and / or manually designed components (e.g., finite-state transducers, lexicons, or text normalization modules). Furthermore, when training E2E models, these models generally do not require bootstrapping from decision trees or time alignment from a separate system.

[0016] While early E2E models have shown high accuracy and training improvements over individually trained models, these E2E models, such as the LAS model, worked by reviewing the entire input sequence before generating output text, and therefore did not allow streaming output as the input was received. Without streaming capabilities, the LAS model cannot perform real-time speech transcription. This deficiency can make it problematic to deploy the LAS model in speech applications that are latency sensitive and / or require real-time speech transcription. This makes the LAS model alone a less than ideal model for mobile technologies (e.g., mobile phones), which often rely on real-time applications (e.g., real-time communication applications).

[0017] Furthermore, a speech recognition system having acoustic, pronunciation, and linguistic models, or such models configured with each other, may rely on a decoder, which must search a relatively large search graph associated with these models. The large search graph makes this type of speech recognition system unsuitable for hosting entirely on-device. Now, when a speech recognition system is hosted "on-device", the device receiving the audio input uses its processor(s) to perform the functions of the speech recognition system. For example, when a speech recognition system is hosted entirely on-device, the device's processor does not need to coordinate with off-device computing resources to perform the functions of the speech recognition system. Devices performing speech recognition that are not entirely on-device rely on remote computing (e.g., a remote computing system or cloud computing) and therefore rely on online connectivity to perform at least some functions of the speech recognition system. For example, the speech recognition system uses a network connection with a server-based model to perform decoding through a large search graph.

[0018] Unfortunately, the reliance on remote connections makes speech recognition systems vulnerable to latency issues and / or the inherent unreliability of communication networks. To circumvent these issues and improve the usefulness of speech recognition, speech recognition systems have evolved again to a form of sequence-to-sequence conversion model known as recurrent neural network transducer (RNN-T). RNN-T does not employ an attention mechanism, and unlike other sequence-to-sequence conversion models that generally require processing an entire sequence (e.g., an audio waveform) to generate an output (e.g., a sentence), RNN-T processes input samples continuously and streams output symbols. This is a feature that is particularly attractive for real-time communication. For example, speech recognition using RNN-T can output characters one by one as they are spoken. Here, RNN-T uses a feedback loop that feeds the symbols predicted by the model back to itself to predict the next symbol. Because RNN-T decoding involves a beam search through a single neural network instead of a large decoder graph, RNN-T can scale to a fraction of the size of server-based speech recognition models. Due to its reduced size, RNN-T can be deployed completely on-device and can run offline (i.e., without a network connection), thus avoiding the problem of unreliable communication networks.

[0019] A speech recognition system also requires that the recognition of speech be accurate in addition to the speech recognition system operating with low latency. Often, for a model performing speech recognition, a metric that may define the accuracy of the model is the word error rate (WER). WER is a measure of how many words are changed compared to the number of words actually spoken. Generally, these word changes refer to substitutions (i.e., when a word is replaced), insertions (i.e., when a word is added), and / or deletions (i.e., when a word is omitted). To illustrate, a speaker may say "car" but the ASR system may transcribe the word "car" as "bar". This is an example of substitution due to phonetic similarity. When measuring the capabilities of an ASR system, compared to other ASR systems, WER may indicate some measure of improvement or quality of capabilities over another system or some baseline.

[0020] Although RNN-T models have shown promise as strong candidate models for on-device speech recognition, RNN-T models alone still lag behind large-scale conventional state-of-the-art models (e.g., separate AM, PM, and LM server-based models) in terms of quality (e.g., speech recognition accuracy). Furthermore, non-streaming E2E, LAS models have speech recognition quality comparable to large-scale conventional state-of-the-art models. In a two-pass model, the non-streaming LAS model, for example, rescores the streamed hypotheses from the first pass. In this second pass LAS model approach, we attend to the acoustics to reemphasize the hypotheses. In contrast, an alternative method known as a class of neural correction models generates hypotheses using text instead of acoustics. In other words, there are different variables that can be attended to refine the hypotheses in the second pass. Thus, the model proposed herein is a variation of the two-pass model of RNN-T / LAS. This variation uses a dilevelation network that combines the acoustic and text hypotheses of the first pass for the second pass of the two-pass model.

[0021] To improve the quality of voice search, embodiments herein are directed to pre-training a second pass of a diliberation network (e.g., as shown in Figures 3A-3C) using text-only data in a masked language model. In addition to using text-only data, unlabeled audio utterances are used for semi-supervised training of a text encoder. By incorporating text-only data in pre-training the diliberation network, joint training of the acoustic and text decoders, and semi-supervised training in a single model, this pre-trained two-pass diliberation network can be more accurate than large-scale conventional speech recognition models. For example, in some tests, the pre-trained two-pass diliberation network achieved a 4.1% voice search performance improvement and a nearly 12% reduction in long-tail WER compared to an untrained two-pass diliberation network, and a relative reduction in WER of 8% when compared to a large-scale conventional recognition model.

[0022] 1A and 1B are exemplary systems 100a, 100b including an audio environment 100 in which a user 10's way of interacting with a computing device, such as a user device 110, may be via voice input. The user device 110 (also commonly referred to as device 110) is configured to capture sounds (e.g., streaming audio data) from one or more users 10 within the audio-enabled environment. Here, the streaming audio data 12 may refer to spoken utterances by the users 10 that serve as audible queries, commands to the device 110, or audible communications captured by the device 110. The audio-enabled system of the device 110 may act on the queries or commands by responding to the queries and / or causing the commands to be executed.

[0023] The user device 110 may correspond to any computing device associated with the user 10 and capable of receiving audio data 12. Some examples of the user device 110 include, but are not limited to, a mobile device (e.g., a mobile phone, a tablet, a laptop, etc.), a computer, a wearable device (e.g., a smart watch), a smart appliance, an Internet of Things (IoT) device, a smart speaker, etc. The user device 110 includes data processing hardware 112 and memory hardware 114 in communication with the data processing hardware 112, which stores instructions that, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations. The user device 110 further includes an audio subsystem 116 with audio capture devices (e.g., microphones) 116, 116a for capturing and converting spoken utterances 12 into electrical signals, and audio output devices (e.g., speakers) 116, 116b for communicating audible audio signals (e.g., as output audio data from the user device 110) within the voice-enabled system 100. In the illustrated example, the user device 110 implements a single audio capture device 116a, however, the user device 110 may implement an array of audio capture devices 116a without departing from the scope of this disclosure, where one or more capture devices 116a of the array may be in communication with the audio subsystem 116 without being physically present at the user device 110. The user device 110 is further configured to perform speech recognition processing on the streaming audio data 12 using a speech recognizer 200. The speech recognizer 200 (also referred to as model 200) resides on the user device 110 of the user 10 (e.g., hardware 110, 112) and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system implemented in a cloud computing environment) that communicates with the user device 10 via a network 40.In some examples, the audio subsystem 116 of the user device 110, including the audio capture device 116a, is configured to receive audio data 12 (e.g., a spoken utterance) and convert the audio data 12 into a digital format compatible with the speech recognizer 200. The digital format can correspond to acoustic frames, such as mel frames (e.g., parameterized acoustic frames). For example, the parameterized acoustic frames correspond to log-mel filter bank energies.

[0024] In some examples, such as FIG. 1A, a user 10 interacts with a program or application 118 of a user device 110 that uses a speech recognizer 200. For example, FIG. 1A shows a user 10 communicating with an automated assistant application. In this example, the user 10 asks the automated assistant, "What time is the concert tonight?" This question from the user 10 is a spoken utterance 12 that is captured by an audio capture device 116a and processed by an audio system 116 of the user device 110. In this example, the speech recognizer 200 of the user device 110 receives an audio input 202 (e.g., as an acoustic frame) of "What time is the concert tonight?" and transcribes the audio input 202 into a transcription 204 (e.g., a textual representation of "What time is the concert tonight?"). The automated assistant of the application 118 can now respond to the question posed by the user 10 using natural language processing. Natural language processing generally refers to the process of interpreting written phrases (e.g., transcript 204) and determining whether the written phrases prompt any actions. In this example, the automated assistant uses natural language processing to recognize that a question from the user 10 considers the user's schedule, and more specifically, a concert on the user's schedule. By using natural language processing to recognize these details, the automated assistant returns a response to the user's query. The response is, "Tonight's concert opens at 8:30 p.m." In some configurations, natural language processing may occur at a remote system in communication with the data processing hardware 112 of the user device 110.

[0025] 1B is another example of speech recognition using a speech recognizer 200. In this example, a user 10 associated with a user device 110 is communicating with a friend named Jane Doe using a communication application 118. Here, the user 10, named Ted, communicates with Jane by having speech inputs transcribed by the speech recognizer 200. The audio capture device 116 captures these speech inputs and communicates them in digital form (e.g., acoustic frames) to the speech recognizer 200. The speech recognizer 200 transcribes these acoustic frames into text, which is sent to Jane via the communication application 118. Because this type of application 118 communicates via text, the transcriptions 204 from the speech recognizer 200 can be sent to Jane without further processing (e.g., natural language processing).

[0026] In some examples, such as FIG. 2, the speech recognizer 200 is configured with an enhanced two-pass architecture in which a first pass 206 is followed by a second pass 208. Generally speaking, the two-pass architecture of the speech recognizer 200 includes a first encoder 210 (e.g., a causal encoder 210), a second encoder 220 (e.g., an acausal encoder 220), an RNN-T decoder 230, and a diliberation model 240. In the two-pass decoding, the second pass 208 can improve the initial output from the first pass 206 using techniques such as lattice rescoring or n-best reranking. In other words, the RNN-T decoder 230 generates streaming predictions, and the diliberation model 240 finishes the predictions. Here, specifically, the diliberation model 240 uses the hypotheses 232y streamed from the RNN-T decoder 230 to generate the predictions. R The direvelation model 240 rescores the hypotheses 232y from the RNN-T decoder 230. R Although it is generally discussed that the diliveration model 240 functions in a rescoring mode in which the diliveration model 240 rescores the utterance, it is also possible that the diliveration model 240 operates in a different mode, such as a beam search mode, depending on the design or other factors (e.g., the length of the utterance).

[0027] As shown in FIG. 2, the first path 206 includes a first encoder 210 and a second encoder 220 arranged in a cascade, which refers to a model structure in which the encoding path includes two encoders 210, 220 that cascade such that the output of one encoder 210 feeds the input of the other encoder 220 before decoding. Here, the encoders 210, 220 can be cascaded regardless of the underlying architecture of each encoder. In some examples, the encoders 210, 220 include a stack of multiple conformer layers of 512 dimensions. Causal convolutional layers and left context attention layers can be used in each conformer layer to strictly constrain the model to not use future inputs. A multi-head (e.g., eight head) attention mechanism can be used in the self-attention layer. The cascade of encoders 210, 220 can include 21 conformer layers. Here, the first encoder 210 may include 17 conformer layers, while the second encoder 220 may include 4 conformer layers that incorporate additional right-side context (e.g., 0.9 seconds). Optionally, other types of layers incorporating self-attention mechanisms, such as transformer layers, can be used instead of conformer layers. The first encoder 210 may be referred to as a causal encoder, and the second encoder 220 may be referred to as a non-causal encoder.

[0028] In other implementations, one encoder is constructed using an LSTM structure, and the other encoder is constructed using a bidirectional LSTM layer or a conformer layer (e.g., a conformer-transducer). In other words, the encoders 210, 220 may have different or similar architectures. For example, the cascaded encoders 210, 220 may be roughly similar to an Acoustic Model (AM) in a conventional ASR system and may include a recurrent network of stacked Long Short Term Memory (LSTM) layers. Here, the first encoder 210 is a streaming encoder that includes a unidirectional Long Short Term Memory (LSTM) layer, and the second encoder 220 is a non-streaming encoder that includes a bidirectional LSTM layer or a conformer layer. In a cascaded encoder, both encoders 210, 220 include LSTM layers, and the second encoder 220, which receives the output of the first encoder 210, can utilize the LSTM layers of the first encoder 210 such that the second encoder 220 includes fewer LSTM layers than the first encoder 210 (and fewer LSTM layers than a completely non-streaming model). By using fewer LSTM layers, a cascaded encoder can reduce the number of computationally expensive bidirectional layers and make the speech recognizer 200 more streamlined than simply combining a traditional streaming model with a traditional non-streaming model.

[0029] The first encoder 210 encodes a sequence of d-dimensional feature vectors (e.g., acoustic frames) x=(x1, x2, . . . , x T ), in the formula x t ∈R d ) and at each time step produces a first high-level feature representation as output 212. This first high-level feature representation may include a causal acoustic embedding, e s Similarly, the second encoder 220 is cascaded to the first encoder 210 and receives as input the first high-level feature e sand is trained to generate a second higher-level feature representation as output 222. This second higher-level feature representation includes a non-causal acoustic embedding, e a Both the first encoder 210 and the second encoder 220 are directly connected to and shared by the RNN-T decoder 230. Thus, the RNN-T decoder 230 receives the first high-level feature representation e s and the second high-level feature representation e a The RNN-T decoder 230 then receives as input both the first high-level feature representation e s and the second high-level feature representation e a The first speech recognition result y r 222. In some examples, each parameterized acoustic frame includes 128-dimensional log-mel features computed within a short shift window (e.g., 32 ms with shifts of 10 ms each). Each feature may be stacked with the previous frame (e.g., the previous three frames) to form a higher dimensional vector (e.g., a 512-dimensional vector using the previous three frames). The features forming the vector may then be downsampled (e.g., to the 30 ms frame rate).

[0030] In some implementations, the RNN-T decoder 230 includes a combination layer and an embedding prediction network, where the prediction network may have two LSTM layers of 2048 hidden units, a 640-dimensional projection per layer, and an embedding layer of 128 units. The RNN-T decoder 230 uses the combination layer to combine the first and second high-level feature representations output by the encoders 210, 220. s , e a , as well as the embedding output y from the predictive network for the preceding prediction r-1 and the first path hypothesis y r The decoder outputs 232 the preceding units of the N preceding non-blank symbols {y i-1 ...,y i-N} and given a sequence of input x, the current subword unit y iProbability distribution P(y i |y i-1 , ..., y0, x). In some examples, the combination network of the RNN-T decoder 230 includes 640 hidden units followed by a softmax layer that predicts 4,096 mixed-case word pieces. In some implementations, the softmax layer is separate from the RNN-T decoder 230 and the output 232 from the RNN-T decoder 230, y r The output of the softmax layer is then used in the beam search process to select the orthographic element. In some implementations, the softmax layer is integrated with the RNN-T decoder 230, whereby the output 232 of the RNN-T decoder 230, y r represents the output of the softmax layer.

[0031] 2, the second pass 208 uses a diliberation model 240 that includes a text encoder 242 and two attention mechanisms 244, 246, a hypothesis attention mechanism 244, and an acoustic attention mechanism 246, in addition to a diliberation decoder 250 (also referred to as a LAS decoder 250). As described in more detail below (e.g., FIGS. 3A-3C), the text encoder 242 of the diliberation model 240 may be pre-trained using text-only (e.g., non-spoken text utterances) input. Here, the speech recognizer 200 uses the second high-level feature representation e of the second encoder 220 in the acoustic attention mechanism 246 to generate a diliberation model 240. a 222 of the RNN-T decoder 230 and the first pass hypothesis y r Attend to both the acoustics (e.g., a y , the output 222 of the first path hypothesis (e.g., rBy attending to both the first and second attention mechanisms 244, 246 (output 232 represented as ,), the diliberation model 240 generates a second pass hypothesis as output 248 (e.g., a predicted sequence). Here, each attention mechanism 244, 246 forms a context vector 245, 247 (e.g., a hypothesis context vector 245 and an acoustic context vector 247, or a first context vector 245 and a second context vector 247), which is input to a diliberation decoder 250 of the diliberation model 240. These context vectors 245, 247 may be concatenated as inputs to the diliberation decoder 250.

[0032] The text encoder 242 further encodes the output 232 of the RNN-T decoder 230 (i.e., the output 232 of the first pass 206) to generate an encoded hypothesis 243 (e.g., B 2, the text encoder 242 forms a multi-head attention mechanism 420 (denoted as ) for each of the output 232. When further encoding the output 232, the text encoder 242 may also encode the output 232 for useful context information that can be included in the encoded hypotheses 243. For example, the text encoder 242 is a bidirectional encoder that can include the context information. The text encoder 242 may also be configured to encode multiple first hypotheses (i.e., the output 232). For example, the text encoder 242 encodes each hypothesis 232 separately and then concatenates each encoded hypothesis. The text encoder 242 may include a stack of multiple multi-head attention blocks 400 (referred to herein as conformer blocks 400), which may include a conformer or a transformer. Each multi-head attention block 400 may include a multi-head attention mechanism 420 (FIG. 4). For example, the text encoder 242 may be a two-layer conformer encoder, with each layer having 640-dimensional projections per layer with multi-token (e.g., two-token) correct context.

[0033] During the second pass 208, the speech recognizer 200 may perform a beam search mode or a rescoring mode to generate output 248 (i.e., second pass hypotheses). In the rescoring mode, the diliberation model 240 may be performed in a teacher-forced mode with output 232. Additionally or alternatively, using the bidirectional text encoder 242 when in the rescoring mode may help improve the relative WER of the diliberation decoder two-pass architecture of the diliberation model 240. When the diliberation decoder 250 operates in a beam search mode, the diliberation decoder 250 generates the second pass hypotheses as output 248 from only output 222 and ignores the output 232 of the RNN-T decoder 230. When the diliberation decoder 250 operates in rescoring mode, the diliberation decoder 250 takes the top K hypotheses (e.g., the four first pass hypotheses) from the RNN-T decoder 230, and then the diliberation decoder 250 runs on each sequence in a supervised forced mode, attending to the output 222 to calculate a score. For example, the score is a combination of the log probability of the sequence and an attention coverage penalty. The diliberation decoder 250 selects the sequence with the highest score as the output 222. Here, the diliberation decoder 250 may include multi-head attention (e.g., four heads) to attend to the output 222. Furthermore, the diliberation decoder 250 may be a two-layer LSTM network followed by a softmax layer for prediction. For example, each layer of the diliberation decoder 250 has 2,048 hidden units followed by a 640-dimensional projection. The softmax layer may include 4,096 dimensions to predict word pieces for the same mixed case from the softmax layer of the RNN-T decoder 230. Much like the attention mechanism inherent in the diliberation decoder 250 described above, the attention mechanisms 244, 246 may have a similar structure such that each attention mechanism 244, 246 includes multi-head attention (e.g., eight heads).

[0034] Neural networks are typically trained by backpropagation, which defines a loss function (e.g., a cross-entropy loss function). For example, the loss function is defined as the difference between the actual output of the network and the desired output of the network. Here, the speech recognizer 200 may be trained using a cross-entropy loss approach, a joint training approach, or a combination of cross-entropy loss and joint training. In the cross-entropy loss approach, a diliberation model, such as the speech recognizer 200 with a diliberation model 240 (i.e., the diliberation-based speech recognizer 200), is trained in a two-step training process. During the first step of the training process, the RNN-T decoder 230 is trained. After the RNN-T decoder 220 is trained, the parameters of the RNN-T decoder 230 are fixed, and only the diliberation model 240 and the additional encoder layer (e.g., the text encoder 242) are trained.

[0035] 3A-3C illustrate exemplary training processes 300a-300c for training the diliberation model 240 of the speech recognizer 200. In some configurations, the training processes 300a-300c are executed on the remote computing device 60 of FIGS. 1A and 1B. The training processes 300a-c obtain a set of training data 320, 320a-n, stored in a sample database 310, and train the diliberation model 240 with the training data 320. The training data 320 includes a plurality of training non-spoken text utterances 330, 330a-n, where each training non-spoken text utterance 330 is not paired with any corresponding spoken utterance of the non-synthetic voice. The sample database 310 may reside in the memory hardware of the remote computing device 60. In the illustrated example, the training data 320 is selected to train the diliberation model 240, which includes the text encoder 242 and the diliberation decoder 250. Here, the diliberation model 240 receives training data 320 and generates outputs that are tested for their accuracy. Although Figures 3A-3C show separate training processes 300a-c, it should be understood that the diliberation decoder 250 may be trained by any combination of the training processes 300a-c.

[0036] Referring to FIG. 3A, the training process 300a trains the dilevelation model 240 by pre-training the text encoder 242 on each training non-spoken text utterance 330 of the training data 320. Here, the training process 300a includes a masking module 340 that processes each training non-spoken text utterance 330 before training the text encoder 242 using cross-entropy loss. The masking module 340 includes a token module 342 and a masker 346. Intuitively, since the text encoder 242 is bidirectional (i.e., has both right and left context), it can easily predict the target word in the training sample. To train the text encoder 242, a percentage of each training non-spoken text utterance 330 is randomly masked. The token module 342 takes each corresponding training non-spoken text utterance 330 and tokenizes the corresponding training non-spoken text utterance 330 into a tokenized sequence 344a-n of sub-word units 344. The masker 346 receives the tokenized sequence of sub-word units 344 and randomly selects a percentage (e.g., 15%) of the tokenized sequence of sub-word units 344 to replace with a different token. For example, the masker 346 replaces each tokenized sub-word unit 344 in a first portion of the tokenized sequence of sub-word units 344 with a mask token 344M and replaces each tokenized sub-word unit 344 in a second portion of the tokenized sequence of sub-word units 344 with a random token 344M. Specifically, the masker 346 replaces each tokenized subword unit 344 in the tokenized sequence of subword units 344 with a mask token 344M 80% of the time, replaces it with a random token 344R 10% of the time, and leaves the tokenized subword unit 344 unchanged 10% of the time.For example, as shown in Figure 3A, the masking module 340 takes the training non-spoken text utterance 330a and tokenizes (i.e., using the token module 340) the training non-spoken text utterance 330a into a sequence of sub-word units 344 to generate four tokenized sub-word units 344. The masker 346 receives the four tokenized sub-word units 344 and outputs two unchanged tokenized sub-word units 344, one mask token 344M, and one random token 344R. Once the training process 300a is complete, the parameters of the text encoder 242 and the dilevelation decoder 250 are jointly updated with additional training, while the parameters of the RNN-T decoder 230 remain fixed.

[0037] 3B, the training process 300b uses the training non-spoken text utterances 330 of the training data 320 to train the diliberation model 240. Here, the training process 300b includes a text-to-speech module 350 that processes each training non-spoken text utterance 330 before training the diliberation model 340. In particular, the text-to-speech module 350 takes each training non-spoken text utterance 330 and uses a text-to-speech model to generate a corresponding synthetic speech representation 352. The training process 300b then uses the training non-spoken text utterances 330 and the corresponding synthetic speech representation 352 to train the diliberation model 240. For example, as shown in FIG. 3B, the text-to-speech module 350 receives the training non-spoken text utterances 330a and uses a text-to-speech model to generate the corresponding synthetic speech representation 352 as an output. The training process 300b then jointly trains the dilivelation model 240 using the training non-spoken text utterances 330a and the corresponding synthetic speech representations 352. The training process 300b can calculate both audio and text attention when using the training non-spoken text utterances 330a and calculate fixed context vectors to replace attention when the corresponding synthetic speech representations 352 are used. The training process 300b can select a mix (e.g., a 1:9 ratio) of the training non-spoken text utterances 330 and the corresponding synthetic speech representations 352 when training the dilivelation model 240.

[0038] Referring to FIG. 3C, the training process 300c trains the diliberation model 240 using the training data 320. Here, the training data 320 further includes a plurality of training untranscribed non-synthesized speech utterances 332, 332a-n. Here, each training untranscribed non-synthesized speech utterance 332 is not paired with a corresponding transcription. The training process 300c can use a trained speech recognition model 360 (also referred to as ASR model 360) that is trained to predict the corresponding transcription of the input utterance as an output based on the input utterance. In particular, the ASR model 360 takes each training untranscribed non-synthesized speech utterance 332 and generates a corresponding transcription 362. Then, the training process 300c trains the diliberation model 240 by training the diliberation decoder 250 using the training untranscribed non-synthesized speech utterances 332 and the corresponding transcription 362. 3C, ASR model 360 receives untranscribed, non-synthesized speech utterances under training 332a and uses ASR model 360 to generate corresponding transcriptions 362 as output. Training process 300c then trains dilevelation decoder 250 using untranscribed, non-synthesized speech utterances under training 332 and corresponding predicted transcriptions 362 as semi-supervised data.

[0039] FIG. 4 presents an example of a conformer block 400 from a stack of conformer layers of the text encoder 242. The conformer block 400 includes a first half-feedforward layer 410 and a second half-feedforward layer 440, between which a multi-head self-attention block 420 and a convolutional layer 430 are disposed, further including a concatenation operator 405. The first half-feedforward layer 410 processes an input hypothesis (e.g., the output 232 from the RNN-T decoder 230). The multi-head self-attention block 420 then receives the input hypothesis concatenated with the output of the first half-feedforward layer 410. Intuitively, the role of the multi-head self-attention block 420 is to separately summarize the context for each input frame to be enhanced. The convolutional layer 430 subsamples the output of the multi-head self-attention block 420 concatenated with the output of the first half-feedforward layer 410. A second semi-feedforward layer 440 then receives the concatenation of the output of the convolutional layer 430 and the multi-head self-attention block 420. A layernorm module 450 processes the output from the second semi-feedforward layer 440. Mathematically, the conformer block 400 transforms the input features x using the modulation features m to generate output features y as follows:

number

[0040] 5 is a flowchart of an exemplary arrangement of operations of a method 500 for performing automatic speech recognition (e.g., ASR) using a diliberation two-pass architecture. At operation 502, the method 500 receives training data 320 including non-spoken text utterances 330, where each non-spoken text utterance 330 is not paired with any corresponding spoken utterance of non-synthetic speech. At operation 504, the method 500 includes training a diliberation model 240 with the non-spoken text utterances 330. The diliberation model 240 includes a text encoder 242 and a diliberation decoder 250.

[0041] At operation 506, the method 500 includes receiving the first pass hypothesis 232 and the acausal acoustic embedding 222 at the trained divelation model 240. The first pass hypothesis 232 is generated by a recurrent neural network transducer (RNN-T) decoder 230 for the acausal acoustic embedding 222 encoded by the acausal encoder 220. The method 500 also includes, at operation 508, encoding the first pass hypothesis 232 generated by the RNN-T decoder 230 using a text encoder 242. At operation 510, the method 500 further includes generating a second pass hypothesis 248 using a divelation decoder 250 that attends to both the first pass hypothesis 232 and the acausal acoustic embedding 222.

[0042] 6 is a schematic diagram of an exemplary computing device 600 that can be used to implement the systems and methods described herein. Computing device 600 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0043] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be mounted on a common motherboard or may exist in other manners as desired. Processor 610 (e.g., data processing hardware 112, or data processing hardware of remote computing device 60 of FIG. 1) may process instructions for execution within computing device 600, including instructions stored in memory 620 or storage device 630, and display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 680 connected to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used as desired, along with multiple memories and memory types. Also, multiple computing devices 600 may be connected, each performing some of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0044] Memory 620 (e.g., memory hardware 114 or memory hardware of remote computing device 60 of FIG. 1) stores information non-transiently within computing device 600. Memory 620 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). Non-transient memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0045] The storage device 630 can provide mass storage for the computing device 600. In some embodiments, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 can be a device array, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional embodiments, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0046] The high-speed controller 640 manages bandwidth-intensive operations of the computing device 600, and the low-speed controller 660 manages low-bandwidth-intensive operations. Such an allocation of roles is merely exemplary. In some implementations, the high-speed controller 640 is coupled to a memory 620, a display 680 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to a storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690 may include various communication ports (USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and can connect, via a network adapter or the like, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0047] The computing device 600 can be implemented in many different forms, as shown in the figure: for example, it can be implemented as a standard server 600a, or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0048] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be specialized or general purpose, coupled to receive data and instructions from the storage system and to transmit data and instructions to the storage system.

[0049] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0050] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic circuit (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0051] The processes and logic flows described herein can be implemented by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be implemented by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, for example, both general purpose and special purpose processors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0052] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device, such as a CRT or LCD monitor, or a touch screen, for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, such as acoustic input, speech input, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0053] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (500) that, when executed on data processing hardware (610), causes the data processing hardware (610) to perform operations, the operations comprising: receiving training data (320) including a plurality of non-spoken text utterances (330), each non-spoken text utterance (330) not paired with any corresponding spoken utterance of unsynthesized speech; training a diliberation model (240) on the plurality of non-spoken text utterances (330), the diliberation model (240) including a text encoder (242) and a diliberation decoder (250); receiving the first pass hypothesis (232) generated by a recurrent neural network transducer (RNN-T) decoder (230) for the non-causal acoustic embedding (222) encoded by a non-causal encoder (220) with the trained diliveration model (240), a first pass hypothesis (232), and a non-causal acoustic embedding (222); encoding the first path hypothesis (232) generated by the RNN-T decoder (230) using the text encoder (242); and generating a second pass hypothesis (248) using the diliveration decoder (250) that attends to both the first pass hypothesis (232) and the non-causal acoustic embedding (222); The computer-implemented method (500).

2. The operation, Receiving a sequence of acoustic frames (202); encoding each acoustic frame (202) of the sequence of acoustic frames (202) into a corresponding causal acoustic embedding (212) using a causal encoder (210); generating the non-causal acoustic embeddings (222) using the non-causal encoder (220) configured to receive as input the encoded causal acoustic embeddings (212); and decoding the non-causal acoustic embedding (222) using the RNN-T decoder (230) to generate the first pass hypothesis (232); The computer-implemented method of claim 1 , further comprising:

3. 3. The computer-implemented method of claim 1 or 2, wherein the dilluviation decoder comprises a long short-term memory (LSTM) network followed by a softmax layer.

4. The computer-implemented method of claim 3 , wherein the LSTM network comprises at least two layers.

5. The computer-implemented method (500) of any of claims 1 to 3, wherein the text encoder (242) includes a stack of multiple self-attention blocks (400) each having a multi-head self-attention mechanism.

6. The computer-implemented method of claim 5 , wherein each self-attention block comprises one of a conformer block or a transformer block.

7. Training the dillivelation model (240) includes, for each non-spoken text utterance (330), tokenizing the corresponding unspoken text utterance (330) into a sequence of sub-word units (344); replacing each tokenized sub-word unit (344) of a first portion of the tokenized sequence of sub-word units (344) with a mask token (344M); and replacing each token sub-word unit (344) of a second portion of the tokenized sequence of sub-word units (344) with a random token (344R); The computer-implemented method (500) of any of claims 1 to 6, further comprising pre-training the text encoder (242) by

8. Training the direvelation model (240) comprises: generating, for each non-spoken text utterance (330) of the received training data (320), a corresponding synthetic speech representation (352) using a text-to-speech model (350); and training the diliveration model (240) using the non-spoken text utterances (330) and corresponding synthetic speech representations (352); The computer-implemented method (500) of any of claims 1 to 7, comprising:

9. The training data (320) further includes a plurality of untranscribed, non-synthesized speech utterances (332), each untranscribed, non-synthesized speech utterance (332) not paired with a corresponding transcription; The operation, using the trained speech recognition model (360) to predict, for each untranscribed, non-synthesized speech utterance (332), the corresponding transcription (362); training the diliberation model (240) further comprises training the diliberation decoder (250) using the untranscribed, non-synthesized speech utterances (332) and the predicted corresponding transcriptions (362) as semi-supervised data. A computer-implemented method (500) according to any of the preceding claims.

10. Generating the second path hypothesis (248) comprises: generating a first context vector (245) using a first attention mechanism (244) that attends to the encoded first path hypothesis (232); generating a second context vector (247) using a second attention mechanism (246) that attends to the non-causal acoustic embeddings (222); and decoding the first context vector (245) and the second context vector (247) in the diversion decoder (250) to form the second path hypothesis (248); 10. The computer-implemented method (500) of claim 1, comprising:

11. A system (100), comprising: Data processing hardware (610); and memory hardware (620) storing instructions that, when executed by the data processing hardware (610), cause the data processing hardware (610) to perform operations, receiving training data (320) including a plurality of non-spoken text utterances (330), each non-spoken text utterance (330) not paired with any corresponding spoken utterance of unsynthesized speech; training a diliberation model (240) on the plurality of non-spoken text utterances (330), the diliberation model (240) including a text encoder (242) and a diliberation decoder (250); receiving the first pass hypothesis (232) generated by a recurrent neural network transducer (RNN-T) decoder (230) for the non-causal acoustic embedding (222) encoded by a non-causal encoder (220) with the trained diliveration model (240), a first pass hypothesis (232), and a non-causal acoustic embedding (222); encoding the first path hypothesis (232) generated by the RNN-T decoder (230) using the text encoder (242); and generating a second pass hypothesis (248) using the diliveration decoder (250) that attends to both the first pass hypothesis (232) and the non-causal acoustic embedding (222); The system (100).

12. The operation, Receiving a sequence of acoustic frames (202); encoding each acoustic frame (202) of the sequence of acoustic frames (202) into a corresponding causal acoustic embedding (212) using a causal encoder (210); generating the non-causal acoustic embeddings (222) using the non-causal encoder (220) configured to receive as input the encoded causal acoustic embeddings (212); and decoding the non-causal acoustic embedding (222) using the RNN-T decoder (230) to generate the first pass hypothesis (232); The system (100) of claim 11, further comprising:

13. 13. The system (100) of claim 11 or 12, wherein the diliberation decoder (250) comprises a long short-term memory (LSTM) network followed by a softmax layer.

14. The system (100) of claim 13, wherein the LSTM network comprises at least two layers.

15. The system (100) of any of claims 11 to 14, wherein the text encoder (242) comprises a stack of multiple self-attention blocks (400) each having a multi-head self-attention mechanism.

16. The system (100) of claim 15, wherein each self-attention block (400) comprises one of a conformer block or a transformer block.

17. Training the dillivelation model (240) includes, for each non-spoken text utterance (330), tokenizing the corresponding unspoken text utterance (330) into a sequence of sub-word units (344); replacing each tokenized sub-word unit (344) of a first portion of the tokenized sequence of sub-word units (344) with a mask token (344M); and replacing each token sub-word unit (344) of a second portion of the tokenized sequence of sub-word units (344) with a random token (344R); 17. The system (100) of claim 11, further comprising pre-training the text encoder (242) by:

18. Training the direvelation model (240) comprises: generating, for each non-spoken text utterance (330) of the received training data (320), a corresponding synthetic speech representation (352) using a text-to-speech model (350); and training the diliveration model (240) using the non-spoken text utterances (330) and corresponding synthetic speech representations (352); The system (100) of any of claims 11 to 17, comprising:

19. The training data (320) further includes a plurality of untranscribed, non-synthesized speech utterances (332), each untranscribed, non-synthesized speech utterance (332) not paired with a corresponding transcription; The operation, using the trained speech recognition model (360) to predict, for each untranscribed, non-synthesized speech utterance (332), the corresponding transcription (362); training the diliberation model (240) further comprises training the diliberation decoder (250) using the untranscribed, non-synthesized speech utterances (332) and the predicted corresponding transcriptions (336) as semi-supervised data. A system (100) according to any one of claims 11 to 18.

20. Generating the second path hypothesis (248) comprises: generating a first context vector (245) using a first attention mechanism (244) that attends to the encoded first path hypothesis (232); generating a second context vector (247) using a second attention mechanism (246) that attends to the non-causal acoustic embeddings (222); and decoding the first context vector (245) and the second context vector (247) in the diversion decoder (250) to form the second path hypothesis (248); The system (100) of any of claims 11 to 19, comprising:

Images