Auditions through plain text and semi-supervised training

Through the combination of RNN-T model and two-pass architecture, using plain text and semi-supervised training, the delay and accuracy problems of speech recognition systems in the prior art in real-time speech transcription are solved, and the low latency on mobile devices is achieved and highly accurate speech recognition improve speech recognition performance.

CN118974742APending Publication Date: 2024-11-15GOOGLE LLC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202380034352.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-19
Filing Date
2023-03-18
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing automatic speech recognition systems have latency and accuracy issues in real-time speech transcription, especially on mobile devices, where streaming and low-latency real-time voice transcription cannot be achieved, and rely on remote computing and large search graphs lead to latency and Network unreliability issues.

Method used

Using the recurrent neural network transducer (RNN-T) model combined with plain text and semi-supervised training, the two-pass architecture of the text encoder and deliberate decoder uses causal and non-causal acoustic embeddings, combining the multi-head self-attention mechanism and the Conformer block, Perform training and pre-training of speech recognition models to improve the accuracy and streaming capabilities of speech recognition.

Benefits of technology

Achieve low latency and high accuracy speech recognition on mobile devices, improving voice search performance by 4.1% and reducing long-tail word error rate by 12%, and reducing relative word error rate by 8% compared to large regular models .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118974742A_ABST
    Figure CN118974742A_ABST
Patent Text Reader

Abstract

A plain text and semi-supervised training auditing method (500) includes receiving training data (320) including non-spoken text utterances (330) each not paired with any corresponding spoken utterances of non-synthetic speech, and training an auditing model (240) including a text encoder (242) and an auditing decoder (250) on the non-spoken text utterances. The method also includes receiving a first pass of hypotheses (232) and non-causal acoustic embedding (222) at the trained auditing model. The first pass of hypotheses is generated by a recurrent neural network transducer (RNN-T) decoder (230) for non-causal acoustic embedding encoded by a non-causal encoder. The method further includes encoding, using a text encoder, a first pass hypothesis generated by the RNN-T decoder, and generating a second pass hypothesis (248) using an auditing decoder that concerns both the first pass hypothesis and non-causal acoustic embedding.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to deliberation via plain text and semi-supervised training. Background Art

[0002] Modern automated speech recognition (ASR) systems focus not only on providing high quality (e.g., low word error rate (WER)), but also on providing low latency (e.g., a short delay between the user speaking and the transcription appearing). In addition, when ASR systems are used today, they are required to decode the utterances in a streaming manner that corresponds to real time or even faster than real time. For example, when the ASR system is deployed on a mobile phone that experiences direct user interaction, an application on the mobile phone using the ASR system may require speech recognition to be performed in a streaming manner so that the word appears on the screen as soon as it is spoken. Here, it is also likely that the user of the mobile phone has a low tolerance for delays. Due to this low tolerance, speech recognition strives to run on the mobile device in a manner that minimizes the impact of delays and inaccuracies that may adversely affect the user experience. Summary of the invention

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes data processing hardware to perform operations, the operations comprising receiving training data, the training data comprising non-verbal text utterances that are each not paired with any corresponding spoken utterance of non-synthesized speech, and training a deliberative model comprising a text encoder and a deliberative decoder on the non-verbal text utterances. The operations also include receiving a first-pass hypothesis and a non-causal acoustic embedding at the trained deliberative model. The first-pass hypothesis is generated by a recurrent neural network transducer (RNN-T) decoder for the non-causal acoustic embedding encoded by the non-causal encoder. The operations also include encoding the first-pass hypothesis generated by the RNN-T decoder using the text encoder, and generating a second-pass hypothesis using a deliberative decoder that focuses on both the first-pass hypothesis and the non-causal acoustic embedding.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operation also includes receiving a sequence of acoustic frames, and encoding each acoustic frame in the sequence of acoustic frames into a corresponding causal acoustic embedding using a causal encoder. The operation also includes generating a non-causal acoustic embedding using a non-causal encoder configured to receive a causally encoded acoustic embedding as input, and decoding the non-causal acoustic embedding using an RNN-T decoder to generate a first pass hypothesis. In some examples, the deliberative decoder includes a long short-term memory (LSTM) network, followed by a softmax layer. Here, the LSTM network may include at least two layers. In some implementations, the text encoder includes a stack of self-attention blocks, each of which has a multi-head self-attention mechanism. In these implementations, each self-attention block may include one of a Conformer block or a Transformer block.

[0005] In some examples, training the deliberative model includes pre-training a text encoder on each non-verbal text utterance by tokenizing the corresponding non-verbal text utterance into a sequence of sub-word units. The example also includes replacing each tokenized sub-word unit in a first portion of the tokenized sub-word unit sequence with a masked token, and replacing each token sub-word unit in a second portion of the tokenized sub-word unit sequence with a random token. Additionally or alternatively, training the deliberative model includes generating a corresponding synthetic speech representation for each non-verbal text utterance of the received training data using a text-to-speech model, and training the deliberative model using the non-verbal text utterance and the corresponding synthetic speech representation.

[0006] In some implementations, the training data also includes untranscribed non-synthetic speech utterances, each untranscribed non-synthetic speech utterance is not paired with a corresponding transcription. In these implementations, the operation also includes predicting the corresponding transcription of each untranscribed non-synthetic speech utterance using the trained speech recognition model. Here, training the deliberative model also includes training a deliberative decoder using the untranscribed non-synthetic speech utterances and the predicted corresponding transcriptions as semi-supervised data. In some examples, generating a second-pass hypothesis includes generating a first context vector using a first attention mechanism focusing on the encoded first-pass hypothesis, generating a second context vector using a second attention mechanism focusing on the non-causal acoustic embedding, and decoding the first context vector and the second context vector at the deliberative decoder to form the second-pass hypothesis.

[0007] Another aspect of the present disclosure provides a system comprising data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations, the operations comprising receiving training data comprising non-verbal text utterances that are each not paired with any corresponding spoken utterance of non-synthesized speech, and training a deliberative model comprising a text encoder and a deliberative decoder on the non-verbal text utterances. The operations also include receiving a first pass hypothesis and a non-causal acoustic embedding at the trained deliberative model. The first pass hypothesis is generated by a recurrent neural network transducer (RNN-T) decoder for a non-causal acoustic embedding encoded by a non-causal encoder. The operations also include encoding the first pass hypothesis generated by the RNN-T decoder using a text encoder, and generating a second pass hypothesis using a deliberative decoder that focuses on both the first pass hypothesis and the non-causal acoustic embedding.

[0008] This aspect may include one or more of the following optional features. In some implementations, the operation also includes receiving a sequence of acoustic frames, and encoding each acoustic frame in the sequence of acoustic frames into a corresponding causal acoustic embedding using a causal encoder. The operation also includes generating a non-causal acoustic embedding using a non-causal encoder configured to receive a causally encoded acoustic embedding as input, and decoding the non-causal acoustic embedding using an RNN-T decoder to generate a first pass hypothesis. In some examples, the deliberative decoder includes a long short-term memory (LSTM) network, followed by a softmax layer. Here, the LSTM network may include at least two layers. In some implementations, the text encoder includes a stack of self-attention blocks, each of which has a multi-head self-attention mechanism. In these implementations, each self-attention block may include one of a Conformer block or a Transformer block.

[0009] In some examples, training the deliberative model includes pre-training a text encoder on each non-verbal text utterance by tokenizing the corresponding non-verbal text utterance into a sequence of sub-word units. The example also includes replacing each tokenized sub-word unit in a first portion of the tokenized sub-word unit sequence with a masked token, and replacing each token sub-word unit in a second portion of the tokenized sub-word unit sequence with a random token. Additionally or alternatively, training the deliberative model includes generating a corresponding synthetic speech representation for each non-verbal text utterance of the received training data using a text-to-speech model, and training the deliberative model using the non-verbal text utterance and the corresponding synthetic speech representation.

[0010] In some implementations, the training data also includes untranscribed non-synthetic speech utterances, each untranscribed non-synthetic speech utterance is not paired with a corresponding transcription. In these implementations, the operation also includes predicting the corresponding transcription of each untranscribed non-synthetic speech utterance using the trained speech recognition model. Here, training the deliberative model also includes training a deliberative decoder using the untranscribed non-synthetic speech utterances and the predicted corresponding transcriptions as semi-supervised data. In some examples, generating a second-pass hypothesis includes generating a first context vector using a first attention mechanism focusing on the encoded first-pass hypothesis, generating a second context vector using a second attention mechanism focusing on the non-causal acoustic embedding, and decoding the first context vector and the second context vector at the deliberative decoder to form the second-pass hypothesis.

[0011] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1A and Figure 1B is a diagram of an example speech environment using a pre-trained deliberative two-pass architecture.

[0013] Figure 2 is a diagram of an example deliberative two-pass architecture for speech recognition.

[0014] FIG. 3A to FIG. 3C is a diagram of an example training process for promoting the deliberative model to learn consistent predictions for the deliberative two-pass architecture.

[0015] Figure 4 is a schematic diagram of an example Conformer block.

[0016] Figure 5 is a flow diagram of an example arrangement of operations for a method of performing speech recognition.

[0017] Figure 6 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0018] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION

[0019] Speech recognition continues to evolve to meet the unconstrained and flexible needs of mobile environments. New speech recognition architectures or improvements to existing architectures are constantly being developed, which aim to improve the quality of automatic speech recognition systems (ASR). For example, speech recognition initially employed multiple models, each of which had a specialized purpose. For example, an ASR system includes an acoustic model (AM), a pronunciation model (PM), and a language model (LM). The acoustic model maps audio segments (i.e., audio frames) to phonemes. The pronunciation model connects these phonemes together to form words, while the language model is used to express the likelihood of a given phrase (i.e., the probability of a sequence of words). Although these individual models work together, each model is trained independently and is usually manually designed on a different dataset.

[0020] The approach of a single model enables speech recognition systems to be quite accurate, especially when the training corpus (i.e., the training data body) of a given model satisfies the validity of the model. However, the need to train a single model independently introduces its own complexity and produces an architecture with an integrated model. These integrated models attempt to use a single neural network to directly map an audio waveform (i.e., an input sequence) to an output sentence (i.e., an output sequence). This produces a sequence-to-sequence method that generates a sequence of words (or graphemes) when a sequence of audio features is given. Examples of sequence-to-sequence models include "attention-based" models and "listen-attend-spell" (LAS) models. The LAS model transcribes speech utterances into characters using a listener component, an attender component, and a speller component. Here, the listener is a recurrent neural network (RNN) encoder that receives an audio input (e.g., a time-frequency representation of a speech input) and maps the audio input to a higher-level feature representation. The attender focuses on higher-level features to learn the alignment between the input features and the predicted sub-word units (e.g., graphemes or word fragments). The speller is an attention-based RNN decoder that generates a sequence of characters from an input by generating a probability distribution over a set of hypothetical words. Using the integrated structure, all components of the model can be jointly trained as a single end-to-end (E2E) neural network. Here, E2E models refer to models whose architecture is built entirely from neural networks. Fully neural networks operate without external and / or manually designed components (e.g., finite state transducers, dictionaries, or text normalization modules). In addition, when training E2E models, these models typically do not require bootstrapping from decision trees or temporal alignment from a separate system.

[0021] Although early E2E models have been shown to be accurate and have training improvements over individually trained models, these E2E models, such as the LAS model, operate by checking the entire input sequence before generating output text, and therefore do not allow streaming output as the input is received. Without streaming capabilities, the LAS model cannot perform real-time speech transcription. Due to this shortcoming, deploying the LAS model for voice applications that are sensitive to latency and / or require real-time speech transcription can be problematic. This makes the LAS model alone not an ideal model for mobile technologies (e.g., mobile phones) that typically rely on real-time applications (e.g., real-time communication applications).

[0022] In addition, a speech recognition system with an acoustic model, a pronunciation model, and a language model or such models combined together may rely on a decoder that must search a relatively large search graph associated with these models. Because the search graph is large, it is not conducive to hosting this type of speech recognition system completely on a device. Here, when the speech recognition system is hosted "on a device", the device receiving the audio input uses its processor to perform the functions of the speech recognition system. For example, when the speech recognition system is hosted completely on a device, the processor of the device does not need to coordinate with any computing resources outside the device to perform the functions of the speech recognition system. The device that does not perform speech recognition completely on the device relies on remote computing (for example, remote computing or cloud computing of a remote computing system), and therefore relies on an online connection to perform at least some functions of the speech recognition system. For example, a speech recognition system uses a network connection with a server-based model to perform decoding with a large search graph.

[0023] Unfortunately, reliance on remote connections makes speech recognition systems susceptible to latency issues and / or inherent unreliability of communication networks. In order to improve the practicality of speech recognition by avoiding these problems, speech recognition systems have evolved again into a form of sequence-to-sequence model called recurrent neural network transducer (RNN-T). RNN-T does not use an attention mechanism, and unlike other sequence-to-sequence models that usually need to process the entire sequence (e.g., audio waveform) to produce output (e.g., sentence), RNN-T continuously processes input samples and streams output symbols, a feature that is particularly attractive for real-time communication. For example, speech recognition using RNN-T can output the characters spoken one by one. Here, RNN-T uses a feedback loop that feeds back the symbols predicted by the model into itself to predict the next symbol. Because decoding RNN-T includes beam searching through a single neural network instead of a large decoder graph, RNN-T can scale to a fraction of the size of a server-based speech recognition model. As the size is reduced, RNN-T can be fully deployed on the device and can run offline (i.e., without a network connection), thus avoiding the problem of unreliable communication networks.

[0024] In addition to the speech recognition system operating with low latency, the speech recognition system also needs to recognize speech accurately. Typically for models that perform speech recognition, a metric that can define model accuracy is the word error rate (WER). WER refers to a measure of how many words have changed compared to the number of words actually spoken. Typically, these word changes refer to substitutions (i.e., when words are replaced), insertions (i.e., when words are added), and / or deletions (i.e., when words are omitted). For example, the speaker says "car", but the ASR system transcribes the word "car" as "bar". This is an example of substitution due to phonetic similarity. When measuring the capabilities of an ASR system relative to other ASR systems, WER can indicate some measure of improvement or quality capability relative to another system or a baseline.

[0025] Although the RNN-T model is expected to be a strong candidate model for speech recognition on the device, the RNN-T model alone still lags behind large-scale state-of-the-art traditional models (e.g., server-based models with separate AM, PM, and LM) in terms of quality (e.g., speech recognition accuracy). However, the non-streaming E2E LAS model has speech recognition quality comparable to large-scale state-of-the-art conventional models. In the two-pass model, the non-streaming LAS model, for example, re-scores the streaming hypothesis from the first pass. This second-pass LAS model method focuses on acoustic effects to re-score the hypothesis. In contrast, an alternative method called a neural correction model class uses text rather than acoustic effects to generate hypotheses. In other words, there are different variables that can be focused on so that the hypothesis is refined in the second pass. Therefore, the model proposed in this article is a variant of the RNN-T / LAS two-pass model. This variant uses a deliberative network that combines acoustic effects and first-pass text hypotheses for the second pass of the two-pass model.

[0026] To improve the quality of speech search, our implementation involves pre-training the second pass of the deliberative network using plain text data from a masked language model (e.g., FIG. 3A to FIG. 3C As shown). In addition to using plain text data, unlabeled audio utterances are used for semi-supervised training of the text encoder. By combining plain text data to pre-train the deliberative network, joint acoustic and text decoder training, and semi-supervised training in a single model, the pre-trained two-pass deliberative network can become more accurate than large conventional speech recognition models. For example, in some tests, the pre-trained two-pass deliberative network achieved a 4.1% improvement in voice search performance and nearly 12% long-tail WER reduction compared to an untrained two-pass deliberative network, and achieved an 8% relative WER reduction compared to a large conventional recognition model.

[0027] Figure 1Aand Figure 1B 1 is an example system 100a, 100b including a speech environment 100 in which a user 10 interacts with a computing device such as a user device 110 by voice input. The user device 110 (also generally referred to as the device 110) is configured to capture sounds (e.g., streaming audio data) from one or more users 10 within a speech-enabled environment. Here, the streaming audio data 12 may refer to the spoken utterances of the user 10, which serve as audible queries, commands to the device 110, or audible communications captured by the device 110. The speech-enabled system of the device 110 may process the query or command by answering the query and / or causing the command to be executed.

[0028] The user device 110 may correspond to any computing device associated with the user 10 and capable of receiving audio data 12. Some examples of the user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, Internet of Things (IoT) devices, smart speakers, etc. The user device 110 includes data processing hardware 112 and memory hardware 114 that communicates with the data processing hardware 112 and stores instructions that, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations. The user device 110 also includes an audio subsystem 116 having an audio capture device (e.g., a microphone) 116, 116a for capturing spoken utterances 12 within the speech-enabled environment 100 and converting the spoken utterances into electrical signals; and a speech output device (e.g., a speaker) 116, 116b for transmitting audible audio signals (e.g., as output audio data from the device 110). Although in the illustrated example, the user device 110 implements a single audio capture device 116a, the user device 110 may implement an array of audio capture devices 116a without departing from the scope of the present disclosure, wherein one or more capture devices 116a in the array may not physically reside on the user device 110, but may communicate with the audio subsystem 116. The user device 110 is also configured to perform speech recognition processing on the streaming audio data 12 using a speech recognizer 200. The speech recognizer 200 (also referred to as a model 200) resides on the user device 110 (hardware 110, 112) of the user 10, and / or resides on a remote computing device 60 (e.g., one or more remote servers of a distributed system executed in a cloud computing environment) that communicates with the user device 10 via a network 40. In some examples, the audio subsystem 116 of the user device 110, including the audio capture device 116a, is configured to receive the audio data 12 (e.g., spoken utterances) and convert the audio data 12 into a digital format compatible with the speech recognizer 200. The digital format may correspond to an acoustic frame such as a Mel frame (eg, a parameterized acoustic frame). For example, the parameterized acoustic frame corresponds to a log Mel filter bank energy.

[0029] In some examples, such as Figure 1A , the user 10 interacts with a program or application 118 of the user device 110 that uses the speech recognizer 200. For example, Figure 1AA user 10 is depicted communicating with an automated assistant application. In this example, the user 10 asks the automated assistant, "What time is the concert tonight?" This question from the user 10 is a spoken utterance 12 that is captured by an audio capture device 116a and processed by an audio subsystem 116 of a user device 110. In this example, a speech recognizer 200 of the user device 110 receives an audio input 202 (e.g., as an acoustic frame) "what time is the concert tonight" and transcribes the audio input 202 into a transcription 204 (e.g., a textual representation of "what time is the concert tonight?"). Here, the automated assistant of the application 118 can use natural language processing to respond to the question posed by the user 10. Natural language processing generally refers to the process of interpreting written language (e.g., transcription 204) and determining whether the written language prompts any action. In this example, the automated assistant uses natural language processing to recognize that the question from the user 10 is about the user's schedule, and more specifically, about a concert on the user's schedule. By utilizing natural language processing to recognize these details, the automated assistant returns a response to the user query, where the response states “Doors open at 8:30 pm for the concert tonight.” In some configurations, natural language processing may occur on a remote system in communication with the data processing hardware 112 of the user device 110 .

[0030] Figure 1B is another example of speech recognition using a speech recognizer 200. In this example, a user 10 associated with a user device 110 is communicating with a friend named Jane Doe using a communication application 118. Here, a user 10 named Ted is communicating with Jane by transcribing his voice inputs through the speech recognizer 200. The audio capture device 116 captures these voice inputs and transmits them to the speech recognizer 200 in digital form (e.g., acoustic frames). The speech recognizer 200 transcribes these acoustic frames into text, which is sent to Jane via the communication application 118. Because this type of application 118 communicates via text, the transcription 204 from the speech recognizer 200 can be sent to Jane without further processing (e.g., natural language processing).

[0031] In some examples, such as Figure 2, the speech recognizer 200 is configured in an enhanced two-pass architecture having a first pass 206 followed by a second pass 208. In general, the two-pass architecture of the speech recognizer 200 includes a first encoder 210 (e.g., a causal encoder 210), a second encoder 220 (e.g., a non-causal encoder 220), an RNN-T decoder 230, and a deliberative model 240. In two-pass decoding, the second pass 208 can utilize techniques such as trellis rescoring or n-best reordering to improve the initial output from the first pass 206. In other words, the RNN-T decoder 230 generates streaming predictions, and the deliberative model 240 finalizes the predictions. Here, specifically, the deliberative model 240 performs a streamed hypothesis 232 y from the RNN-T decoder 230. R Although it is generally discussed that the deliberative model 240 is in the hypothesis 232 y from the RNN-T decoder 230 R The deliberative model 240 operates in a rescoring mode in which rescoring is performed, but the deliberative model 240 can also operate in a different mode such as a beam search mode depending on the design or other factors (e.g., utterance length).

[0032] like Figure 2 As shown, the first pass 206 includes a first encoder 210 and a second encoder 220 arranged in cascade, which refers to a model structure in which the encoding path includes two encoders 210, 220, which are cascaded so that the output of one encoder 210 feeds the input of another encoder 220 before decoding. Here, the encoders 210, 220 can be cascaded regardless of the underlying architecture of each encoder. In some examples, the encoders 210, 220 include a stack of 512-dimensional conformer layers. Causal convolutions and left-context attention layers can be used for each conformer layer to strictly restrict the model from using future inputs. A multi-head (e.g., 8-head) attention mechanism can be used in the self-attention layer. The cascaded encoders 210, 220 can include 21 conformer layers. Here, the first encoder 210 may include 17 conformer layers, while the second encoder 220 may include four conformer layers that accept additional right contexts (e.g., 0.9 seconds). Alternatively, other types of layers such as transformer layers in combination with a self-attention mechanism may be used instead of the conformer layer. The first encoder 210 may be referred to as a causal encoder, and the second encoder 220 may be referred to as a non-causal encoder.

[0033] In other implementations, one encoder is constructed with an LSTM structure, while the other encoder is constructed using a bidirectional LSTM layer or a conformer layer (e.g., a conformer-transducer). In other words, the encoders 210, 220 may have different architectures or similar architectures. For example, the cascaded encoders 210, 220 may be roughly similar to the acoustic model (AM) in a traditional ASR system, and may include a recurrent network of stacked long short-term memory (LSTM) layers. Here, the first encoder 210 is a streaming encoder including a unidirectional long short-term memory (LSTM) layer, while the second encoder 220 is a non-streaming encoder including a bidirectional LSTM layer or a conformer layer. In a cascaded encoder in which both encoders 210, 220 include LSTM layers, the second encoder 220 receiving the output of the first encoder 210 may utilize the LSTM layer of the first encoder 210, so that the second encoder 220 includes fewer LSTM layers than the first encoder 210 (and fewer LSTM layers than a completely non-streaming model). By having fewer LSTM layers, the cascaded encoder can reduce the number of computationally more expensive bidirectional layers, making the speech recognizer 200 leaner than simply combining a traditional streaming model with a traditional non-streaming model.

[0034] The first encoder 210 reads a sequence x = (x1, x2, · · · , x T ), where x t ∈ ℝ d , and produces a first high-order feature representation as output 212 at each time step. The first high-order feature representation may include a causal acoustic embedding and is denoted as e s Similarly, the second encoder 220 is cascade-connected to the first encoder 210 and is trained to receive the first high-order feature e s as input, and produces a second high-order feature representation as output 222. The second high-order feature representation includes a non-causal acoustic embedding and is denoted as e a . Both the first encoder 210 and the second encoder 220 are directly connected to and shared by the RNN-T decoder 230. Therefore, the RNN-T decoder 230 receives the first high-level feature representation e s and the second high-order feature representation e a Then, the RNN-T decoder 230 converts the first high-level feature representation e s and the second high-order feature representation e a Decoded into the first hypothesis speech recognition result y r222. In some examples, each parameterized acoustic frame includes 128-dimensional log-Mel features calculated within a short shift window (e.g., 32 milliseconds and shifted every 10 milliseconds). Each feature can be stacked with a previous frame (e.g., the previous three frames) to form a higher dimensional vector (e.g., a 512-dimensional vector using the three previous frames). The features forming the vector can then be downsampled (e.g., to a frame rate of 30 milliseconds).

[0035] In some implementations, the RNN-T decoder 230 includes a joint layer and an embedding prediction network. Here, the prediction network may have: two LSTM layers, each with 2,048 hidden units and 640-dimensional projections; and an embedding layer of 128 units. The RNN-T decoder 230 uses a joint layer to combine the first high-order feature representation and the second high-order feature representation output by the encoders 210 and 220. s 、e a , and the embedding output y from the prediction network for the previous prediction r-1 ) to generate the first hypothesis y r Output 232. The decoder output 232 may be the current subword unit y i (Given a sequence of N previous non-whitespace symbols {y i−1 ,...,y i–N}) and the probability distribution P(y) on the input x i |y i−1 ,...,y0,x). In some examples, the joint network of the RNN-T decoder 230 includes 640 hidden units followed by a Softmax layer that predicts 4,096 mixed case word fragments. In some implementations, the Softmax layer is separate from the RNN-T decoder 230 and processes the output 232, y0, from the RNN-T decoder 230. r The output of the Softmax layer is then used in a beam search process to select orthogonal elements. In some implementations, the Softmax layer is integrated with the RNN-T decoder 230 so that the output 232 of the RNN-T decoder 230, y r Represents the output of the Softmax layer.

[0036] Continue to refer Figure 2 , the second pass 208 uses a deliberative model 240, which includes a text encoder 242 and two attention mechanisms 244, 246, a hypothesis attention mechanism 244 and an acoustic attention mechanism 246, in addition to a deliberative decoder 250 (also called LAS decoder 250). As described in more detail below (e.g., FIG. 3A to FIG. 3C), the text encoder 242 of the deliberation model 240 can be pre-trained using plain text (e.g., non-verbal text utterance) input. Here, the speech recognizer 200 focuses on the second high-order feature representation e of the second encoder 220 at the acoustic attention mechanism 246. a Output 222 to focus on the acoustic effect and focus on the first pass hypothesis y by focusing on the output 232 of the RNN-T decoder 230 at the hypothesis attention mechanism 244 r By focusing on the acoustic effect (e.g., denoted as e a The output 222) and the first pass hypothesis (e.g., denoted as y r The deliberative model 240 generates a second pass hypothesis as output 248 (e.g., a predicted sequence) based on the output 232 of the deliberative model 240. Here, each attention mechanism 244, 246 forms a context vector 245, 247 (e.g., a hypothesis context vector 245 and an acoustic context vector 247, or a first context vector 245 and a second context vector 247), which is input to the deliberative decoder 250 of the deliberative model 240. These context vectors 245, 247 can be concatenated as inputs to the deliberative decoder 250.

[0037] The text encoder 242 further encodes the output 232 of the RNN-T decoder 230 (i.e., the output 232 of the first pass 206) to form an encoded hypothesis 243 (e.g., shown as h B ). When the output 232 is further encoded, the text encoder 242 may also encode the output 232 to obtain useful context information to include in the encoded hypothesis 243. For example, the text encoder 242 is a bidirectional encoder capable of including context information. The text encoder 242 may also be configured to encode multiple first hypotheses (i.e., the output 232). For example, the text encoder 242 encodes each hypothesis 232 separately and then concatenates each encoded hypothesis together. The text encoder 242 may include a stack of multi-headed attention blocks 400 (referred to herein as conformer blocks 400), which may include a conformer or a transformer. Each multi-headed attention block 400 may include a multi-headed attention mechanism 420 ( Figure 4 ). For example, the text encoder 242 can be a two-layer conformer encoder, where each layer has a 640-dimensional projection per layer with a multi-token (eg, two-token) right context.

[0038] During the second pass 208, the speech recognizer 200 may perform a beam search mode or a re-scoring mode to generate outputs 248 (i.e., second pass hypotheses). In the re-scoring mode, the deliberative model 240 may run on the outputs 232 in a forced teaching mode. Additionally or alternatively, when in the re-scoring mode, using a bidirectional text encoder 242 may help improve the relative WER of the deliberative decoder two-pass architecture of the deliberative model 240. When the deliberative decoder 250 operates in the beam search mode, the deliberative decoder 250 generates only the second pass hypotheses from the outputs 222 as outputs 248, while ignoring the outputs 232 of the RNN-T decoder 230. When the deliberative decoder 250 operates in the re-scoring mode, the deliberative decoder 250 obtains the top K hypotheses (e.g., 4 first pass hypotheses) from the RNN-T decoder 230, and then the deliberative decoder 250 runs on each sequence in a forced teaching mode (it focuses on the outputs 222) to calculate a score. For example, the score combines the log probability of the sequence with an attention coverage penalty. The deliberative decoder 250 selects the sequence with the highest score as the output 222. Here, the deliberative decoder 250 may include multi-head attention (e.g., with four heads) to focus on the output 222. In addition, the deliberative decoder 250 may be a two-layer LSTM network followed by a softmax layer for prediction. For example, each layer of the deliberative decoder 250 has 2,048 hidden units followed by a 640-dimensional projection. The softmax layer may include 4,096 dimensions to predict the same mixed case word fragments from the softmax layer of the RNN-T decoder 230. Much like the attention mechanism inherent to the deliberative decoder 250 described above, the attention mechanisms 244, 246 may have a similar structure such that each attention mechanism 244, 246 includes multi-head attention (e.g., eight heads).

[0039] Neural networks are typically trained by back propagation that defines a loss function (e.g., a cross entropy loss function). For example, a loss function is defined as the difference between the actual output of the network and the expected output of the network. Here, a cross entropy loss method, a joint training method, or a combination of a cross entropy loss and joint training can be used to train the speech recognizer 200. In the cross entropy loss method, a deliberative model such as a speech recognizer 200 with a deliberative model 240 (i.e., a deliberative-based speech recognizer 200) is trained in a two-step training process. During the first step of the training process, the RNN-T decoder 230 is trained. After the RNN-T decoder 220 has been trained, the parameters of the RNN-T decoder 230 are fixed, and only the deliberative model 240 and the additional encoder layer (e.g., the text encoder 242) are trained.

[0040] FIG. 3A to FIG. 3C2 shows an example training process 300a-300c for training the deliberative model 240 of the speech recognizer 200. In some configurations, the training process 300a-300c is performed in Figure 1A and Figure 1B The training process 300a–c obtains a set of training data 320, 320a–n stored in a sample database 310 and trains the deliberative model 240 on the training data 320. The training data 320 includes a plurality of training non-verbal text utterances 330, 330a-n. Here, each training non-verbal text utterance 330 is not paired with any corresponding spoken utterance of non-synthesized speech. The sample database 310 may reside on the memory hardware of the remote computing device 60. In the example shown, the training data 320 is selected to train the deliberative model 240, which includes a text encoder 242 and a deliberative decoder 250. Here, the deliberative model 240 receives the training data 320 and generates an output whose accuracy has been tested. Although FIG. 3A to FIG. 3C Individual training processes 300a-c are depicted, but it should be understood that the deliberative decoder 250 may be trained by any combination of the training processes 300a-c.

[0041] refer to Figure 3A, the training process 300a trains the deliberative model 240 by pre-training the text encoder 242 on each training non-verbal text utterance 330 in the training data 320. Here, the training process 300a includes a masking module 340, which processes each training non-verbal text utterance 330 before training the text encoder 242 using a cross entropy loss. The masking module 340 includes a word unit module 342 and a masker 346. Intuitively, since the text encoder 242 is bidirectional (i.e., it has both right and left contexts), it can easily predict the target word in the training sample. In order to train the text encoder 242, a percentage of each training non-verbal text utterance 330 is randomly masked. The word unit module 342 obtains each corresponding training non-verbal text utterance 330 and tokenizes the corresponding training non-verbal text utterance 330 into a sequence of tokenized sub-word units 344, 344a-n. The masker 346 receives a sequence of lemmatized sub-word units 344 and randomly selects a percentage (e.g., 15%) of the sequence of lemmatized sub-word units 344 to replace with different word units. For example, the masker 346 replaces each lemmatized sub-word unit 344 in a first portion of the sequence of lemmatized sub-word units 344 with a masked word unit 344M, and replaces each lemmatized sub-word unit 344 in a second portion of the sequence of lemmatized sub-word units 344 with a random word unit 344M. Specifically, the masker 346 replaces each lemmatized sub-word unit 344 in the sequence of lemmatized sub-word units 344 with a masked word unit 344M 80% of the time, replaces it with a random word unit 344R 10% of the time, and leaves the lemmatized sub-word unit 344 unchanged 10% of the time. For example, as Figure 3A As shown, the masking module 340 obtains the training non-verbal text utterance 330a and tokenizes (i.e., using the token module 340) the training non-verbal text utterance 330a into a sequence of sub-word units 344 to produce four tokenized sub-word units 344. The masker 346 receives the four tokenized sub-word units 344 and outputs two unchanged tokenized sub-word units 344, a masked token 344M, and a random token 344R. Once the training process 300a is completed, the parameters of the text encoder 242 and the deliberative decoder 250 are jointly updated in additional training, while the parameters of the RNN-T decoder 230 remain fixed.

[0042] refer to Figure 3B, the training process 300b uses the training non-verbal text utterances 330 in the training data 320 to train the deliberative model 240. Here, the training process 300b includes a text-to-speech module 350 that processes each training non-verbal text utterance 330 before training the deliberative model 340. Specifically, the text-to-speech module 350 obtains each training non-verbal text utterance 330 and generates a corresponding synthetic speech representation 352 using a text-to-speech model. Then, the training process 300b uses the training non-verbal text utterances 330 and the corresponding synthetic speech representations 352 to train the deliberative model 240. For example, Figure 3B As shown, the text-to-speech module 350 receives the training non-verbal text utterance 330a and generates a corresponding synthesized speech representation 352 as an output using a text-to-speech model. The training process 300b then uses the training non-verbal text utterance 330a and the corresponding synthesized speech representation 352 to jointly train the deliberative model 240. The training process 300b can calculate both audio and text attention when using the training non-verbal text utterance 330a and the fixed context vector instead of the attention when using the corresponding synthesized speech representation 352. The training process 300b can select a mixture of the training non-verbal text utterance 330 and the corresponding synthesized speech representation 352 when training the deliberative model 240 (e.g., a ratio of 1:9).

[0043] refer to Figure 3C , the training process 300c uses the training data 320 to train the deliberative model 240. Here, the training data 320 also includes multiple training untranscribed non-synthetic speech utterances 332, 332a-n. Here, each training untranscribed non-synthetic speech utterance 332 is not paired with a corresponding transcription. The training process 300c can use a trained speech recognition model 360 (also known as an ASR model 360), which is trained to predict the corresponding transcription of an input utterance as an output based on the input utterance. Specifically, the ASR model 360 obtains each training untranscribed non-synthetic speech utterance 332 and generates a corresponding transcription 362. The training process 300c then trains the deliberative model 240 by training the deliberative decoder 250 using the training untranscribed non-synthetic speech utterances 332 and the corresponding transcriptions 362. For example, as Figure 3C As shown, the ASR model 360 receives the training untranscribed non-synthetic speech utterance 332a and generates the corresponding transcription 362 as output using the ASR model 360. The training process 300c then trains the deliberative decoder 250 using the training untranscribed non-synthetic speech utterance 332 and the predicted corresponding transcription 362 as semi-supervised data.

[0044] Figure 4An example of a stacked Conformer block 400 of Conformer layers from a text encoder 242 is provided. The Conformer block 400 includes a first half feed-forward layer 410, a second half feed-forward layer 440, and a concatenation operator 405, wherein a multi-head self-attention block 420 and a convolution layer 430 are disposed between the first half feed-forward layer 410 and the second half feed-forward layer 440. The first half feed-forward layer 410 processes an input hypothesis (e.g., the output 232 from the RNN-T decoder 230). Subsequently, the multi-head self-attention block 420 receives the input hypothesis concatenated with the output of the first half feed-forward layer 410. Intuitively, the role of the multi-head self-attention block 420 is to summarize the context to be enhanced separately for each input frame. The convolution layer 430 subsamples the output of the multi-head self-attention block 420 concatenated with the output of the first half feed-forward layer 410. Thereafter, the second half feed-forward layer 440 receives the concatenation of the convolution layer 430 output with the multi-head self-attention block 420. The layer normalization (layernorm) module 450 processes the output from the second half of the feed-forward layer 440. Mathematically, the conformer block 400 transforms the input feature x using the modulation feature m to produce the output feature y as follows:

[0045] Figure 5 1 is a flow chart of an example arrangement of operations of a method 500 for performing automated speech recognition (e.g., ASR) using a deliberative two-pass architecture. At operation 502, the method 500 receives training data 320 including non-verbal text utterances 330. Here, each non-verbal text utterance 330 is not paired with any corresponding spoken utterance of non-synthesized speech. At operation 504, the method 500 includes training a deliberative model 240 on the non-verbal text utterances 330. The deliberative model 240 includes a text encoder 242 and a deliberative decoder 250.

[0046] At operation 506, the method 500 includes receiving the first-pass hypothesis 232 and the non-causal acoustic embedding 222 at the trained deliberative model 240. The first-pass hypothesis 232 is generated by the recurrent neural network transducer (RNN-T) decoder 230 for the non-causal acoustic embedding 222 encoded by the non-causal encoder 220. The method 500 also includes encoding the first-pass hypothesis 232 generated by the RNN-T decoder 230 using the text encoder 242 at operation 508. At operation 510, the method 500 also includes generating a second-pass hypothesis 248 using the deliberative decoder 250 that focuses on both the first-pass hypothesis 232 and the non-causal acoustic embedding 222.

[0047] Figure 66 is a schematic diagram of an example computing device 600 that can be used to implement the systems and methods described in this document. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only, and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0048] The computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 connected to the memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 connected to a low-speed bus 670 and the storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 610 (e.g., the data processing hardware 112 of FIG. 1 or the data processing hardware of the remote computing device 60) can process instructions for execution within the computing device 600, including instructions stored in the memory 620 or stored on the storage device 630, to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 680 coupled to the high-speed interface 640. In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. Furthermore, multiple computing devices 600 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0049] Memory 620 (e.g., memory hardware 114 of FIG. 1 or memory hardware of a remote computing device 60) stores information non-temporarily within computing device 600. Memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporary memory 620 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0050] The storage device 630 is capable of providing mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various implementations, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configurations). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0051] The high-speed controller 640 manages bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 640 is coupled to a memory 620, a display 680 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to a storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, for example, through a network adapter.

[0052] The computing device 600 can be implemented in a variety of different forms, as shown. For example, the computing device can be implemented as a standard server 600a or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0053] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be special purpose or general purpose and can be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0054] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0055] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0056] The processes and logic flows described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0057] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor) or a touch screen for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0058] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (500) which, when executed on data processing hardware (610), causes the data processing hardware (610) to perform operations comprising: receiving training data (320) comprising non-verbal text utterances (330), each non-verbal text utterance (330) not being paired with any corresponding spoken utterance of non-synthesized speech; training a deliberative model (240) on the non-verbal text utterance (330), the deliberative model (240) comprising a text encoder (242) and a deliberative decoder (250); Receiving, at the trained deliberative model (240), a first-pass hypothesis (232) and a non-causal acoustic embedding (222), the first-pass hypothesis (232) being generated by a recurrent neural network transducer RNN-T decoder (230) for the non-causal acoustic embedding (222) encoded by a non-causal encoder (220); encoding the first pass hypothesis (232) generated by the RNN-T decoder (230) using the text encoder (242); as well as A second pass hypothesis (248) is generated using the deliberative decoder (250) that focuses on both the first pass hypothesis (232) and the non-causal acoustic embedding (222).

2. The computer-implemented method (500) of claim 1, wherein the operations further comprise: receiving a sequence of acoustic frames (202); encoding each acoustic frame (202) in the sequence of acoustic frames (202) into a corresponding causal acoustic embedding (212) using a causal encoder (210); generating the non-causal acoustic embedding (222) using the non-causal encoder (220) configured to receive the encoded causal acoustic embedding (212) as input; as well as The non-causal acoustic embedding (222) is decoded using the RNN-T decoder (230) to generate the first-pass hypothesis (232).

3. A computer-implemented method (500) as described in claim 1 or 2, wherein the deliberative decoder (250) includes a long short-term memory (LSTM) network followed by a softmax layer.

4. The computer-implemented method (500) of claim 3, wherein the LSTM network comprises at least two layers.

5. The computer-implemented method (500) of any one of claims 1 to 3, wherein the text encoder (242) comprises a stack of self-attention blocks (400) each having a multi-head self-attention mechanism.

6. The computer-implemented method (500) of claim 5, wherein each self-attention block (400) comprises one of a Conformer block or a Transformer block.

7. The computer-implemented method (500) of any one of claims 1 to 6, wherein training the deliberative model (240) comprises pre-training the text encoder (242) on each non-verbal text utterance (330) by: lemmatizing the corresponding non-verbal text utterance (330) into a sequence of sub-word units (344); replacing each lemma-ized subword unit (344) in a first portion of the sequence of lemma-ized subword units (344) with a mask lemma (344M); and Each lemma subword unit (344) in a second portion of the sequence of lemma subword units (344) is replaced with a random lemma (344R).

8. The computer-implemented method (500) of any one of claims 1 to 7, wherein training the deliberative model (240) comprises: generating, using a text-to-speech model (350), a corresponding synthesized speech representation (352) for each non-verbal text utterance (330) of the received training data (320); and The deliberation model (240) is trained using the non-verbal text utterance (330) and the corresponding synthesized speech representation (352).

9. The computer-implemented method (500) of any one of claims 1 to 8, wherein: The training data (320) also includes untranscribed non-synthesized speech utterances (332), each untranscribed non-synthesized speech utterance (332) not being paired with a corresponding transcription; and The operations also include: predicting the corresponding transcription (362) for each untranscribed non-synthesized speech utterance (332) using the trained speech recognition model (360); and Training the deliberative model (240) also includes using the untranscribed non-synthetic speech utterances (332) and the predicted corresponding transcriptions (362) as semi-supervised data to train the deliberative decoder (250).

10. The computer-implemented method (500) of any one of claims 1 to 9, wherein generating the second-pass hypothesis (248) comprises: generating a first context vector (245) using a first attention mechanism (244) focusing on the encoded first-pass hypothesis (232); generating a second context vector (247) using a second attention mechanism (246) focusing on the non-causal acoustic embedding (222); as well as The first context vector (245) and the second context vector (247) are decoded at the deliberative decoder (250) to form the second pass hypothesis (248).

11. A system (100), comprising: Data processing hardware (610); and Memory hardware (620) storing instructions that, when executed on the data processing hardware (610), cause the data processing hardware (610) to perform operations, the operations comprising: receiving training data (320) comprising non-verbal text utterances (330), each non-verbal text utterance (330) not being paired with any corresponding spoken utterance of non-synthesized speech; training a deliberative model (240) on the non-verbal text utterance (330), the deliberative model (240) comprising a text encoder (242) and a deliberative decoder (250); Receiving, at the trained deliberative model (240), a first-pass hypothesis (232) and a non-causal acoustic embedding (222), the first-pass hypothesis (232) being generated by a recurrent neural network transducer RNN-T decoder (230) for the non-causal acoustic embedding (222) encoded by a non-causal encoder (220); encoding the first pass hypothesis (232) generated by the RNN-T decoder (230) using the text encoder (242); as well as A second pass hypothesis (248) is generated using the deliberative decoder (250) that focuses on both the first pass hypothesis (232) and the non-causal acoustic embedding (222).

12. The system (100) of claim 11, wherein the operations further comprise: receiving a sequence of acoustic frames (202); encoding each acoustic frame (202) in the sequence of acoustic frames (202) into a corresponding causal acoustic embedding (212) using a causal encoder (210); generating the non-causal acoustic embedding (222) using the non-causal encoder (220) configured to receive the encoded causal acoustic embedding (212) as input; as well as The non-causal acoustic embedding (222) is decoded using the RNN-T decoder (230) to generate the first-pass hypothesis (232).

13. The system (100) of claim 11 or 12, wherein the deliberative decoder (250) comprises a Long Short Term Memory (LSTM) network followed by a softmax layer.

14. The system (100) of claim 13, wherein the LSTM network comprises at least two layers.

15. The system (100) of any one of claims 11 to 14, wherein the text encoder (242) comprises a stack of self-attention blocks (400) each having a multi-head attention mechanism.

16. The system (100) of claim 15, wherein each self-attention block (400) comprises one of a Conformer block or a Transformer block.

17. The system (100) of any one of claims 11 to 16, wherein training the deliberation model (240) comprises pre-training the text encoder (242) on each non-verbal text utterance (330) by: lemmatizing the corresponding non-verbal text utterance (330) into a sequence of sub-word units (344); replacing each lemma-ized subword unit (344) in a first portion of the sequence of lemma-ized subword units (344) with a mask lemma (344M); and Each lemma subword unit (344) in a second portion of the sequence of lemma subword units (344) is replaced with a random lemma (344R).

18. The system (100) of any one of claims 11 to 17, wherein training the deliberation model (240) comprises: generating, using a text-to-speech model (350), a corresponding synthesized speech representation (352) for each non-verbal text utterance (330) of the received training data (320); and The deliberation model (240) is trained using the non-verbal text utterance (330) and the corresponding synthesized speech representation (352).

19. The system (100) of any one of claims 11 to 18, wherein: The training data (320) also includes untranscribed non-synthesized speech utterances (332), each untranscribed non-synthesized speech utterance (332) not being paired with a corresponding transcription; and The operations also include: predicting the corresponding transcription (362) for each untranscribed non-synthesized speech utterance (332) using the trained speech recognition model (360); and Training the deliberative model (240) also includes using the untranscribed non-synthetic speech utterances (332) and the predicted corresponding transcriptions (336) as semi-supervised data to train the deliberative decoder (250).

20. The system (100) of any one of claims 11 to 19, wherein generating the second pass hypothesis (248) comprises: generating a first context vector (245) using a first attention mechanism (244) focusing on the encoded first-pass hypothesis (232); generating a second context vector (247) using a second attention mechanism (246) focusing on the non-causal acoustic embedding (222); as well as The first context vector (245) and the second context vector (247) are decoded at the deliberative decoder (250) to form the second pass hypothesis (248).

Citation Information

Patent Citations

  • Two-pass end to end speech recognition

    CN114097025A

  • Acoustic model learning device, voice recognition device, and computer program for acoustic model learning

    JP2010243914A