Attention-Based Joint Acoustic and Text Modeling on-Device End-to-End

Abstract

To provide an attention-based joint acoustic and a text on-device end-to-end model.SOLUTION: A method (300)includes: receiving a training example (203) for a listen-attend-spell (LAS) decoder (230) of a two-pass streaming neural network model (200); and determining whether the training example corresponds to a supervised audio-text pair or an unpaired text sequence. The method also includes determining a cross entropy loss based on a log probability associated with a language context vector (246) of the training example when the training example corresponds to the unpaired text sequence. The method also includes updating the LAS decoder and the language context vector based on the determined cross entropy loss.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] This disclosure relates to two-pass end-to-end speech recognition. [Background technology]

[0002] Modern automated speech recognition (ASR) systems focus on providing not only high quality (e.g., low word error rate (WER)) but also low latency (e.g., short delay between when a user speaks and when a transcription appears). Furthermore, current use of ASR systems requires that the ASR system decodes speech in a streaming manner that is compatible with real time or even faster than real time. For example, if the ASR system is installed on a mobile phone that directly interacts with the user, an application on the mobile phone that uses the ASR system may require that speech recognition be streamed so that it appears on the screen as soon as the word is spoken. Here, mobile phone users may also have a low tolerance for delay. Due to this low tolerance, speech recognition aims to operate on mobile devices in a way that minimizes the impact of delays and inaccuracies that can negatively affect the user experience. Summary of the Invention

[0003] One aspect of the disclosure provides a computer-implemented method that, when executed on the data processing hardware, causes the data processing hardware to perform operations including receiving training examples for a listen-attend-spell (LAS) decoder of a two-pass streaming neural network model and determining whether the training examples correspond to supervised speech-text pairs or to unpaired text sequences. If the training examples correspond to unpaired text sequences, the operations also include determining a cross entropy loss based on a log probability associated with a context vector of the training examples and updating the LAS decoder and the context vector based on the determined cross entropy loss.

[0004] Embodiments of the present disclosure may include one or more of any of the following features. In some embodiments, the operations also include receiving a second training example for a LAS decoder of the two-pass streaming neural network, determining that the second training example corresponds to a supervised speech-text pair, and updating acoustic context vector parameters associated with the LAS decoder and the acoustic context vector based on a log-probability for the acoustic context vector. In some examples, determining whether the training example corresponds to a supervised speech-text pair or an unpaired text sequence includes identifying a domain identifier indicating whether the training example corresponds to a supervised speech-text pair or an unpaired text sequence.

[0005] In an additional embodiment, the LAS decoder is updated to reduce a word error rate (WER) of a two-pass streaming neural network model for long-tail entities. The log probability can be defined by an interpolation of a first individual log probability generated from an acoustic context vector and a second individual log probability generated from a text context vector. Furthermore, the LAS decoder can operate in a beam search mode based on hypotheses generated by a recurrent neural network transducer (RNN-T) decoder through a first pass of the two-pass streaming neural network model. In some examples, the operations also include generating context vectors for training examples using an attention mechanism configured to summarize encoder features from encoded acoustic frames.

[0006] Another aspect of the disclosure provides a computer-implemented method that, when executed on the data processing hardware, causes the data processing hardware to perform operations including receiving training examples for a listen-attend-spell (LAS) decoder of a two-pass streaming neural network model; determining whether the training examples correspond to supervised speech-text pairs or to unpaired training data; if the training examples correspond to the unpaired training data, generating missing portions of the unpaired training data to form a generated speech-text pair; and updating context vectors associated with the LAS decoder and the unpaired data based on the generated speech-text pair.

[0007] This aspect may include one or more of any of the following features. In some embodiments, the operations also include determining an acoustic context vector based on the generated speech-text pair, and determining an interpolation of a first individual log-probability generated from the acoustic context vector and a second individual log-probability generated from the text context vector. In these embodiments, updating the LAS decoder is further based on an interpolation of the first individual log-probability and the second individual log-probability.

[0008] In some examples, determining whether the training examples correspond to supervised speech-text pairs or unpaired training data includes identifying a domain identifier indicating whether the training examples correspond to supervised speech-text pairs or unpaired training data. Further, updating the LAS decoder may reduce a word error rate (WER) of the two-pass streaming neural network model for long-tail entities. In some embodiments, the operations also include generating a context vector for the training examples using an attention mechanism configured to summarize encoder features from the encoded acoustic frames.

[0009] Yet another aspect of the present disclosure discloses a system comprising data processing hardware and memory hardware in communication with the data processing hardware and storing instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including receiving training examples for a listen-attend-spell (LAS) decoder of a two-pass streaming neural network model, determining whether the training examples correspond to supervised speech-text pairs or to unpaired text sequences, and if the training examples correspond to unpaired text sequences, determining a cross-entropy loss based on a log-probability associated with a context vector of the training examples, and updating the LAS decoder and the context vector based on the determined cross-entropy loss.

[0010] This aspect may include one or more of any of the following features. In some embodiments, the operations also include receiving a second training example of a two-pass streaming neural network LAS decoder, determining that the second training example corresponds to a supervised speech-text pair, and updating acoustic context vector parameters associated with the LAS decoder and the acoustic context vector based on a log-probability of the acoustic context vector. In some examples, determining whether the training example corresponds to a supervised speech-text pair or an unpaired text sequence includes identifying a domain identifier indicating whether the training example corresponds to a supervised speech-text pair or an unpaired text sequence.

[0011] In an additional embodiment, the LAS decoder is updated to reduce the word error rate (WER) of the two-pass streaming neural network model for long-tail entities. The log-probabilities can be defined by an interpolation of a first individual log-probability generated from an acoustic context vector and a second individual log-probability generated from a text context vector. Furthermore, the LAS decoder can operate in a beam search mode based on hypotheses generated by a recurrent neural network transducer (RNN-T) decoder through a first pass of the two-pass streaming neural network model. In some examples, the operations also include generating context vectors for training examples using an attention mechanism configured to summarize encoder features from encoded acoustic frames.

[0012] Yet another aspect of the present disclosure provides a system comprising data processing hardware and memory hardware in communication with the data processing hardware and storing instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including receiving training examples for a Listen-Attend-Spell (LAS) decoder of a two-pass streaming neural network model, determining whether the training examples correspond to supervised speech-text pairs or to unpaired training data, if the training examples correspond to the unpaired training data, generating missing portions of the unpaired training data to form a generated speech-text pair, and updating context vectors associated with the LAS decoder and the unpaired data based on the generated speech-text pair.

[0013] This aspect may include one or more of any of the following features. In some embodiments, the operations also include determining an acoustic context vector based on the generated speech-text pair, and determining an interpolation of a first individual log-probability generated from the acoustic context vector and a second individual log-probability generated from the text context vector. In these embodiments, updating the LAS decoder is further based on an interpolation of the first individual log-probability and the second individual log-probability.

[0014] In some examples, determining whether the training examples correspond to supervised speech-text pairs or unpaired training data includes identifying a domain identifier indicating whether the training examples correspond to supervised speech-text pairs or unpaired training data. Further, updating the LAS decoder may reduce a word error rate (WER) of the two-pass streaming neural network model for long-tail entities. In some embodiments, the operations also include generating a context vector for the training examples using an attention mechanism configured to summarize encoder features from the encoded acoustic frames.

[0015] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the detailed description below. Other aspects, features, and advantages will become apparent from the detailed description and drawings, and from the claims. [Brief description of the drawings]

[0016] [Figure 1A] FIG. 1 is a schematic diagram of an exemplary speaking environment using a two-pass architecture with joint acoustic and text models. [Figure 1B] FIG. 1 is a schematic diagram of an exemplary speaking environment using a two-pass architecture with joint acoustic and text models. [Figure 2A] FIG. 1 is a schematic diagram of an exemplary conventional two-pass architecture for speech recognition. [Figure 2B] FIG. 2B is a schematic diagram of an exemplary joint acoustic and text model incorporated into the two-pass architecture of FIG. 2A for speech recognition. [Diagram 3] 2C is a flowchart of an example configuration for operation of a method for implementing the two-pass architecture of FIG. 2B incorporating joint acoustic and text models. [Figure 4] 2C is a flowchart of an example configuration for operation of a method for implementing the two-pass architecture of FIG. 2B incorporating joint acoustic and text models. [Diagram 5] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Like reference symbols in the various drawings indicate like elements. Speech recognition continues to evolve to meet the demands of unconstrainedness and agility in the mobile environment. New speech recognition architectures or improvements on existing architectures continue to be developed to improve the quality of automatic speech recognition systems (ASR). For example, speech recognition initially employed multiple models, with each model having a dedicated purpose. For example, ASR systems included an acoustic model (AM), a pronunciation model (PM), and a language model (LM). The acoustic model mapped segments of speech (i.e., frames of speech) to phonemes. The pronunciation model strung these phonemes together to form words, and the language model was used to represent the likelihood of a given phrase (i.e., the probability of a sequence of words). Although these individual models worked in tandem, each model was trained separately and often manually designed on a different dataset.

[0018] Although separate model approaches have allowed for considerable improvements in the accuracy of speech recognition systems, especially when the training corpus (i.e., the collection of training data) for a given model corresponds to the effectiveness of the model, the need to train each model separately brings its own complexity, leading to the adoption of architectures with integrated models. These integrated models attempt to directly map speech waveforms (i.e., input sequences) to output sentences (i.e., output sequences) using a single neural network. This resulted in a sequence-to-sequence approach in which, given any sequence of speech features, a sequence of words (or graphemes) is generated. Examples of sequence-to-sequence models include the "attention-based" model and the "listen-attend-spell" (LAS) model. The LAS model converts spoken utterances into text using a listener, an attendant, and a speller component. Here, the listener is a recurrent neural network (RNN) encoder that receives speech input (e.g., a time-frequency representation of speech input) and maps the speech input to a higher-level feature representation. The attender directs attention to the higher-level features to learn alignments between input features and predicted sub-word units (e.g., graphemes or word pieces). The speller is an attention-based RNN decoder that generates character sequences from the input by generating probability distributions over a set of hypothesis words. The unified structure allows all components of the model to be jointly trained as a single end-to-end (E2E) neural network. Here, an E2E model refers to a model whose architecture is composed entirely of neural networks.A fully neural network functions without any external and / or manually designed components (e.g., finite state transducers, lexicons, or text normalization modules). Furthermore, when training E2E models, these models typically do not require bootstrapping from decision trees or time tuning from another system.

[0019] While early E2E models were accurate and provided training improvements over models trained separately, these E2E models, such as the LAS model, worked by reviewing the entire input sequence before generating output text, and therefore could not stream output as the input was received. Without streaming capabilities, LAS models cannot perform real-time voice transcription. This deficiency can cause problems when used with voice applications that are delay-sensitive and / or require real-time voice transcription. This makes the LAS model, by itself, a less than ideal model for mobile technologies (e.g., mobile phones) that often rely on real-time applications (e.g., real-time communication applications).

[0020] Furthermore, a speech recognition system having an acoustic model, a pronunciation model, and a language model, or models that are composed together, may rely on a decoder that needs to search a relatively large search graph associated with these models. A large search graph makes it unfavorable to host this type of speech recognition system entirely on-device. Here, when a speech recognition system is hosted "on-device", the device that receives the speech input uses its processor(s) to perform the functions of the speech recognition system. For example, when a speech recognition system is hosted entirely on-device, the device's processor does not need to cooperate with computing resources outside the device to perform the functions of the speech recognition system. A device that performs speech recognition that is not entirely on-device relies on remote computing (e.g., a remote computing system or cloud computing), and thus an online connection, to perform at least some functions of the speech recognition system. For example, the speech recognition system uses a network connection with a server-based model to perform decoding through a large search graph.

[0021] Unfortunately, the reliance on remote connections makes speech recognition systems vulnerable to latency issues and / or unreliability inherent in communication networks. To improve the usefulness of speech recognition by avoiding these issues, speech recognition systems have again evolved into a form of sequence-to-sequence model known as recurrent neural network transducer (RNN-T). Unlike other sequence-to-sequence models that do not employ attention mechanisms and must process entire sequences (e.g., speech waveforms) to generate an output (e.g., a sentence), RNN-T processes input samples continuously and streams output symbols, a feature that makes it particularly attractive for real-time communication. For example, speech recognition using RNN-T may output letters one by one as they are spoken. Here, RNN-T predicts the next symbol using a feedback loop that feeds back the symbols predicted by the model to itself. Because RNN-T decoding involves a beam search through a single neural network rather than a large decoder graph, RNN-T can scale to a fraction of the size of server-based speech recognition models. The reduced size allows RNN-T to be fully on-device and operate offline (i.e., without a network connection), thus avoiding network reliability issues.

[0022] In addition to operating with low latency, speech recognition systems are required to accurately recognize speech. For models that perform speech recognition, the word error rate (WER) is often used as a metric to define the accuracy of the model. WER is a measure of how many words have been changed compared to the number of words actually spoken. Generally, these word changes refer to substitutions (i.e., when words are replaced), insertions (i.e., when words are added), and / or deletions (i.e., when words are omitted). For example, a speaker may say "car," but the ASR system may transcribe the word "car" as "bar." This is an example of substitution due to phonetic similarity. When measuring the capabilities of an ASR system compared to other ASR systems, WER can indicate a measure of improvement or quality assurance capabilities compared to another system or a baseline.

[0023] Although the RNN-T model has shown promise as a strong candidate model for on-device speech recognition, the RNN-T model alone lags behind large-scale state-of-the-art conventional models (e.g., server-based models with separate AM, PM, and LM) in terms of quality (e.g., speech recognition accuracy). However, the non-streaming E2E,LAS model has speech recognition quality comparable to large-scale state-of-the-art conventional models. To leverage the quality of the non-streaming E2E LAS model, a two-pass speech recognition system (e.g., as shown in Figure 2A) was developed that includes a first-pass component of the RNN-T network followed by a second-pass component of the LAS network. With this design, the two-pass model benefits from the streaming nature of the low-latency RNN-T model and improves the accuracy of the RNN-T model through the second pass incorporating the LAS network. Although the LAS network increases the latency compared to the RNN-T model alone, the latency increase is reasonably small and meets the latency constraints for on-device operation. In terms of accuracy, the two-pass model achieves a 17-22% WER reduction when compared to RNN-T alone, and has a similar WER when compared to large-scale conventional models.

[0024] Unfortunately, this two-pass model with a first pass of the RNN-T network and a second pass of the LAS network has some drawbacks. For example, this kind of two-pass model has the problem that it is trained only on supervised speech-text pairs. Training only on supervised speech-text pairs causes the two-pass model to perform poorly on rare words or long tail entities. Long tail entities refer to a large number of entities that have relatively few (i.e., low frequency) instances in the text corpus. In other words, uncommon and rare words are therefore inherently uncommon even in small training datasets. When training a streaming E2E model such as a two-pass model, the training is done using a training set for a traditional acoustic model (AM) that is a small fraction of the size of the text data used to train a traditional language model (LM). Although techniques have been proposed to improve the performance of two-pass models for rare words, many of the proposed techniques significantly increase model size (e.g., incorporating recurrent neural network language models (RNN-LM) into speech recognition systems), training time (e.g., training using unsupervised speech-text pairs), and / or inference costs.

[0025] To increase the effectiveness of the two-pass model for rare words, embodiments herein are directed to incorporating a joint acoustic and text decoder (JATD) in the two-pass model architecture. The JATD feature provides a domain identifier (ID) that indicates whether the training examples correspond to supervised speech-text pairs or to speech-text examples generated from unpaired data (e.g., text-only or speech-only samples). In the case of unpaired data, the missing half of the pair may be synthesized by using a text-to-speech (TTS) system to generate the missing speech portion or an ASR system to generate the missing text portion. During training, the encoder generates an acoustic context vector that is fed to the decoder when the encoder receives paired examples. Here, the successful generation of the acoustic context vector represents or forms a domain ID that indicates the paired data. On the other hand, when the decoder receives unpaired examples, a fixed but learnable context vector domain ID is used to bypass the encoder network. Both of these examples are used for training in the decoder, allowing the decoder to be trained simultaneously on both paired and unpaired data without increasing model size. Furthermore, instead of approaches that share attention and decoder parameters across all modes, which only change the parameters of the input to the encoder based on the input source, this JATD approach only shares the decoder parameters and uses different attention context parameters.On a test set of various proper nouns and rare words, the JATD model achieved a relative reduction in WER of 3-10% compared to a two-pass architecture trained only on paired data.

[0026] 1A and 1B are an example of a speech environment 100. In the speech environment 100, a user's 10 way of interacting with a computing device, such as a user device 110, may be through voice input. The user device 110 (also commonly referred to as device 110) is configured to capture sounds (e.g., streaming voice data) from one or more users 10 in the speech-enabled environment 100. Here, the streaming voice data 12 may refer to speech spoken by the user 10 that serves as an audible query, a command to the device 110, or an audible conversation captured by the device 110. The speech-enabled system of the device 110 may process the query or command by responding to the query and / or by causing the command to be executed.

[0027] The user device 110 may correspond to any computing device associated with the user 10 and capable of receiving voice data 12. Some examples of the user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, Internet of Things (IoT) devices, smart speakers, etc. The user device 110 includes data processing hardware 112 and memory hardware 114 in communication with the data processing hardware 112 and storing instructions that, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations. The user device 110 further includes an audio subsystem having voice capture devices (e.g., microphones) 116, 116a for capturing and converting spoken utterances 12 into electrical signals within the speech response system 100, and speech output devices (e.g., speakers) 116, 116b for conveying audible voice signals (e.g., as output voice data from the device 110). In the illustrated example, the user device 110 implements a single voice capture device 116a, but the user device 110 may implement an array of voice capture devices 116a without departing from the scope of this disclosure, whereby one or more capture devices 116a in the array may not be physically present on the user device 110 but may be in communication with the voice subsystem 116. The user device 110 (e.g., using hardware 112, 114) is further configured to perform voice recognition processing on the streaming voice data 12 using a voice recognizer 200. In some examples, the voice subsystem 116 of the user device 110, including the voice capture device 116a, is configured to receive the voice data 12 (e.g., spoken utterances) and convert the voice data 12 into a digital format compatible with the voice recognizer 200. The digital format may correspond to acoustic frames (e.g., parameterized acoustic frames), such as mel frames.For example, a parameterized acoustic frame corresponds to the energy of a log-mel filterbank.

[0028] In some examples, such as FIG. 1A, a user 10 interacts with a program or application 118 of a user device 110 that uses a speech recognizer 200. For example, FIG. 1A illustrates a state in which a user 10 is communicating with an automated assistant application. In this example, the user 10 asks the automated assistant, "What time is the concert tonight?" This question from the user 10 is a spoken utterance 12 that is captured by a voice capture device 116a and processed by the voice subsystem 116 of the user device 110. In this example, the speech recognizer 200 of the user device 110 receives a speech input 202 of "What time is the concert tonight" (e.g., as an acoustic frame) and transcribes the speech input 202 into a transcription 204 (e.g., a textual representation of "What time is the concert tonight?"). The automated assistant of the application 118 can now use natural language processing to answer the question posed by the user 10. Natural language processing generally refers to the process of interpreting written language (e.g., transcript 204) and determining whether the written language prompts some action. In this example, the automated assistant uses natural language processing to recognize that a question from user 10 is about the user's schedule, more specifically, a concert on the user's schedule. By recognizing these details with natural language processing, the automated assistant returns an answer to the user's query: "Doors open at 8:30 pm for the concert tonight." In some configurations, natural language processing may occur on a remote system in communication with the data processing hardware 112 of the user device 110.

[0029] FIG. 1B is another example of speech recognition using a speech recognizer 200. In this example, a user 10 associated with a user device 110 is communicating with a friend named Jane Doe using a communication application 118. Here, the user 10 named Ted communicates with Jane by having the speech recognizer 200 transcribe his or her speech inputs. The voice capture device 116 captures these speech inputs and conveys them in a digital format (e.g., acoustic frames) to the speech recognizer 200. The speech recognizer 200 transcribes these acoustic frames into text that is sent to Jane via the communication application 118. Because this type of application 118 communicates via text, the transcription 204 from the speech recognizer 200 may be sent to Jane without further processing (e.g., natural language processing).

[0030] In some examples, such as in FIG. 2A and FIG. 2B, the speech recognizer 200 is configured with a two-pass architecture. In general, the two-pass architecture of the speech recognizer 200 includes at least one shared encoder 210, an RNN-T decoder 220, and an LAS decoder 230. Here, FIG. 2A shows a conventional two-pass architecture, and FIG. 2B shows an extended two-pass architecture incorporating a JATD model 240. As can be seen from these figures, the extended two-pass architecture of FIG. 2B is based on the basic two-pass structure of FIG. 2A. In two-pass decoding, the second pass 208 (e.g., shown as the LAS decoder 230) may improve the initial output from the first pass 206 (e.g., shown as the RNN-T decoder 220) using techniques such as lattice rescoring or n-best re-ranking. In other words, the RNN-T decoder 220 generates streaming predictions, and the LAS decoder 230 finalizes the predictions. Specifically, the LAS decoder 230 receives the streamed hypotheses y from the RNN-T decoder 220. R In general, the LAS decoder 230 rescores the hypothesis y 222 streamed from the RNN-T decoder 220.R Although described as functioning in a rescoring mode in which the LAS decoder 222 is rescored, the LAS decoder 230 may also operate in a different mode, such as a beam search mode, depending on design or other factors (e.g., the length of the utterance).

[0031] At least one encoder 210 is configured to receive as audio input 202 audio frames corresponding to streaming audio data 12. The audio frames may be pre-processed by the audio subsystem 116 into parameterized audio frames (e.g., mel-frames and / or spectral frames). In some embodiments, the parameterized audio frames correspond to log-mel filter bank energies having log-mel features. For example, the parameterized input audio frames output by the audio subsystem 116 and input to the encoder 210 may be a set of parameterized input audio frames x=(x1,...,x T ), where:

[0032]

number

[0033] Although the structure of the encoder 210 can be implemented in different ways, in some embodiments, the encoder 210 is a long-short term memory (LSTM) neural network. For example, the encoder 210 includes eight LSTM layers, where each layer includes 2048 hidden units followed by a 640-dimensional projection layer. In some examples, a time-reduction layer with reduction factor N=2 is inserted after the second LSTM layer of the encoder 210.

[0034] In some configurations, the encoder 210 is a shared encoder network. In other words, instead of each path network 206, 208 having its own separate encoder, each path 206, 208 shares a single encoder 210. By sharing an encoder, the ASR speech recognizer 200 using a two-pass architecture can reduce its model size and / or its computational cost. Here, the reduction in model size can help enable the speech recognizer 200 to perform well entirely on-device.

[0035] 2A also includes an additional encoder, such as an acoustic encoder 250, to adapt the output 212 of the encoder 210 to be suitable for the second pass 208 of the LAS decoder 230. The acoustic encoder 250 is configured to further encode the output 212 into an encoded output 252. In some embodiments, the acoustic encoder 250 is an LSTM encoder (e.g., a two-layer LSTM encoder) that further encodes the output 212 from the encoder 210. By including an additional encoder, the encoder 210 may still be retained as a shared encoder between the passes 206, 208.

[0036] Through a first pass 206, an encoder 210 receives each acoustic frame of the speech input 202 and generates an output 212 (e.g., denoted as an encoding of the acoustic frame, e). An RNN-T decoder 220 receives the output 212 for each frame and generates a hypothesis y R The RNN-T decoder 220 generates an output 222, denoted as , in a streaming manner at each time step. In some embodiments, the RNN-T decoder 220 includes a prediction network and a combination network, where the prediction network may have two LSTM layers with 2048 hidden units and 640-dimensional projections (per layer), as well as an embedding layer with 128 units. The output 212 of the encoder 210 and the prediction network may be fed into a combination network including a softmax prediction layer. In some examples, the combination network of the RNN-T decoder 220 includes 640 hidden units followed by a softmax layer that predicts 4096 mixed-case word pieces.

[0037] In the two-pass model of FIG. 2A, over the second pass 208, the LAS decoder 230 receives the output 212 (or the encoded output 252) from the encoder 210 for each frame and generates a hypothesis y LWhen the LAS decoder 230 operates in beam search mode, the LAS decoder 230 generates the output 232 designated as output 212 (or output 252) only and ignores the output 222 of the RNN-T decoder 220. When the LAS decoder 230 operates in rescoring mode, the LAS decoder 230 obtains the top-K hypotheses from the RNN-T decoder 220, and then the LAS decoder 230 operates in supervised forced mode for each sequence while paying attention to the output 212 (or output 252) to calculate a score. For example, the score is a combination of the log probability of the sequence and an attention coverage penalty. The LAS decoder 230 selects the sequence with the highest score as the output 232. Here, in the rescoring mode, the LAS decoder 230 may include multi-headed attention (e.g., with four heads) to direct attention to the output 212 (or output 252). Furthermore, the LAS decoder 230 may be a two-layer LAS decoder 230 with a softmax layer for prediction. For example, each layer of the LAS decoder 230 has 2048 hidden units followed by a 640-dimensional projection. The softmax layer may include 4,096 dimensions to predict the same mixed-case word pieces from the softmax layer of the RNN-T decoder 220.

[0038] In some embodiments, training of the two-pass model of FIG. 2A is done in two stages. During the first stage, the encoder 210 and the RNN-T decoder 220

[0039]

number

[0040]

number

[0041]

number

[0042] 2B, the first pass 206 remains the same, but the second pass 208 uses a JATD model 240 that includes an attention mechanism 242 before decoding with the LAS decoder 230. Here, the attention mechanism 242 receives the encoded output 212 (or output 252) and generates context vectors c, c that summarize the encoded features for each output step. A , c L The attention mechanism 242 is configured to modify the context vector c passed to the LAS decoder 230 depending on the type of input training data. In other words, the attention mechanism 242 determines an acoustic context vector 244, c when the input training data (i.e., a particular training example) is a supervised speech-text pair. A and generates a fixed linguistic context vector 246, c when the input training data (e.g., training examples 203) are unpaired text sequences. L When the JATD model 240 is integrated with the LAS decoder 230, the JATD model 240 modifies both the inference and training of the LAS decoder 230 to utilize paired and / or unpaired data.

[0043] During inference, the LAS decoder 230 computes log probabilities based on the following formula: For example, using the acoustic input, the acoustic context vector c a determines the log-probability for the LAS decoder 230 at each decoder step u, where

[0044]

number

[0045]

number

[0046]

number

[0047]

number

[0048]

number

[0049] FIG. 4 is another flow chart of an example operation configuration for a method 400 of performing automatic speech recognition (e.g., ASR). At operation 402, the method 400 receives training examples for the LAS decoder 230 of a two-pass streaming neural network model, where the training examples are configured to train the LAS decoder 230. At operation 404, the method 400 determines whether the training examples correspond to supervised speech-text pairs or to unpaired training data. If the training examples correspond to unpaired training data, at operation 406, the method 400 generates missing portions of the unpaired training data to form a generated speech-text pair. At operation 408, the method 400 updates the context vector c associated with the LAS decoder 230 and the unpaired data based on the generated speech-text pair.

[0050] 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems (e.g., speech recognizer 200) and methods (e.g., methods 300, 400) described herein. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components, their connections and relationships, and their functions illustrated herein are merely exemplary and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0051] Computing device 500 includes a processor 510 (e.g., data processing hardware), a memory 520 (e.g., memory hardware), a storage device 530, a high-speed interface / controller 540 that connects to memory 520 and a high-speed expansion port 540, and a low-speed interface / controller 560 that connects to a low-speed bus 570 and storage device 530. Each of components 510, 520, 530, 540, 550, and 560 may be interconnected using various buses and mounted on a common motherboard or in other suitable manner. Processor 510 processes instructions for execution within computing device 500, including instructions stored in memory 520 or storage device 530, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 connected to high-speed interface 540. In other embodiments, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and multiple types of memories. Also, multiple computing devices 500 may be connected, with each device providing a portion of the required processing (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0052] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 520 may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0053] The storage device 530 can provide mass storage for the computing device 500. In some embodiments, the storage device 530 is a computer-readable medium. In various different embodiments, the storage device 530 can be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. In additional embodiments, the computer program product is tangibly embodied in an information medium. The computer program product includes instructions that, when executed, perform one or more methods as described above. The information medium is a computer-readable medium or a machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.

[0054] The high-speed controller 540 manages the bandwidth-intensive operations of the computing device 500, and the low-speed controller 560 manages the lower bandwidth-intensive operations. This distribution of roles is merely exemplary. In some embodiments, the high-speed controller 540 is connected to the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and the high-speed expansion port 550 that accepts various expansion cards (not shown). In some embodiments, the low-speed controller 560 is connected to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590, which includes various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be connected to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device, such as a switch or router, for example, via a network adapter.

[0055] The computing device 500 can be implemented in a number of different forms, as shown in the drawing, such as a standard server 500a, or multiple times within a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0056] Various embodiments of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, connected to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0057] These computer programs (also known as programs, software, software applications, or codes) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus and / or devices (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0058] The processes and logic flows described herein can be performed by one or more programmable processors executing one or more computer programs that perform functions by processing input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, and any one or more processors of any type of digital computer. In general, a processor receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. In general, a computer is operatively coupled to receive and / or transfer data from one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0059] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device, such as, for example, a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to a user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices may also be used to provide interaction with a user, with feedback provided to the user, which may be, for example, any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, the computer may interact with the user by sending and receiving documents to and from a device used by the user (e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser).

[0060] Although several embodiments have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. 1. A computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations, the operations comprising: receiving a sequence of acoustic frames characterizing an utterance; encoding each audio frame in the sequence of audio frames using a shared encoder to generate a corresponding encoded audio frame; In each output step of the plurality of output steps for the second pass decoder, determining an acoustic context vector that summarizes the encoded audio frame using an attention mechanism; using the acoustic context vector to determine an acoustic-based probability distribution over possible output labels; using the attention mechanism to determine a linguistic context vector based on a sequence of decoded labels previously output by the second pass decoder; using the linguistic context vector to determine a text-based probability distribution over possible output labels; interpolating the acoustic-based probability distributions over possible output labels and the text-based probability distributions over possible output labels; determining a transcription of the utterance based on the interpolation of the acoustic-based probability distributions over possible output labels and the text-based probability distributions over possible output labels at each of the plurality of output steps.

2. The method of claim 1 , wherein determining the language context vector based on the sequence of decoded labels previously output by the second pass decoder ignores the encoded acoustic frames.

3. 2. The method of claim 1 , wherein determining the acoustic-based probability distribution over possible output labels using the acoustic context vector is further based on a sequence of the decoded labels previously output by a second pass decoder.

4. 2. The method of claim 1, wherein the second-pass decoder operates in a beam search mode based on hypotheses generated by a recurrent neural network transducer (RNN-T) decoder during the first pass.

5. The operation includes: processing the encoded acoustic frames using a first pass decoder to generate a list of top K speech recognition hypotheses for the utterance; the second pass decoder operating in a rescoring mode to rescore each speech recognition hypothesis in the top K list of speech recognition hypotheses; The method of claim 1 , wherein each speech recognition hypothesis in the top K list of speech recognition hypotheses corresponds to a candidate transcription of the utterance.

6. The method of claim 1 , wherein the output labels in the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels comprise word pieces.

7. 2. The method of claim 1, wherein the second pass decoder comprises a Listen-Attend-Spell (LAS) decoder.

8. 2. The method of claim 1 , wherein the acoustic-based probability distribution for possible output labels and the text-based probability distribution for possible output labels are weighted differently than the text-based probability distribution for possible output labels.

9. the speech characterized by the sequence of acoustic frames is captured in streaming audio by a user device; The method of claim 1 , wherein the data processing hardware is resident on the user device.

10. 10. The method of claim 9, wherein the operations further include performing natural language processing on the transcription to identify an action for a digital assistant application to perform.

11. 1. A system comprising: Data processing hardware; and memory hardware in communication with the data processing hardware and storing instructions, the instructions, when executed on the data processing hardware, receiving a sequence of acoustic frames characterizing an utterance; encoding each audio frame in the sequence of audio frames using a shared encoder to generate a corresponding encoded audio frame; In each output step of the plurality of output steps for the second pass decoder, determining an acoustic context vector that summarizes the encoded audio frame using an attention mechanism; determining an acoustic-based probability distribution over possible output labels using the acoustic context vector; determining a linguistic context vector based on a sequence of decoded labels previously output by the second pass decoder using the attention mechanism; determining a text-based probability distribution over possible output labels using the linguistic context vector; interpolating the acoustic-based probability distributions over possible output labels and the text-based probability distributions over possible output labels; determining a transcription of the utterance based on the interpolation of the acoustic-based probability distributions over possible output labels and the text-based probability distributions over possible output labels at each of the plurality of output steps.

12. 12. The system of claim 11, wherein determining the language context vector based on the sequence of decoded labels previously output by the second pass decoder ignores the encoded acoustic frames.

13. 12. The system of claim 11, wherein using the acoustic context vector to determine the acoustic-based probability distribution over possible output labels is further based on a sequence of the decoded labels previously output by a second pass decoder.

14. 12. The system of claim 11, wherein the second pass decoder operates in a beam search mode based on hypotheses generated by a recurrent neural network transducer (RNN-T) decoder through the first pass.

15. The operation includes: processing the encoded acoustic frames using a first pass decoder to generate a list of top K speech recognition hypotheses for the utterance; the second pass decoder further comprises operating in a rescoring mode to rescore each speech recognition hypothesis in the top K list of speech recognition hypotheses; The system of claim 11 , wherein each speech recognition hypothesis in the top K list of speech recognition hypotheses corresponds to a candidate transcription of the utterance.

16. The system of claim 11 , wherein the output labels in the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels comprise word pieces.

17. 12. The system of claim 11, wherein the second pass decoder comprises a Listen-Attend-Spell (LAS) decoder.

18. 12. The system of claim 11, wherein the acoustic-based probability distribution for possible output labels and the text-based probability distribution for possible output labels are weighted differently than the text-based probability distribution for possible output labels.

19. the speech characterized by the sequence of acoustic frames is captured in streaming audio by a user device; The system of claim 11 , wherein the data processing hardware is resident on the user device.

20. 20. The system of claim 19, wherein the operations further include performing natural language processing on the transcription to identify an action to be performed by a digital assistant application.

Images