Streaming RNN Transducers with Non-Autoregressive Decoding

Abstract

The method (400) comprises receiving an initial alignment (234) of candidate hypotheses (120a) generated by the transducer decoder model (230) during a first pass. The candidate hypotheses correspond to candidate transcriptions of the utterance (106), and the initial alignment of the candidate hypotheses comprises a sequence of output labels. Each output label corresponds to a whitespace symbol or a hypothetical subword unit. The method also comprises receiving a subsequent sequence of audio encodings characterizing the utterance (222). During the initial refinement step, the method also comprises generating a new alignment (324) to the rescored sequence of output labels by using a non-autoregressive decoder (300). The non-autoregressive decoder is configured to receive the initial alignment of the candidate hypotheses and the subsequent sequence of audio encodings.

Description

[Technical Field]

[0001] This disclosure relates to the study of streaming RNN transducers with non-autoregressive decoding. [Background technology]

[0002] Automatic speech recognition (ASR) systems have evolved from multiple models, each with its own dedicated purpose, to integrated models that directly map audio waveforms (i.e., input sequences) to output sentences (i.e., output sequences) using a single neural network. This integration results in a sequence-to-sequence approach, where a sequence of audio features is given to generate a sequence of words (or graphemes). The integrated structure allows all components of the model to be jointly trained as a single end-to-end (E2E) neural network. Here, an E2E model refers to a model whose architecture is entirely built on neural networks. That is, it is a complete neural network function without external and / or manually designed components (e.g., finite-state transducers, lexicons, text normalization modules). In addition, when training E2E models, these models typically do not require bootstrapping from decision trees or time-training from another system. These E2E ASR systems have made tremendous progress, outperforming traditional automatic speech recognition ASR systems in several common benchmarks, including word error rate (WER). Summary of the Invention [Problem to be solved by the invention]

[0003] For example, many applications that involve user interaction, such as voice search or on-device dictation, require models to perform recognition in a streaming manner. Other applications, such as offline video capture, do not require models to stream and can utilize future context to improve performance. In many cases, models will benefit from performing recognition in a streaming manner, simultaneously improving performance similar to non-streaming models that utilize future context. [Means for solving the problem]

[0004] One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for performing streaming RNN-T exploration with non-autoregressive decoding. The operations include receiving an initial alignment of candidate hypotheses generated by a transducer decoder model during a first pass. The candidate hypotheses correspond to candidate transcriptions of the utterance, and the initial alignment of the candidate hypotheses includes a sequence of output labels, each of which corresponds to a whitespace symbol or a hypothesized subword unit. The operations also include receiving a subsequent sequence of audio encodings characterizing the utterance. During an initial refinement step, the operations include generating a new alignment for the rescored sequence of output labels using a non-autoregressive decoder configured to receive the initial alignment of candidate hypotheses generated by the transducer model during the first pass and the subsequent sequence of audio encodings.

[0005] Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the non-autoregressive decoder includes multiple transform layers, each configured to perform self-attention on text features associated with the initial alignment and cross-attention on a subsequent sequence of audio encodings representing both keys and values ​​by using the self-attention performed on the text features as a query to provide a transform layer output. In these embodiments, each transform layer subsequent to an initial transform layer in the multiple transform layers receives the transform layer output from a corresponding previous transform layer as a text feature. The final transform layer in the multiple transform layers provides the transform layer output to a final softmax layer configured to predict a new alignment for the rescored sequence of output labels.

[0006] In some examples, during each of one or more additional refinement steps following the initial refinement step, the operations further include generating a new alignment for the rescored sequence of output labels by using a non-autoregressive decoder configured to receive a new alignment for the rescored sequence of output labels generated during the previous refinement step. Generating a new alignment for the rescored sequence of output labels may include inserting, deleting, or substituting one or more output labels in the initial alignment of the candidate hypotheses.

[0007] In some implementations, the operations further include generating, during the first pass, an initial sequence of audio encodings by a causal encoder based on the sequence of acoustic frames corresponding to the utterance. In these implementations, a subsequent sequence of audio encodings is encoded by a non-causal encoder based on the initial sequence of audio encodings. The transducer decoder may generate candidate hypotheses using the initial sequence of audio encodings. In some examples, candidate transcriptions of the candidate hypotheses comprise a sequence of output labels, each corresponding to a hypothesized subword unit.

[0008] Another aspect of the present disclosure provides a system comprising data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving an initial alignment of candidate hypotheses generated by a transducer decoder model during a first pass. The candidate hypotheses correspond to candidate transcriptions of the utterance, and the initial alignment of the candidate hypotheses comprises a sequence of output labels, each of which corresponds to a whitespace symbol or a hypothesized subword unit. The operations also include receiving a subsequent sequence of audio encodings characterizing the utterance. During an initial refinement step, the operations include generating a new alignment for the rescored sequence of output labels using a non-autoregressive decoder configured to receive the initial alignment of candidate hypotheses generated by the transducer model during the first pass and the subsequent sequence of audio encodings.

[0009] Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the non-autoregressive decoder includes multiple transform layers, each configured to provide a transform layer output by performing self-attention on text features associated with the initial alignment and performing cross-attention on a subsequent sequence of audio encodings representing both keys and values ​​using the self-attention performed on the text features as a query. In these embodiments, each transform layer subsequent to an initial transform layer in the multiple transform layers receives the transform layer output from a corresponding previous transform layer as a text feature. The final transform layer in the multiple transform layers provides the transform layer output to a final softmax layer configured to predict a new alignment for the rescored sequence of output labels.

[0010] In some examples, during each of one or more additional refinement steps following the initial refinement step, the operations further include generating a new alignment for the rescored sequence of output labels by using a non-autoregressive decoder configured to receive a new alignment for the rescored sequence of output labels generated during the previous refinement step. Generating a new alignment for the rescored sequence of output labels may include inserting, deleting, or substituting one or more output labels in the initial alignment of the candidate hypotheses.

[0011] In some implementations, the operations further include generating, by a causal encoder during the first pass, an initial sequence of audio encodings based on the sequence of acoustic frames corresponding to the utterance. In these implementations, a subsequent sequence of audio encodings is encoded by a non-causal encoder based on the initial sequence of audio encodings. The transducer decoder may generate candidate hypotheses using the initial sequence of audio encodings. In some examples, candidate transcriptions of the candidate hypotheses comprise a sequence of output labels, each corresponding to a hypothesized subword unit.

[0012] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram of an example of a speech recognition system. [Figure 2] 1 is a schematic diagram of an example of a speech recognition model that employs non-autoregressive decoding. [Figure 3] FIG. 3 is a schematic diagram of an example non-autoregressive decoder of the speech recognition model of FIG. 2 during an initial refinement step. [Figure 4] 1 is a flowchart of an exemplary sequence of operations of a computer-implemented method for conducting a study with non-autoregressive decoding. [Figure 5] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0014] Like reference symbols in the various drawings refer to like elements. End-to-end (E2E) automatic speech recognition (ASR) models are traditionally structured to operate in either streaming or non-streaming modes. Traditionally, E2E ASR models have an encoder and a decoder as their main components. Applications involving end-user interaction, such as voice search or on-device dictation, may require the model to perform recognition in a streaming manner. Here, streaming recognition refers to the automatic speech recognition ASR model outputting each spoken word with as little latency as possible. Other applications, such as offline video captioning, do not require the model to stream and can utilize future context to improve performance. For example, our model demonstrates significant improvement in recognizing rare or out-of-vocabulary (OOV) words compared to long short-term memory (LSTM) models or transform-rescoring models. That is, our model excels at refining initial speech recognition results by examining the full audio context using an attention mechanism.

[0015] Improved performance of the considered model comes at the cost of increased latency and larger model size, making the considered model unsuitable for streaming or on-device applications. In particular, the considered model is often an autoregressive model constrained to consider initial speech recognition results in left-to-right order. Non-autoregressive models, on the other hand, are not constrained to consider initial speech recognition results in left-to-right order. That is, non-autoregressive models can simultaneously update multiple positions (e.g., output frames) of the initial speech recognition result at each output step. Thus, non-autoregressive models tend to have lower latency but lower accuracy (e.g., word error rate (WER)) than similarly sized single-pass autoregressive models.

[0016] Embodiments herein are directed to methods and systems for testing a streaming recurrent neural network transducer (RNN-T) with non-autoregressive decoding. More specifically, a non-autoregressive decoder receives an initial alignment of candidate hypotheses for an utterance generated by a transducer decoder model during a first pass. Here, the transducer decoder may be a small autoregressive model that generates candidate hypotheses with low WER and low delay. The non-autoregressive decoder also receives a subsequent sequence of audio encodings that characterize the utterance. During the initial refinement step, the non-autoregressive decoder generates a new alignment for the rescored sequence of output labels. In particular, because the subsequent sequence of audio encodings is generated by a cascading encoder using an additional right context, the non-autoregressive decoder benefits from the additional audio context before testing. That is, the non-autoregressive decoder is not constrained to perform testing in a left-to-right order and generates a new alignment based on label dependencies from the additional right context. Furthermore, as will become apparent, a non-autoregressive decoder may perform any number of additional refinement steps following the initial refinement step, whereby a new alignment is generated for each additional refinement step.

[0017] 1 is an example of a system 100 operating in a voice environment. In a voice environment, the way a user 104 interacts with a computing device, such as a user device 10, may be through voice input. The user device 10 (also generally referred to as device 10) is configured to capture sound (e.g., streaming audio data) from one or more users 104 in the voice environment. Here, streaming audio data may refer to voice utterances 106 by the users 104 that function as audible queries, commands to the user device 10, or audible communications captured by the device 10. The voice-enabled system of the user device 10 may process the queries or commands by responding to the queries and / or causing commands to be performed / fulfilled by one or more downstream applications.

[0018] The user device 10 may correspond to any computing device associated with a user 104 and enabled to receive audio data. Some examples of the user device 10 include, but are not limited to, a mobile device (e.g., a mobile phone, a tablet, a laptop), a computer, a wearable device (e.g., a smart watch), a smart appliance, an Internet of Things (IoT) device, a vehicle infotainment system, a smart display, a smart speaker, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 in communication with the data processing hardware 12, storing instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 with audio capture devices (e.g., microphones) 16, 16a that capture and convert audio utterances 106 in an audio environment into electrical signals, and audio output devices (e.g., speakers) 16, 16b that communicate audible audio signals (e.g., as output audio data from the user device 10). In the example shown, the user device 10 implements a single audio capture device 16a, but the user device 10 may implement an array of audio capture devices 16a without departing from the scope of this disclosure, whereby one or more capture devices 16a in the array may not physically reside on the user device 10 but may communicate with the audio system 16.

[0019] In a speech environment, an automatic speech recognition (ASR) system 118 implements an automatic speech recognition ASR model 200 and resides on the user device 10 of the user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system running in a cloud computing environment) that communicates with the user device 10 via a network 40. In some examples, the automatic speech recognition ASR model 200 may be a recurrent neural network transducer (RNN-T) model. The user device 10 and / or the remote computing device 60 also include an audio subsystem 108 configured to receive utterances 106 spoken by the user 104 and captured by the audio capture device 16 a, and to convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the automatic speech recognition ASR system 118. In the illustrated example, a user speaks each utterance 106, and an audio subsystem 108 converts the utterance 106 into corresponding audio data (e.g., a sequence of acoustic frames) 110 for input to an automatic speech recognition ASR system 118. An automatic speech recognition ASR model 200 then receives as input the sequence of acoustic frames 110 corresponding to the utterance 106, thereby generating / predicting a corresponding transcription 120 (e.g., a speech recognition result / hypothesis) of the utterance 106 at each output step as the automatic speech recognition ASR model 200 receives (e.g., processes) each acoustic frame 110 in the sequence of acoustic frames 110.

[0020] In the illustrated example, the automatic speech recognition ASR model 200 performs streaming speech recognition to generate initial speech recognition results (e.g., candidate hypotheses) 120, 120a, and may generate final speech recognition results (e.g., final hypotheses) 120, 120b by improving the initial speech recognition result 120a. The initial speech recognition result 120a and the final speech recognition result 120b may correspond to either a partial speech recognition result or an entire speech recognition result. In other words, the initial speech recognition result 120a and the final speech recognition result 120b may correspond to either a portion of the utterance 106 or the entire utterance 106. For example, a partial speech recognition result may correspond to a portion of a spoken speech utterance, or even a portion of a spoken term. However, as described below, the automatic speech recognition ASR model 200 performs additional processing on the final speech recognition result 120b, and therefore the final speech recognition result 120b may lag behind the initial speech recognition result 120a.

[0021] The user device 10 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of the transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in further detail below, the user interface generator 107 may stream an initial speech recognition result 120a during time 1, and then stream a final speech recognition result 120b during time 2. In particular, the automatic speech recognition ASR model 200 outputs the final speech recognition result 120b in a streaming manner, even though the final speech recognition result 120b is an improvement over the initial speech recognition result 120a. In some configurations, the transcription 120 output from the automatic speech recognition ASR system 118 is processed (e.g., by a natural language understanding (NLU) module executing on the user device 10 or the remote computing device 60) to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., running on any combination of the user device 10 or the remote computing device 60) may convert the transcription 120 into synthesized speech for audible output by the user device 10 and / or another device.

[0022] In the illustrated example, a user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) on a user device 10 that uses an automatic speech recognition ASR system 118. For example, FIG. 1 shows a conversation between the user 104 and the digital assistant application 50 as the user 104 communicates with the digital assistant application 50, which displays a digital assistant interface 18 on the screen of the user device 10. In this example, the user 104 asks the digital assistant application 50, "What time is the concert tonight?" This question from the user 104 is a speech utterance 106 that is captured by an audio capture device 16a and processed by the audio system 16 of the user device 10. In this example, the audio system 16 receives the speech utterance 106 and converts it into a sequence of acoustic frames 110 for input to the automatic speech recognition ASR system 118.

[0023] Continuing with the example, the automatic speech recognition ASR model 200 receives a sequence of acoustic frames 110 corresponding to an utterance 106 as the user 104 speaks, encodes the sequence of acoustic frames 110, and then decodes the encoded sequence of acoustic frames 110 into an initial speech recognition result 120a. During time 1, the user interface generator 107 presents a representation of the initial speech recognition result 120a of the utterance 106 to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming manner, whereby words, word fragments, and / or individual characters are displayed on the screen as they are spoken. In some examples, the first look-ahead audio context is equal to zero.

[0024] During time 2, user interface generator 107 presents a representation of final speech recognition result 120b of utterance 106 to user 104 of user device 10 via digital assistant interface 18 in a streaming manner, whereby words, word fragments, and / or individual characters are displayed on the screen as soon as they are generated by automatic speech recognition ASR model 200. In some implementations, user interface generator 107 replaces the representation of initial speech recognition result 120a presented at time 1 with the representation of final speech recognition result 120b presented at time 2. Here, time 1 and time 2 may include timestamps corresponding to when user interface generator 107 presented each speech recognition result 120. In this example, the timestamp for time 1 indicates that user interface generator 107 presented initial speech recognition result 120a at an earlier time than final speech recognition result 120b. For example, because the final speech recognition result 120b is estimated to be more accurate than the initial speech recognition result 120a, the final speech recognition result 120b, which is ultimately displayed as the transcription 120, may correct any terms that may have been misrecognized in the initial speech recognition result 120a. In this example, the streaming initial speech recognition result 120a output by the automatic speech recognition ASR model 200 is displayed on the screen of the user device 10 at time 1, thereby associated with low latency and providing the user 104 with a sense of responsiveness that the user's query is being processed. Meanwhile, the final speech recognition result 120b output by the automatic speech recognition ASR model 200 and displayed on the screen at time 2 utilizes additional speech recognition and / or language models to improve the quality of the speech recognition in terms of accuracy, but at the expense of increased latency. However, because the initial speech recognition result 120a is displayed as the user speaks the utterance 106, the long latency associated with generating and ultimately displaying the final speech recognition result 120b is not noticeable to the user 104.

[0025] In the example shown in FIG. 1 , digital assistant application 50 can respond to a question posed by user 104 using natural language processing. Natural language processing generally refers to the process of interpreting written language (e.g., initial speech recognition result 120a and / or final speech recognition result 120b) and determining whether the written language prompts any action. In this example, digital assistant application 50 uses natural language processing to recognize that the question from user 104 is about the user's schedule, and more specifically, about a concert on the user's schedule. By recognizing these details through natural language processing, the automated assistant returns response 19 to the user's query, where response 19 states, "The venue opens at 6:30 PM, and the concert starts at 8 PM." In some configurations, natural language processing occurs on a remote server 60 that communicates with data processing hardware 12 of user device 10.

[0026] Referring now to FIG. 2, in some examples, an automatic speech recognition (ASR) model 200 includes a cascading encoder (encoder) 204, a transducer decoder (transducer decoder) 230, and a non-autoregressive decoder (decoder) 300. The cascading encoder 204 refers to a model structure in which two encoders 210, 220 are included in the encoding path, and the output of the first encoder (first encoder) 210 feeds the input of the second encoder (second encoder) 220 before decoding. Here, the first encoder 210 and the second encoder 220 can be cascaded regardless of the underlying architecture of each encoder. The encoders 210, 220 can each include a multi-head (e.g., eight-head) attention layer stack. In some examples, the multi-head attention layer stack of the encoders 210, 220 includes a stack of 512-dimensional conformer layers. In other examples, a transform layer can be used instead of a conformer layer.

[0027] The first encoder 210 may be a causal encoder with 17 conformer layers with a multi-head (e.g., eight-head) attention mechanism, each used as a self-attention layer. Furthermore, each conformer layer of the first encoder 210 may use a causal convolutional layer and a left-context attention layer to restrict the first encoder from using any future input (e.g., a right context equal to zero). On the other hand, the second encoder 220 may be a non-causal encoder with four conformer layers with a multi-head (e.g., eight-head) attention mechanism, each used as a self-attention layer. Each conformer layer of the second encoder may use a non-causal convolutional layer and a right-context attention layer to enable the second encoder 220 to use (e.g., focus on) future input. That is, the second encoder 220 may receive and process an additional right context (e.g., 2.88 seconds) to generate the encoder output.

[0028] Continuing with reference to FIG. 2, the first encoder 210 encodes a sequence of d-dimensional feature vectors (e.g., a sequence of acoustic frames 110) x=(x1, x2, . . . , x T )(where

[0029]

number

[0030] , and generates a first high-order feature representation 212 of a corresponding acoustic frame 110 in the sequence of acoustic frames 110 at each output step. Similarly, the second encoder 220 is cascaded to the first encoder 210, and receives the first high-order feature representation 212 as input, and generates a second high-order feature representation 222 for the corresponding first high-order feature representation 212 (e.g., an initial sequence of audio encoding) at each output step. In particular, the second encoder 220 focuses on additional right context to generate each of the second high-order feature representations 222 (e.g., a subsequent sequence of audio encoding). However, in some cases, the second encoder 220 generates the second high-order feature representation 222 without receiving any acoustic frames 110 as input. In these examples, the second encoder 220 generates the second high-order feature representation 222 using only the first high-order feature representation 212 as input. The cascading encoder 204 may operate in a streaming manner, and at each output step, the cascading encoder 204 generates a first high-level feature representation 212 and a second high-level feature representation 222 corresponding to either a portion of the utterance or the entire utterance.

[0031] The transducer decoder 230 may include an RNN-T architecture having a joint network 232 and a prediction network 236. In some examples, the transducer decoder 230 is an autoregressive model with a model size smaller than that of the non-autoregressive decoder 300. The transducer decoder generates a decoder output by combining the first high-level feature representation 212 output by the first encoder 210 and a dense representation 238 output by the prediction network 236 using the joint network 232. That is, the joint network 232 is configured to receive as input the dense representation 238 output by the prediction network 236 and the first high-level feature representation 212 generated by the first encoder 210, thereby generating a candidate hypothesis 120a at each output step. Although not shown, the transducer decoder 230 may include a final softmax layer that receives the output of the transducer decoder 230. In some implementations, the softmax layer is separate from the transducer decoder 230 and processes the output from the transducer decoder 230. The output of the softmax layer is then used in the beam search process to select the orthogonal projection elements. In some implementations, the softmax layer is integrated into the transducer decoder 230, such that the output of the transducer decoder 230 represents the output of the softmax layer.

[0032] In some implementations, the candidate hypotheses 120a output by the transducer decoder 230 comprise a probability distribution over possible initial alignments 234 (e.g., a probability associated with each possible initial alignment 234). In other words, the joint network 232 generates a probability distribution over possible initial alignments 234 at each output step (e.g., time step). Here, each "possible initial alignment 234" corresponds to a sequence of output labels / frames, each corresponding to a space symbol or a hypothetical subword unit. Each hypothetical subword unit may represent a grapheme (symbol / character) or part of a word in a specified natural language. For example, when the natural language is English, the sequence of output labels (i.e., the sequence of output frames) comprises 28 symbols, and therefore comprises, for example, one label for each of the 26 letters of the English alphabet, one label specifying a space, and one label specifying a space symbol. Thus, the transducer decoder 230 may output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels. This set of values ​​may be a vector (e.g., a one-hot vector) and may indicate a probability distribution over the set of output labels. In some scenarios, the output labels are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of output labels is not limited to such. For example, the set of output labels may include whitespace, word fragments, and / or whole words in addition to or instead of graphemes. The output labels may also be other types of phonetic units, such as phonemes or subphonemes.

[0033] In some implementations, the output distribution of the transducer decoder 230 comprises a posterior probability value for each different output label in each output frame of the sequence of output frames. Thus, if there are 100 different output labels representing different graphemes, whitespace characters, or other symbols, the initial alignment 234 output by the transducer decoder 230 may include 100 different probability values, one for each output label, in each output frame of the sequence of output frames. In some cases, the transducer decoder 230 outputs a single output label having the highest corresponding probability value in each output frame. For example, the transducer decoder 230 may select the hypothetical subword unit "adventure" as each output frame in the sequence of output frames based on "adventure" having the highest corresponding probability from the probability distribution in each output frame.

[0034] Alternatively, the transducer decoder 230 may select a blank symbol for each output frame in the sequence of output frames based on determining that the corresponding probability of each of the hypothetical sub-word units does not meet the threshold probability value. In other words, when the transducer decoder 230 does not generate a corresponding probability for any of the hypothetical sub-word units that meets the threshold probability value, the transducer decoder 230 is unlikely to select the correct hypothetical sub-word unit, and thus the transducer decoder 230 selects a blank symbol. For example, the transducer decoder 230 may generate an initial alignment 234 of "φφ_pullφφ_pamp er s φφ," where φ represents a blank symbol and "_," "pull," "pamp," "er," and "s" each represent a hypothetical sub-word unit corresponding to the utterance "pull campers." Notably, the initial alignment 234 output by the transducer decoder 230 does not accurately correspond to the spoken speech.

[0035] In some examples, the transducer decoder 230 generates a candidate transcription for the candidate hypothesis 120a based on the initial alignment 234. In particular, the candidate transcription for the candidate hypothesis 120a comprises a sequence of output labels, each corresponding to a hypothesized subword unit. Thus, a difference between the candidate transcription for the candidate hypothesis 120a and the initial alignment 234 for the candidate hypothesis 120a is that the output labels for the initial alignment 234 may include whitespace characters, while the candidate transcription does not include whitespace characters. Thus, the transducer decoder 230 may generate the candidate transcription for the candidate hypothesis 120a by removing all whitespace characters from the initial alignment 234. Continuing with the above example, the transducer decoder 230 may generate a transcription of "pull pampers" using the initial alignment 234 by removing all whitespace characters φ. The transducer decoder 230 may output a transcription of the candidate hypothesis 120a to the user device 10 (FIG. 1).

[0036] Within the transducer decoder 230, the prediction network 236 may have two 2,048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. The prediction network 236 receives as input the sequence of non-empty symbols output by the final softmax layer of the joint network 232 to generate a dense representation 238 at each output step. The joint network 232 receives the dense representation 238 of the previous initial alignment 234 and uses the dense representation 238 to generate a subsequent initial alignment 234. The non-autoregressive decoder 300 is configured to receive the initial alignment 234 of the candidate hypotheses 120a generated by the transducer decoder 230 at each output step and the second high-level feature representation 222 generated by the second encoder 220 at each output step to generate a final hypothesis 120b at each output step. The final hypothesis 120 may include a new alignment 324 to the rescored sequence of output labels.

[0037] FIG. 3 illustrates a non-autoregressive decoder 300 performing an initial refinement step. The non-autoregressive decoder 300 may include a stack of multi-head attention layers 310. In some examples, the stack of multi-head attention layers comprises multiple transform layers 310. Thus, the terms stack of multi-head attention layers 310 and multiple transform layers 310 may be used interchangeably herein. In other examples, a conformer layer may be used instead of a transform layer. As illustrated in FIG. 3, the multiple transform layers 310 comprise three transform layers 310a-310c for clarity only, but it will be understood that the multiple transform layers 310 may include any number of transform layers 310.

[0038] Each of the transformation layers 310 is configured to perform self-attention on text features associated with the initial alignment 234 of the candidate hypotheses 120a. An initial transformation layer 310 of the multiple transformation layers 310 performs self-attention by extracting text features from the initial alignment 234 itself. As shown in Figure 3, the first transformation layer (first transformation layer) 310, 310a includes the initial transformation layer 310 and is configured to perform self-attention by extracting text features from the initial alignment 234. On the other hand, each of the transformation layers 310 subsequent to the initial transformation layer 310 of the multiple transformation layers 310 receives a transformation layer output 312 from the corresponding previous transformation layer 310 and extracts text features from the transformation layer output 312. Continuing to refer to Figure 3, the second conversion layer (second conversion layer) 310, 310b performs self-attention by extracting text features from the first conversion layer output 312, 312a output by the first conversion layer 310a, and the third conversion layer (third conversion layer) 310, 310c performs self-attention by extracting text features from the second conversion layer output 312, 312b output by the second conversion layer 310b.

[0039] Each of the transform layers 310 is further configured to provide (i.e., generate) a transform layer output 312 by performing cross-attention on a second high-level feature representation 222 representing both keys and values ​​by using the self-attention performed on the text features as a query. The transform layer 310 may receive the second high-level feature representation 222 directly from the second encoder 220 or from a corresponding previous transform layer 310. As shown in Figure 3, the first transform layer 310a generates the first transform layer output 312a by performing cross-attention on the second high-level feature representation 222 by using the self-attention performed on the text features from the initial alignment 234 as a query. Furthermore, the second transformation layer 310b and the third transformation layer 310c generate the second transformation layer output 312b and the third transformation layer output 312c, respectively, by performing cross-attention on the second higher-level feature representation 222 using the self-attention performed on the text features from each transformation layer output 312 as a query.

[0040] The final transform layer 310 of the multiple transform layers provides a transform layer output 312 to a final softmax layer 320 configured to predict the final hypothesis 120b. As shown in Figure 3, the third transform layer 310c is the final transform layer 310 of the multiple transform layers 310, whereby the third transform layer 310c sends a third transform layer output 312c to the final softmax layer 320. The non-autoregressive decoder 300 may send the final hypothesis 120b to the user device 10 (Figure 1).

[0041] The final hypothesis output 120b output by the non-autoregressive decoder 300 may include a probability distribution for possible new alignments 324, where each "possible new alignment 324" corresponds to a sequence of output labels / frames, each corresponding to a whitespace symbol or hypothetical subword unit. The probability distribution output by the non-autoregressive decoder 300 may include a posterior probability value for each different output label in each output frame of the sequence of output frames. Thus, if there are 100 different output labels representing different graphemes, whitespace symbols, or other symbols, the new alignment 324 output by the non-autoregressive decoder 300 may include 100 different probability values, one for each output label, in each output frame of the sequence of output frames. In some cases, the non-autoregressive decoder 300 outputs a single output label having the highest corresponding probability value in each output frame. In these examples, the non-autoregressive decoder 300 may simultaneously output a single output label having the highest corresponding probability value in each output frame (e.g., performing parallel greedy decoding). Alternatively, the transducer decoder 230 may select a null symbol as each output frame in the sequence of output frames based on determining that the corresponding probability of each of the hypothesized subword units does not meet a threshold probability value.

[0042] Although the probability distributions output by the non-autoregressive decoder 300 may be similar to those output by the transducer decoder 230, the posterior probability values ​​may differ for each output frame due to the additional processing performed by the non-autoregressive decoder 300 using multiple transform layers 310 and the second higher-order feature representation 222. That is, the non-autoregressive decoder 300 improves the initial alignment 234 by using the second higher-order feature representation 222 and the transform layer output 312 to generate a new alignment 324. More specifically, the non-autoregressive decoder 300 may improve the initial alignment 234 by deleting one or more output labels of the initial alignment 234. The non-autoregressive decoder 300 may also improve the initial alignment 234 by inserting or substituting one or more of the rescored sequence of output labels of the new alignment 324 into the sequence of output labels of the initial alignment 234. For example, the non-autoregressive decoder 300 may generate a new alignment 324 of "φ_pullφφ_pamp er sφφ" by receiving the initial alignment 234 "φ_pullφφ_pamp er sφφ" and the corresponding second higher-level feature representation 222. In this example, the non-autoregressive decoder 300 not only removes whitespace from the beginning of the initial alignment 234 but also adds whitespace to the end of the initial alignment 234, and replaces the hypothesized subword unit "pamp" with the hypothesized subword unit "camp" to generate the new alignment 324. Therefore, the new alignment 324 improves the error of the initial alignment 234, so that the new alignment 324 correctly corresponds to the utterance 106 "pull campers."

[0043] In some examples, the non-autoregressive decoder 300 generates a final transcription of the final hypothesis 120b based on the new alignment 324. In particular, the final transcription of the final hypothesis 120b comprises a sequence of output labels, each corresponding to a subword unit of the hypothesis. Thus, a difference between the final transcription of the final hypothesis 120b and the new alignment 324 of the final hypothesis 120b is that the output labels of the new alignment 324 may include whitespace characters, while the final transcription does not include whitespace characters. Thus, the non-autoregressive decoder 300 may generate the final transcription by removing all whitespace characters from the new alignment 324. Continuing with the above example, the transducer decoder 230 may generate a transcription of "pull campers" using the new alignment 324 by removing all of the whitespace characters φ.

[0044] Although FIG. 3 only shows the non-autoregressive decoder 300 performing an initial refinement step to generate a new alignment 324, it is understood that the non-autoregressive decoder 300 may perform one or more (e.g., any number) additional refinement steps. During each additional refinement step following the initial refinement step (FIG. 3), the non-autoregressive decoder 300 is configured to receive the new alignment 324 generated during the previous refinement step to generate another new alignment for the rescored sequence of output labels. For example, a second refinement step (e.g., a step following the initial refinement step in FIG. 3) would receive the new alignment 324 generated during the initial refinement step. Thus, in this example, the non-autoregressive decoder 300 uses the new alignment 324 (e.g., rather than the initial alignment 234) as input to the first transform layer 310a. In some implementations, the non-autoregressive decoder 300 performs a predetermined number of refinement steps before outputting the final hypothesis 120b to the user device 10 (FIG. 1). In other embodiments, the non-autoregressive decoder 300 continues to perform additional refinement steps until the new alignment 324 meets a confidence threshold.

[0045] 4 is a flowchart illustrating an example sequence of operations for a method 400 for performing streaming RNN-T considerations with non-autoregressive decoding. Method 400 may be performed by data processing hardware 510 (FIG. 5) using instructions stored in memory hardware 520 (FIG. 5). Data processing hardware 510 and memory hardware 520 may reside on user device 10 and / or remote computing device 60 of FIG. 1, which corresponds to computing device 500 (FIG. 5).

[0046] At operation 402, the method 400 includes receiving an initial alignment 234 of candidate hypotheses 120a generated by the transducer decoder model 230 during a first pass, where the candidate hypotheses 120a correspond to candidate transcriptions of the utterance 106. The candidate transcriptions include a sequence of output labels, each corresponding to a hypothesized subword unit. Meanwhile, the initial alignment 234 of the candidate hypotheses 120a includes a sequence of output labels, each corresponding to a whitespace symbol or a hypothesized subword unit. At operation 404, the method 400 includes receiving a second higher-level feature representation 222 (e.g., a subsequent sequence of audio encoding) characterizing the utterance 106. At operation 406, the method 400 includes generating a new alignment 324 for the rescored sequence of output labels during the initial refinement step using the non-autoregressive decoder 300. In particular, the non-autoregressive decoder 300 is configured to receive the initial alignment 234 of the candidate hypotheses 120a already generated by the transducer decoder model 230 during the first pass and the second high-level feature representation 222. Furthermore, the non-autoregressive decoder 300 may generate the final hypothesis 120b by removing whitespace symbols from the sequence of output labels of the new alignment 324.

[0047] 5 is a schematic diagram of an example computing device 500 that may be used to implement the systems and methods described in this document. Computing device 500 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the implementation of the invention(s) described and / or claimed in this document.

[0048] Computing device 500 includes processor 510, memory 520, storage device 530, high-speed interface / controller 540 connecting to memory 520 and high-speed expansion port 550, and low-speed interface / controller 560 connecting to low-speed bus 570 and storage device 530. Each of components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be mounted on a common motherboard or otherwise mounted as desired. Processor 510 processes instructions for execution within computing device 500, including instructions stored in memory 520 or on storage device 530, and can display graphical information for a graphical user interface (GUI) on an external input / output device, such as display 580 connected to high-speed interface 540. Other implementations may use multiple processors and / or multiple buses, along with multiple memories and types of memories, as desired. Also, multiple computing devices 500 may be connected, each providing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0049] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory, read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disks or tapes, etc.

[0050] Storage device 530 is enabled to provide mass storage to computing device 500. In some implementations, storage device 530 is a computer-readable medium. In various different implementations, storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, or an array of devices comprising a tape device, flash memory, or other similar solid-state memory device, or devices in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product comprises instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.

[0051] High-speed controller 540 manages bandwidth-intensive operations of computing device 500, while low-speed controller 560 manages lower-bandwidth intensive operations. Such role assignments are merely exemplary. In some implementations, high-speed controller 540 is coupled to memory 520, display 580 (e.g., via a graphics processor or accelerator), and high-speed expansion port 550, which may accept various expansion cards (not shown). In some implementations, low-speed controller 560 is coupled to storage device 530 and low-speed expansion port 590. Low-speed expansion port 590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled, for example, via a network adapter, to one or more input / output devices, such as a keyboard, pointing device, scanner, etc., or to a networking device, such as a switch or router.

[0052] The computing device 500, as shown in the figure, may be implemented in a variety of forms, such as as a standard server 500a, or multiple such servers 500a in a group, as a laptop computer 500b, or as part of a rack server system 500c.

[0053] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or readable on a programmable system having at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0054] These computer programs (also known as programs, software, software applications, or code) comprise machine instructions for a programmable processor and may be implemented in a high-level procedural programming language, and / or an object-oriented programming language, and / or an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic circuit (PLD)) used to provide machine instructions and / or data to a programmable processor, comprising a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0055] The processes and logic flows described herein may be performed by one or more programmable processors (also called data processing hardware) executing one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs may include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random-access memory, or both. Essential elements of a computer include a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or is operably coupled to receive data from or transfer data to a mass storage device, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0056] To provide for user interaction, one or more aspects of the present disclosure can be implemented in a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touchscreen, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Similarly, other types of devices can be used to provide for user interaction. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user. For example, a computer may interact by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0057] Although several embodiments have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (400), when executed by data processing hardware (510), causes the data processing hardware (510) to perform operations, the operations including: receiving an initial alignment (234) of candidate hypotheses (120a) generated by a transducer-decoder model (230) during a first pass based on an initial sequence (212) of audio encodings characterizing the utterance (106), the candidate hypotheses (120a) corresponding to candidate transcriptions of the utterance (106), the initial alignment (234) of the candidate hypotheses (120a) comprising a sequence of output labels each corresponding to a whitespace symbol or a hypothetical subword unit; receiving a subsequent sequence (222) of audio encodings characterizing the utterance (106); generating a new alignment (324) for the rescored sequence of output labels using a non-autoregressive decoder (300) configured to receive the initial alignment (234) of the candidate hypotheses (120a) generated by the transducer decoder model (230) during the first pass and the subsequent sequence of audio encodings (222) during an initial refinement step; during each of one or more additional refinement steps following the initial refinement step, generating a new alignment (324) for the rescored sequence of output labels by using the non-autoregressive decoder (300) configured to receive the new alignment (324) of the rescored sequence of output labels generated during a previous refinement step; A computer-implemented method (400) comprising:

2. The non-autoregressive decoder (300) comprises a plurality of transform layers (310), each of which comprises: performing self-attention on text features associated with the initial alignment (234); providing a transformation layer output (312) by performing cross-attention on a subsequent sequence (222) of the audio encodings, representing both keys and values, using the self-attention performed on the text features as a query; configured to perform The computer-implemented method (400) of claim 1.

3. Each of the plurality of transformation layers (310) subsequent to the initial transformation layer (310) receives the transformation layer output (312) from the corresponding previous transformation layer (310) as the text feature. The computer-implemented method (400) of claim 2.

4. a final transform layer (310) of the plurality of transform layers (310) provides the transform layer output (312) to a final softmax layer (320) configured to predict the new alignment (324) of the rescored sequence of output labels; The computer-implemented method (400) of claim 2.

5. generating the new alignment (324) for the rescored sequence of output labels comprises inserting, deleting, or substituting one or more output labels of the initial alignment (234) of the candidate hypotheses (120a); The computer-implemented method (400) of any one of claims 1 to 4.

6. The operations further include generating, during the first pass, by a causal encoder (210) the initial sequence of audio encodings (212) based on a sequence of acoustic frames (110) corresponding to speech (106). The computer-implemented method (400) of any one of claims 1 to 4.

7. the subsequent sequence of audio encodings (222) are encoded by a non-causal encoder (220) based on the initial sequence of audio encodings (212); The computer-implemented method (400) of claim 6.

8. The transducer decoder model (230) generates the candidate hypotheses (120a) by using the initial sequence (212) of the audio encoding. The computer-implemented method (400) of claim 6.

9. the candidate transcription of the candidate hypothesis (120a) comprises a sequence of output labels each corresponding to a hypothesized subword unit; The computer-implemented method (400) of any one of claims 1 to 4.

10. data processing hardware (510); and memory hardware (520) in communication with the data processing hardware (510), the memory hardware (520) storing instructions that, when executed by the data processing hardware (510), cause the data processing hardware (510) to perform operations, the operations including: receiving an initial alignment (234) of candidate hypotheses (120a) generated by a transducer-decoder model (230) during a first pass based on an initial sequence (212) of audio encodings characterizing the utterance (106), the candidate hypotheses (120a) corresponding to candidate transcriptions of the utterance (106), the initial alignment (234) of the candidate hypotheses (120a) comprising a sequence of output labels each corresponding to a whitespace symbol or a hypothetical subword unit; receiving a subsequent sequence (222) of audio encodings characterizing the utterance (106); generating a new alignment (324) for the rescored sequence of output labels using a non-autoregressive decoder (300) configured to receive the initial alignment (234) of the candidate hypotheses (120a) generated by the transducer decoder model (230) during the first pass and the subsequent sequence of audio encodings (222) during an initial refinement step; during each of one or more additional refinement steps following the initial refinement step, generating a new alignment (324) for the rescored sequence of output labels by using the non-autoregressive decoder (300) configured to receive the new alignment (324) of the rescored sequence of output labels generated during a previous refinement step; A system (100) comprising:

11. The non-autoregressive decoder (300) comprises a plurality of transform layers (310), each of which comprises: performing self-attention on text features associated with the initial alignment (234); providing a transformation layer output (312) by performing cross-attention on a subsequent sequence (222) of the audio encodings, representing both keys and values, using the self-attention performed on the text features as a query; configured to perform The system (100) of claim 10.

12. Each of the plurality of transformation layers (310) subsequent to the initial transformation layer (310) receives the transformation layer output (312) from the corresponding previous transformation layer (310) as the text feature. The system (100) of claim 11.

13. a final transform layer (310) of the plurality of transform layers (310) provides the transform layer output (312) to a final softmax layer (320) configured to predict the new alignment (324) of the rescored sequence of output labels; The system (100) of claim 11.

14. generating the new alignment (324) for the rescored sequence of output labels comprises inserting, deleting, or substituting one or more output labels of the initial alignment (234) of the candidate hypotheses (120a); A system (100) according to any one of claims 10 to 13.

15. The operations further include generating, during the first pass, by a causal encoder (210) the initial sequence of audio encodings (212) based on a sequence of acoustic frames (110) corresponding to speech (106). A system (100) according to any one of claims 10 to 13.

16. the subsequent sequence of audio encodings (222) are encoded by a non-causal encoder (220) based on the initial sequence of audio encodings (212); The system (100) of claim 15.

17. The transducer decoder model (230) generates the candidate hypotheses (120a) by using the initial sequence (212) of the audio encoding. The system (100) of claim 15.

18. the candidate transcription of the candidate hypothesis (120a) comprises a sequence of output labels each corresponding to a hypothesized subword unit; A system (100) according to any one of claims 10 to 13.

Images