Reducing Streaming ASR Model Delay Using Self-Alignment

Abstract

The streaming speech recognition model (200) includes an audio encoder (210) configured to receive a sequence of acoustic frames (110) and generate a high-order feature representation (202) for a corresponding acoustic frame in the sequence of acoustic frames. The streaming speech recognition model also includes a label encoder (220) configured to receive a sequence of non-blank symbols (242) output by a final softmax layer (240) and generate a dense representation (222). The streaming speech recognition model also includes a joint network (230) configured to receive as input the high-order feature representation generated by the audio encoder and the dense representation generated by the label encoder and generate a probability distribution (232) over possible speech recognition hypotheses. Here, the streaming speech recognition model is trained to reduce prediction delay by using self-alignment to prompt an alignment path one frame left from a reference forced alignment frame.

Description

[Technical field]

[0001] The present disclosure relates to reducing streaming automatic speech recognition (ASR) model delay using self-alignment. [Background technology]

[0002] Automatic speech recognition (ASR), the process of taking audio input and transcribing it into text, is a very important technology used by mobile and other devices. In general, ASR attempts to provide an accurate transcription of what a person said by taking an audio input (e.g., an utterance) and transcribing the audio input into text. Modern ASR models continue to improve both accuracy (e.g., low word error rate (WER)) and latency (e.g., delay between a user's utterance and the transcription) based on the ongoing development of deep neural networks. The current use of ASR systems requires that the ASR system decodes the utterance in a streaming manner that is comparable to or sometimes faster than real-time, but is also accurate. However, streaming end-to-end models that optimize sequence likelihood without delay constraints introduce a large delay between the audio input and the predicted text due to such models learning to improve their predictions by using more distant future context. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent Application No. 17 / 210465 Summary of the Invention [Means for solving the problem]

[0004] One aspect of the present disclosure provides a streaming speech recognition model including an audio encoder configured to receive as input a sequence of acoustic frames and generate a high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames at each of a plurality of time steps. The streaming speech recognition model also includes a label encoder configured to receive as input a sequence of non-blank symbols output by a final softmax layer and generate a dense representation at each of the plurality of time steps. The streaming speech recognition model also includes a joint network configured to receive as input the high-order feature representation generated by the audio encoder at each of the plurality of time steps and the dense representation generated by the label encoder at each of the plurality of time steps and generate a probability distribution over possible speech recognition hypotheses at the corresponding time step at each of the plurality of time steps. Here, the streaming speech recognition model is trained to reduce prediction delay by prompting an alignment path one frame left of a reference forced alignment frame at each time step for each training batch using self-alignment.

[0005] Implementations of the present disclosure may include one or more of any of the following features: In some implementations, the streaming speech recognition model includes a transformer-transducer model. In these implementations, the audio encoder may include a stack of transformer layers, each of which includes a normalization layer, a masked multi-head attention layer with relative position coding, a residual connection, a stacking / unstacking layer, and a feedforward layer. Here, the stacking / unstacking layer may be configured to change the frame rate of the corresponding transform layer to adjust the processing time by the transformer-transducer model during training and inference. In some examples, the label encoder includes a stack of transformer layers, each of which includes a normalization layer, a masked multi-head attention layer with relative position coding, a residual connection, a stacking / unstacking layer, and a feedforward layer.

[0006] In some cases, the label encoder may include a bigram embedding lookup decoder model. In some examples, the streaming speech recognition model includes one of a recurrent neural-transducer (RNN-T) model, a transformer-transducer model, a convolutional network-transducer (ConvNet-transducer) model, or a conformer-transducer model. Training the streaming speech recognition model to reduce prediction delay using self-alignment may include using self-alignment without constraining the alignment of the decoding graph using an external aligner model. In some implementations, the streaming speech recognition model runs on a user device or a server. In some examples, each acoustic frame in the sequence of acoustic frames includes a x-dimensional feature vector.

[0007] Another aspect of the disclosure provides a computer-implemented method for causing data processing hardware to perform operations for training a streaming speech recognition model to reduce prediction delay using self-alignment when executed on the data processing hardware. The operations include receiving a sequence of acoustic frames corresponding to an utterance as an input to the streaming speech recognition model. The streaming speech recognition model is configured to learn an alignment probability between the sequence of acoustic frames and an output sequence of label tokens. The operations also include generating a speech recognition result for the utterance as an output from the streaming speech recognition model. The speech recognition result includes generating an output sequence of label tokens using a decoding graph. The operations also include generating a speech recognition model loss based on the speech recognition result and a ground truth transcription of the utterance. The operations also include obtaining a reference-forced alignment path from the decoding graph that includes a reference-forced alignment frame, and identifying one frame to the left of each reference-forced alignment frame in the reference-forced alignment path from the decoding graph. The operations also include summing label transition probabilities based on the identified frames to the left from each forced alignment frame in the reference forced alignment path, and updating a streaming speech recognition model based on the sum of the label transition probabilities and the speech recognition model loss.

[0008] Implementations of the present disclosure may include one or more of any of the following features. In some implementations, the operations further include: generating, by an audio encoder of the streaming speech recognition model, a high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames at each of the multiple time steps; receiving, as an input to a label encoder of the streaming speech recognition model, a sequence of non-blank symbols output by the final softmax layer; generating, by the label encoder, a dense representation at each of the multiple time steps; receiving, as an input to a joint network of the streaming speech recognition model, the high-order feature representation generated by the audio encoder at each of the multiple time steps and the dense representation generated by the label encoder at each of the multiple time steps; and generating, by the joint network, a probability distribution at each of the multiple time steps over possible speech recognition hypotheses at the corresponding time step. In some examples, the label encoder includes a stack of transformer layers, each of which includes a normalization layer, a masked multi-head attention layer with relative position encoding, a residual connection, a stacking / unstacking layer, and a feedforward layer. The label encoder may include a bigram embedding lookup decoder model.

[0009] In some implementations, the streaming speech recognition model includes a transformer-transducer model. The audio encoder may include a stack of transformer layers, each of which includes a normalization layer, a masked multi-head attention layer with relative position coding, a residual connection, a stacking / unstacking layer, and a feedforward layer. Here, the stacking / unstacking layers may be configured to change the frame rate of the corresponding transformer layer to adjust the processing time by the transformer-transducer model during training and inference.

[0010] In some implementations, the streaming speech recognition model includes one of a recurrent neural-transducer (RNN-T) model, a transformer-transducer model, a convolutional network-transducer (ConvNet-transducer) model, or a conformer-transducer model. The streaming speech recognition model may be executed on a user device or a server. In some examples, the operations further include training the streaming speech recognition model to reduce prediction delay using self-alignment without constraining alignment of the decoding graph using an external aligner model.

[0011] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of an audio environment implementing a transducer model for performing streaming speech recognition. [Diagram 2] FIG. 1 is a schematic diagram of an exemplary transducer model architecture. [Diagram 3] 1 is a plot of an example decoding graph showing self-aligned and forced-aligned paths. [Figure 4] FIG. 2 is a schematic diagram of an example transformer architecture. [Diagram 5] 1 is a flowchart of an example configuration of operations for a method for reducing streaming ASR model delay using self-alignment. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Like reference numbers in the various drawings indicate like elements.

[0014] Automatic speech recognition (ASR) systems target not only quality / accuracy (e.g., low word error rate (WER)) but also low latency (e.g., short delay between user utterance and the emerging transcription). Recently, end-to-end (E2E) ASR models have been widely utilized in achieving state-of-the-art performance in accuracy and latency. In contrast to traditional hybrid ASR systems that include separate acoustic, pronunciation, and language models, E2E models apply a sequence-to-sequence approach to jointly learn acoustic and language modeling in a single neural network that is trained end-to-end from training data, e.g., speech-transcription pairs. Here, E2E models refer to models whose architecture is entirely composed of neural networks. A complete neural network works without external components and / or manually designed components (e.g., finite-state transducers, dictionaries, or text normalization modules). Also, when training E2E models, these models generally do not require bootstrapping from decision trees or time alignment from a separate system.

[0015] Currently, when using an ASR system, the ASR system may be required to decode the utterance in a streaming manner, which corresponds to displaying a description of the utterance in real time or even faster than real time as the user speaks. As an example, when the ASR system is displayed on a user computing device that experiences direct user interaction, such as a mobile phone, an application (e.g., a digital assistant application) that runs on the user device and uses the ASR system needs to stream speech recognition so that words, word pieces, and / or individual characters are displayed on the screen as they are spoken. Users of user devices may also have a low tolerance for latency. For example, when a user speaks a query that requests a digital assistant to retrieve details from a calendar application to check an appointment, the user wants the digital assistant to provide a response conveying the retrieved details as quickly as possible. Due to this low tolerance, the ASR system works hard on the user device to minimize the impact of latency and inaccuracies that may adversely affect the user's experience.

[0016] One form of sequence-to-sequence model, called recurrent neural network transducer (RNN-T), does not use an attention mechanism, and unlike other sequence-to-sequence models that generally must process an entire sequence (audio waveform) to generate an output (e.g., a sentence), RNN-T processes input samples continuously and streams output symbols. This makes it particularly attractive for real-time communications. For example, speech recognition with an RNN-T may output characters one by one in response to an utterance. Here, the RNN-T predicts the next symbol using a feedback loop that feeds the symbol predicted by the model back to itself. Because decoding an RNN-T involves a beam search with a single neural network rather than a large decoder graph, the RNN-T may scale to a fraction of the size of a server-based speech recognition model. The size reduction allows the entire RNN-T to be deployed on a device and may even be able to operate offline (i.e., without a network connection), thus avoiding unreliability issues with communication networks. RNN-T models, which utilize long short-term memory (LSTM) to provide a sequence encoder, are generally suitable for providing streaming transcription capabilities that recognize conversational queries (e.g., "set the timer," "don't forget to buy milk," etc.) as well as latency-sensitive applications, but have limited ability to look ahead to audio-text, thereby still lagging behind state-of-the-art conferencing models (e.g., server-based models with separate AM, PM, and LM) and attention-based sequence-to-sequence models (e.g., Listen and Attend Spell (LAS)) in terms of quality (e.g., speech recognition accuracy, often measured by Word Error Rate (WER)).

[0017] Recently, transformer-transducer (TT) and conformer-transducer (CT) model architectures have been introduced to further improve the RNN-T model architecture by replacing the LSTM layer in the audio encoder and / or prediction network with the respective transformer or conformer layer. In general, the TT and CT model architectures can access future audio frames (e.g., right context) when calculating self-attention in their respective transformer or conformer layers. Thus, the TT and CT model architectures may utilize the future right context to operate in a non-streaming transcription mode to improve speech recognition performance when the latency constraint is relaxed. That is, the duration of the prediction delay is proportional to the amount of audio frames accessed in the future. However, similar to the RNN-T, the TT and CT model architectures may also operate in a streaming transcription mode where the self-attention depends only on past acoustic frames (e.g., left context).

[0018] Streaming speech recognition models such as transducer models (e.g., RNN-T, TT, and CT) optimize sequence likelihood without delay constraints, and therefore have a large delay between audio input and predicted text, as these models learn to improve prediction by using more distant future context. Recent approaches to reduce prediction delay include constrained alignment techniques that penalize word boundaries based on audio alignment information obtained from an external alignment model by masking alignment paths that exceed a certain threshold delay. This technique is effective in reducing the latency of streaming end-to-end models, but requires a highly accurate external alignment model to minimize WER degradation, which may further complicate the model training step. Other techniques that blindly reduce delay by selecting the most efficient direction in the RNN-T decoding graph often select a direction that is not optimal for all audio inputs due to the lack of alignment information, which may further exacerbate the trade-off between delay and WER.

[0019] To mitigate the drawbacks associated with simply blindly shortening delay by using an external alignment model or selecting the most efficient direction from the decoding graph, implementations herein are directed to shortening predicted delay in streaming speech recognition models by using self-alignment. In particular, self-alignment does not require the use of an external alignment model or blindly optimizes delay, but instead utilizes a reference-enforced alignment learned from a trained speech recognition model to select the optimal low-latency direction that shortens delay. The reference-enforced alignment may include a Viterbi-enforced alignment. That is, self-alignment always identifies a path in the decoding graph that is one frame to the left of the Viterbi-enforced alignment at each time step. Self-alignment has advantages over existing methods for constraining delay. First, since self-alignment does not require an external alignment model, the training complexity for self-alignment is much lower than supervised methods. Second, self-alignment has minimal impact on ASR training by constraining only the most probable alignment paths. In contrast, other methods affect many alignment paths by masking them or by changing the weights on the label transition probabilities. The delay-constrained regularization term conflicts with the main ASR loss, so it is important to minimize its intervention on the main loss in optimizing the delay-performance tradeoff. Self-alignment only regularizes a single path by pushing it leftward.

[0020] 1 is an example of an audio environment 100. In the audio environment 100, a method for a user 104 to interact with a computing device, such as a user device 10, may be via voice input. The user device 10 (also commonly referred to as device 10) is configured to capture voice (e.g., streaming audio data) from one or more users 104 in the audio environment 100. Here, the streaming audio data may refer to utterances 106 by the user 104 that act as audible queries, commands for the device 10, and audible communications captured by the device 10. The voice-enabled system of the device 10 may process the queries or commands by answering the queries and / or executing / completing the commands by one or more downstream applications.

[0021] The user device 10 may represent any computing device associated with a user 104 and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, a mobile device (e.g., a mobile phone, a tablet, a laptop, etc.), a computer, a wearable device (e.g., a smart watch), a smart appliance, an Internet of Things (IoT) device, a vehicle infotainment system, a smart display, a smart speaker, etc. The user device 10 includes a data processing hardware 12 and a memory hardware 14 in communication with the data processing hardware 12, which stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 having audio capture devices (e.g., microphones) 16, 16a for capturing and converting speech 106 in the audio environment 100 into electrical signals, and audio output devices (e.g., speakers) 16, 16b for communicating audible audio signals (e.g., as output audio data from the device 10). Although the user device 10 implements a single audio capture device 16a in the illustrated example, the user device 10 may implement an array of audio capture devices 16a without departing from the scope of this disclosure, and one or more capture devices 16a in the array may communicate with the audio system 16 without being physically present on the user device 10.

[0022] In the speech environment 100, an automatic speech recognition (ASR) system 118 transducer model 200 resides on a user device 10 of a user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system executing in a cloud-computing environment) that communicates with the user device 10 via a network 40. The user device 10 and / or the remote computing device 60 also include an audio subsystem 108 configured to receive utterances 106 by the user 104 captured by an audio capture device 16a and convert the utterances 106 into a corresponding digital format related to input acoustic frames 110 that can be processed by the ASR system 118. In the illustrated example, the user utters respective utterances 106, and the audio subsystem 108 converts the utterances 106 into corresponding audio data (e.g., acoustic frames) 110 that are input to the ASR system 118. The transducer model 200 then receives as input audio data 110 corresponding to the utterance 106 and generates / predicts as output a corresponding transcription 120 (e.g., recognition result / hypothesis) of the utterance 106. The transducer model 200 provides streaming speech recognition model results without access to look-ahead audio, and thus provides streaming transcription capabilities in real-time as the user 104 is uttering the utterance 106. For example, a digital assistant application 50 running on the user device 10 may need to stream speech recognition such that words, word pieces, and / or individual characters are displayed on a screen as they are spoken.

[0023] The user device 10 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of the transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 may display partial speech recognition results 120a in a streaming manner for time 1 and then display final speech recognition results 120b for time 2. In some configurations, the transcription 120 output from the ASR system 118 is processed by, for example, a natural language understanding (NLU) module executing on the user device 10 or the remote computing device 60 to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device 10 or the remote computing device 60) may convert the transcription into synthetic speech for audible output by the user device 10 and / or another device.

[0024] In the illustrated example, a user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) of the user device 10 that uses an ASR system 118. For example, FIG. 1 shows the user 104 communicating with the digital assistant application 50, which displays a digital assistant interface 18 on the screen of the user device 10, and illustrates a conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, "What time is the concert tonight?" This question from the user 104 is an utterance 106 that is captured by the voice capture device 16a and processed by the audio system 16 of the user device 10. In this example, the audio system 16 receives the utterance 106 and converts it into acoustic frames 110 that are input to the ASR system 118.

[0025] Continuing with the example, while receiving acoustic frames 110 corresponding to an utterance 106 as the user 104 speaks, the transducer model 200 encodes the acoustic frames 110 and then decodes the encoded acoustic frames 110 into partial speech recognition results 120a. During time 1, the user interface generator 107 presents a representation of the partial speech recognition results 120a of the utterance 106 to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming manner, whereby words, word pieces, and / or individual characters are displayed on the screen as they are spoken. In some examples, the first look-ahead audio context is equal to zero.

[0026] During time 2, the user interface generator 107 presents a representation of the final speech recognition result 120b of the utterance 106 to the user 104 of the user device 10 via the digital assistant interface 18. The final speech recognition result 120b may simply be the partial speech recognition result 120a when the user finishes speaking. In some cases, the ASR system 118 may include another speech recognition and / or use an external language model to rescore the partial speech recognition result. In some cases, the same transducer model 200 may process the audio again after the user finishes speaking, instead utilizing the right look-ahead audio context to generate the final speech recognition result 120b. This disclosure is not related to how the final speech recognition result 120b is obtained, but instead is directed to limiting the delay in the streaming partial speech recognition result 120a output by the transducer model 200.

[0027] In the example shown in FIG. 1, the digital assistant application 50 may use natural language processing to respond to a question posed by the user 104. Natural language processing generally refers to the process of interpreting written language (e.g., partial speech recognition results 120a and / or final speech recognition results 120b) and determining whether the written language prompts some action. In this example, the digital assistant application 50 uses natural language processing to recognize that the question from the user 104 is a question about the user's schedule, and more specifically, a question about a concert on the user's schedule. By recognizing such details through natural language processing, the automated assistant returns a response 19 to the user's query, in this case, the response 19 is presented as "The venue opens at 6:30 p.m. and the concert starts at 8 p.m." In some configurations, the natural language processing is performed on a remote server 60 that communicates with the data processing hardware 12 of the user device 10.

[0028] Referring to FIG. 2, the transducer model 200 may enable end-to-end (E2E) speech recognition by incorporating an acoustic model, a pronunciation model, and a language model into a single neural network, where neither a dictionary nor a separate text normalization component is required. Various structures and optimization mechanisms can increase accuracy and reduce model training time. In the illustrated example, the transducer model 200 includes a transformer-transducer (TT) model architecture, which adheres to latency constraints associated with interactive applications. The TT model 200 has a small computational footprint and lower memory requirements than traditional ASR architectures, making the TT model architecture suitable for performing speech recognition on the entire user device 10 (e.g., no communication with a remote server 60 is required). The TT model 200 includes an audio encoder 210, a label encoder 220, and a joint network 230. The audio encoder 210 is generally similar to an acoustic model (AM) in a conventional ASR system and includes a neural network with multiple transformer layers. For example, the audio encoder 210 may generate a sequence of d-dimensional feature vectors (e.g., audio frames 110 (FIG. 1)) x = (x1, x2, ..., x T ), where

number

[0029] Similarly, the label encoder 220 may also include a neural network or lookup table embedding model in the transformer layer, which, like a language model (LM), embeds the sequence y 0 , ..., y ui-1 We use a dense representation 222 (e.g., Ih u In implementations where the label encoder 220 includes a neural network of transform layers, each transform layer may include a normalization layer, a masked multi-head attention layer with relative position encoding, a residual connection, a feedforward layer, and a dropout layer. In these implementations, the label encoder 220 may include two transform layers. In implementations where the label encoder 220 includes a lookup table embedding model with bigram label contexts, the embedding model is configured to learn a d-dimensional weight vector for each possible bigram label context, where d is the dimension of the output of the audio encoder 210 and the label encoder 220. In some examples, the total number of parameters in the embedding model is N 2 ×d, where N is the vocabulary size for the labels. Here, the learned weight vector is then used as an embedding of the bigram label context in the TT model 200 to generate a fast label encoder 220 running time.

[0030] Finally, in the TT model architecture, the representations generated by the audio encoder 210 and the label encoder 220 are fed into a dense layer J u,t The joint network 230 then predicts the alignment distribution (e.g., alignment probability 232) over the next output symbol as follows: Pr(z u,t |x,t,y1,…, y u-1) (1) In the above equation, x is the audio input, y is the ground truth label sequence, and z is the alignment that belongs to y. In other words, the joint network 230 generates a probability distribution 232 over possible speech recognition hypotheses at each output step (e.g., time step), where a "probable speech recognition hypothesis" corresponds to a set of output labels (also called "phonetic units"), each representing a grapheme (e.g., symbol / character) or word piece in a specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 characters in the English alphabet and one label designating a space. Thus, the joint network 230 may output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels. This set of values ​​may be a vector (e.g., a one-hot vector) and may indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation and other symbols), although the set of output labels is not so limited. For example, the set of output labels can include word pieces and / or entire words in addition to or instead of graphemes. The output distribution of the joint network 230 can include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels, each representing a different grapheme or other symbol, the output z of the joint network 230 can include a posterior probability value for each of the different output labels. u,t can contain 100 different probability values, one for each output label. The probability distributions can then be used in a beam search process (e.g., by a softmax layer 240) to select and assign scores to candidate orthogonal elements (e.g., graphemes, wordpieces, and / or words) to determine the transcription 120.

[0031] The softmax layer 240 may use any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol 242 predicted by the TT model 200 in the corresponding output step. Thus, the set of output symbols 242 predicted by the TT model 200 may be collectively referred to as the output sequence of label tokens 242. In this way, the TT model 200 makes no conditional independence assumptions and the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels that have been output so far.

[0032] To determine the log-conditional probability of y given an audio input x, sum over all the alignment distributions that correspond to y:

number

[0033] 2 illustrates a transducer model 200 including a TT model architecture, the transducer model 200 may include an RNN-T model architecture, a Convolutional Neural Network-Transducer (CNN-Transducer) model architecture, a Convolutional Network-Transducer (ConvNet-Transducer) model architecture, or a Conformer-Transducer model architecture without departing from the scope of this disclosure. An exemplary CNN-Transducer model architecture is described in detail in "Contextnet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context," https: / / arxiv.org / abs / 2005.03191, the contents of which are incorporated herein by reference in their entirety. An exemplary conformer-transducer model architecture is described in detail in "Conformer: Convolution-augmented transformer for speech recognition," https: / / arxiv.org / abs / 2005.08100, the contents of which are incorporated herein by reference in their entirety.

[0034] The transducer model 200 is trained on a training dataset of audio data corresponding to speech paired with a corresponding transcription. Training of the transducer model 200 may be performed on a remote server 60, and the trained transducer model 200 may be pushed to the user device 10. The transducer model 200 is trained using a cross-entropy error based on Viterbi forced alignment. The alignment delay includes the delay between the input audio frames and the streamed decoded output labels. Conventional models can learn accurate alignment after multiple iterations because they train the alignment model interactively with the realigned labels. TT or CT models that access future frames when calculating self-attention in each transformer or conformer layer may include an alignment delay consistent with conventional models. However, transducer models in streaming mode, where self-attention depends only on past frames, will experience an alignment delay.

[0035] Implementations herein are directed to reducing predicted delay in streaming transducer model 200 by using self-alignment. In particular, self-alignment does not require the use of an external alignment model and does not blindly optimize delay, but instead utilizes reference-enforced alignments learned from trained speech recognition models to select optimal low-latency directions that reduce delay. The reference-enforced alignments may include Viterbi-enforced alignments. That is, self-alignment always identifies a path in the decoding graph that is one frame to the left of the Viterbi-enforced alignment at each time step.

[0036] FIG. 3 shows a plot of a decoding graph 300 for a transducer model 200 with a TT model architecture for the output sequence of label tokens 242 (FIG. 2) “I like it”. The x-axis shows each acoustic frame at each time step, and the y-axis shows the output label tokens 242 (FIG. 2). The thick non-solid circles and arrows represent each token that is not included in the alignment path described below. The constrained alignment path 310 (represented by the thick circle and the thick arrow as shown in FIG. 3) includes a word boundary threshold value equal to 2. The forced alignment path 320 (represented by the dotted circle and the dotted arrow as shown in FIG. 3) learned from the trained transducer model 200 (also called the reference forced alignment path 320) and the left alignment path 330 (represented by the dashed circle, for example) include one frame to the left of every frame of the forced alignment path 320. During training of the transducer model 200, for each training batch, self-alignment always encourages the left alignment path 330 (represented by the dashed circle and dashed arrow as shown in FIG. 3) by pushing the model's forced alignment path 320 leftward. The training loss may be expressed as:

number

[0037] FIG. 4 illustrates an example transformer layer 400 among multiple transformer layers of the audio encoder 210. Here, during each time step, an initial transformer layer 400 receives as input a corresponding audio frame 110 and generates a corresponding output representation / embedding 450 that is received as input by the next transformer layer 400. That is, each transformer layer 400 following the initial transformer layer 400 may receive an input embedding 450 that corresponds to the output representation / embedding generated as output by the immediately preceding transformer layer 400. A final transformer layer 400 (e.g., the last transformer layer in the final stack 320) generates a high-order feature representation 202 (e.g., ah in relation to FIG. 2) for the corresponding audio frame 110 at each of multiple time steps. t (represented by

[0038] The input to the label encoder 220 (FIG. 2) is the sequence of non-blank symbols y0, ..., y ui-1 Thus, when the label encoder 220 includes a transformer layer, the initial transformer layer may receive the input embedding 111 by passing the one-hot vector through a lookup table.

[0039] Each transformer layer 400 of the audio encoder 210 includes a normalization layer 404, a masked multi-head attention layer 406 with relative position coding, a residual connection 408, a stacking / unstacking layer 410, and a feedforward layer 412. The masked multi-head attention layer 406 with relative position coding provides a flexible way to control the amount (i.e., duration) of look-ahead audio context used by the TT model 200. Specifically, after the normalization layer 404 normalizes the acoustic frame 110 and / or the input embeddings 111, the masked multi-head attention layer 406 projects the input to a value for all heads. The masked multi-head layer 406 may then mask the attention scores to the previous frames of the current acoustic frame 110 to generate an output that is conditional only on the previous acoustic frame 110. The weighted average values ​​for all heads are then concatenated and passed to dense layer 2 416 where the residual connections 414 are added to the normalized input and the output of the dense layer 416 to form the final output of the multi-head attention layer with relative position encoding 406. The residual connections 408 are added to the output of the normalization layer 404 by adder 430 and provided as input to the masked multi-head attention layer 406 or the feedforward layer 412, respectively. The stacking / unstacking layers 410 can be used to change the frame rate for each transformer layer 400 to accelerate training and inference.

[0040] The feedforward layer 412 applies a normalization layer 404, which is then sequentially applied to a dense layer 1 420, a normalized linear layer (ReLu) 418, and a dense layer 2 416. ReLu 418 is used as the activation for the output of dense layer 1 420. Similar to the multi-head attention layer with relative position coding 406, the residual connections 414 of the output from the normalization layer 404 are added to the output of dense layer 2 416 by a summer 430.

[0041] 5 illustrates a flowchart of an example configuration of operations for a method 500 of training a streaming speech recognition model to reduce prediction delay using self-alignment. The method includes receiving a sequence of acoustic frames 110 corresponding to an utterance 106 as an input to a streaming speech recognition model (e.g., a transducer model) 200 in operation 502. The streaming speech recognition model 200 is configured to learn alignment probabilities 232 between the sequence of acoustic frames 110 and an output sequence of label tokens 242. The method 500 includes generating a speech recognition result 120 for the utterance 106 including an output sequence of label tokens 242 using a decoding graph 300 as an output from the streaming speech recognition model 200 in operation 504. The method 500 includes generating a speech recognition model loss based on the speech recognition result 120 and a ground truth transcription of the utterance 106 in operation 506.

[0042] The method 500 includes, at operation 508, obtaining the reference forced alignment path 320 from the decoding graph 300. The method 500 includes, at operation 510, identifying from the decoding graph 300 one frame to the left of each reference forced alignment frame in the reference forced alignment path 320. The method 500 includes, at operation 512, summing label transition probabilities based on the identified frames to the left of each forced alignment frame. The method 500 includes, at operation 514, updating the streaming speech recognition model 200 based on the sum of the label transition probabilities and the speech recognition model loss.

[0043] 6 is a schematic diagram of an exemplary computing device 600 that may be used to implement the systems and methods described herein. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the invention described and / or claimed herein.

[0044] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be mounted on a common motherboard or otherwise mounted as needed. Processor 610 can process instructions executed within computing device 600, including instructions stored in memory 620 or on storage device 630 to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 680 coupled to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used along with multiple memories and multiple types of memories, as needed. Also, multiple computing devices 600 may be connected such that each device performs a portion of the required operations (eg, a bank of servers, a group of blade servers, or a multi-processor system).

[0045] The memory 620 stores information non-temporarily within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 620 may be a physical device used to temporarily or persistently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disks or tapes.

[0046] The storage device 630 can provide mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods as described above. The information carrier is a computer or machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0047] The high-speed controller 640 manages bandwidth-intensive operations for the computing device 600, while the low-speed controller 660 manages lower bandwidth-intensive operations. Such allocation of duties is merely exemplary. In some implementations, the high-speed controller 640 is coupled to the memory 620, the display 680 (e.g., via a graphics processor or accelerator), and the high-speed expansion port 650. The high-speed expansion port 650 may accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and the low-speed expansion port 690. The low-speed expansion port 690 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be coupled to one or more input / output devices, such as, for example, a keyboard, a pointing device, a scanner, or a networking device such as a switch or router via a network adapter.

[0048] Computing device 600 may be implemented in a number of different ways as shown: For example, computing device 600 may be implemented as a standard server 600a or multiple times within a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0049] Various implementations of the systems and techniques described herein may be realized in digital electrical and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor. The programmable processor may be dedicated or general purpose and may be coupled to receive and transmit data and instructions from a storage system, at least one input device, and at least one output device.

[0050] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0051] Non-transient memory may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) temporarily or persistently for use by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disks or tapes.

[0052] These computer programs (also known as programs, software, software applications or codes) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0053] The processes and logic flows described herein can be executed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by operating on input data to generate output. The processes and logic flows can also be executed by special purpose logic circuitry, for example, FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, and any one or more processors of any type of digital computer. In general, a processor receives instructions and data from a read-only memory or a random access memory, or both. The basic elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. In general, a computer also includes one or more mass storage devices, for example, magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to one or more mass storage devices, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, examples of which include semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD ROM disks. The processor and memory may be supplemented by or incorporated in special purpose logic circuitry.

[0054] To enable interaction with a user, one or more aspects of the present disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and possibly a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to enable interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, speech input, or tactile input. The computer can also interact with the user by sending documents to and receiving documents from a device being used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0055] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0056] 1-2 hours 10 User Devices 12 Data Processing Hardware 14 Memory Hardware 16 Audio System 16a Audio capture device 16b Audio Output Device 18 Digital Assistant Interface 19 responses 50 Digital Assistant Applications 60 Remote Computing Devices 100 Audio Environment 104 users 106 utterances 107 User Interface Generator 108 Audio Subsystem 110 Acoustic frames, audio data 111 Entered embedding 118 ASR System 120 Transcription 120a Partial speech recognition result 120b Final voice recognition result 200 TT model, transducer model 202 Higher-level feature representation 210 Audio Encoder 220 Label Encoder 230 Joint Network 232 Alignment Probability 240 Softmax Layer 242 output symbols, non-blank symbols, label tokens 300 Decoding Graph 310 Alignment Path 320 Forced Alignment Path 330 Left Alignment Path 400 Transformer Layer 404 Normalization layer 406 Masked Multi-Head Attention Layer 408, 414 Residual Connection 410 Stacking / Unstacking Layers 412 Feedforward Layer 416, 420 Dense layer 418 Normalized linear layer 430 Adder 450 Output Representation / Embedding 600 computing devices 600a Server 600b Laptop Computer 600c Rack Server System 610 Processor 620 Memory 630 Storage Devices 640 High Speed ​​Interface / Controller 650 High Speed ​​Expansion Port 660 Low Speed ​​Interface / Controller 670 Slow Bus 680 Display 690 Low Speed ​​Expansion Port

Claims

1. A computer program for causing a computer to realize functions of a streaming speech recognition model (200), the streaming speech recognition model (200) comprising: An audio encoder (210), receiving as input a sequence of acoustic frames (110); At each of a plurality of time steps, generate a high-level feature representation (202) for a corresponding acoustic frame (110) in the sequence of acoustic frames (110). An audio encoder (210) configured to: A label encoder (220), receiving as input the sequence of non-blank symbols (242) output by the final softmax layer (240); At each of the plurality of time steps, generate a dense representation (222). A label encoder (220) configured as follows: A joint network (230), receiving as input the high-level feature representation (202) generated by the audio encoder (210) at each of the plurality of time steps and the dense representation (222) generated by the label encoder (220) at each of the plurality of time steps; At each of the plurality of time steps, generating a probability distribution (232) over possible speech recognition hypotheses at the corresponding time step. A joint network (230) configured as follows: Equipped with The streaming speech recognition model (200) is trained to use self-alignment to reduce prediction delay by prompting the alignment path one frame to the left of a reference forced alignment frame at each time step for each training batch.

2. The computer program product of claim 1 , wherein the streaming speech recognition model (200) comprises a transformer-transducer model.

3. The audio encoder (210) comprises a stack of transformer layers (400), each of which: A normalization layer (404); a masked multi-head attention layer with relative position coding (406); Residual connection (408); a stacking / unstacking layer (410); A computer program product as claimed in claim 2, further comprising a feedforward layer (412).

4. 4. The computer program product of claim 3, wherein the stacking / unstacking layer (410) is configured to modify a frame rate of a corresponding transformer layer (400) to accommodate processing time by the transformer-transducer model during training and inference.

5. The label encoder (220) comprises a stack of transformer layers (400), each transformer layer (400) having: A normalization layer (404); a masked multi-head attention layer with relative position coding (406); Residual connection (408); a stacking / unstacking layer (410); A computer program product according to any one of claims 2 to 4, further comprising a feedforward layer (412).

6. The computer program product of claim 1 , wherein the label encoder (220) comprises a bigram embedding lookup decoder model.

7. The streaming speech recognition model (200) Recurrent Neural-Transducer (RNN-T) model, Transformer-Transducer Model, A Convolutional Network-Transducer (ConvNet-Transducer) model, or A computer program according to any one of claims 1 to 6, comprising one of the following: a conformer-transducer model.

8. 8. The computer program product of claim 1, wherein training the streaming speech recognition model to reduce prediction delay using self-alignment comprises using self-alignment without constraining alignment of the decoding graph using an external aligner model.

9. 9. The computer program product of claim 1, wherein the streaming speech recognition model (200) runs on a user device (10) or on a server (60).

10. 10. The computer program product of claim 1, wherein each acoustic frame (110) in the sequence of acoustic frames (110) comprises a dimensional feature vector.

11. 1. A computer-implemented method (500) that, when executed on data processing hardware (12), causes the data processing hardware (12) to perform operations for training a streaming speech recognition model (200) to reduce prediction delay using self-alignment, the operations comprising: receiving a sequence of acoustic frames (110) corresponding to an utterance (106) as an input to the streaming speech recognition model (200), the streaming speech recognition model (200) being configured to learn alignment probabilities between the sequence of acoustic frames (110) and an output sequence of label tokens (242); generating a speech recognition result (120) for the utterance (106) using a decoding graph (300) as output from the streaming speech recognition model (200), the speech recognition result (120) including an output sequence of label tokens (242); generating a speech recognition model loss based on the speech recognition result (120) and a ground truth transcription of the utterance (106); obtaining a reference-forced alignment path (320) from the decoding graph (300) that includes a reference-forced alignment frame; Identifying, from the decoding graph (300), one frame to the left of each fiducial forced alignment frame in the fiducial forced alignment path (320); summing label transition probabilities based on the identified frames to the left of each forced alignment frame in the reference forced alignment path (320); updating the streaming speech recognition model based on the sum of the label transition probabilities and the speech recognition model loss.

12. The operation includes: generating, at each of a plurality of time steps, a high-level feature representation (202) for a corresponding audio frame (110) in the sequence of audio frames (110) by an audio encoder (210) of the streaming speech recognition model (200); receiving, as an input to a label encoder (220) of the streaming speech recognition model (200), a sequence of non-blank symbols (242) output by a final softmax layer; generating, by the label encoder (220), a dense representation (222) at each of the plurality of time steps; receiving as inputs to a joint network (230) of the streaming speech recognition model (200) the high-level feature representation (202) generated by the audio encoder (210) at each of the plurality of time steps and the dense representation (222) generated by the label encoder (220) at each of the plurality of time steps; and generating, by the joint network, at each of the plurality of time steps, a probability distribution over possible speech recognition hypotheses at a corresponding time step.

13. The label encoder (220) comprises a stack of transformer layers (400), each transformer layer (400) having: A normalization layer (404); a masked multi-head attention layer with relative position coding (406); Residual connection (408); a stacking / unstacking layer (410); and a feedforward layer (412).

14. 14. The computer-implemented method of claim 12 or 13, wherein the label encoder comprises a bigram embedding lookup decoder model.

15. 15. The computer-implemented method (500) of any one of claims 11 to 14, wherein the streaming speech recognition model (200) comprises a transformer-transducer model.

16. The audio encoder (210) comprises a stack of transformer layers (400), each of which: A normalization layer (404); a masked multi-head attention layer with relative position coding (406); Residual connection (408); a stacking / unstacking layer (410); and a feedforward layer (412).

17. 17. The computer-implemented method of claim 16, wherein the stacking / unstacking layer is configured to modify a frame rate of the corresponding transformer layer to accommodate processing time by a transformer-transducer model during training and inference.

18. The streaming speech recognition model (200) Recurrent Neural-Transducer (RNN-T) model, Transformer-Transducer Model, A Convolutional Network-Transducer (ConvNet-Transducer) model, or 18. The computer-implemented method (500) of any one of claims 11 to 17, further comprising one of: a conformer-transducer model.

19. The computer-implemented method (500) of any one of claims 11 to 18, wherein the streaming speech recognition model (200) runs on a user device (10) or a server (60).

20. 20. The computer-implemented method (500) of any one of claims 11 to 19, wherein the operations further comprise training the streaming speech recognition model (200) to reduce prediction delay using self-alignment without constraining alignment of the decoding graph (300) using an external aligner model.

Images