Reducing streaming ASR model latency using self alignment

Abstract

To provide a speech recognition model and computer implementation method that reduce streaming automatic speech recognition (ASR) model latency using self alignment.SOLUTION: A streaming speech recognition model 200 includes: an audio encoder that receives a sequence of acoustic frames 110 and generates higher-order feature representations 202 for a corresponding acoustic frame in the sequence of acoustic frames; a label encoder 220 that receives a sequence of non-blank symbols 242 output by a final softmax layer 240 and generates a dense representation 222; and a joint network 230 that receives, as input, the higher-order feature representations generated by the audio encoder and the dense representation generated by the label encoder, and generates a probability distribution 232 over possible speech recognition hypotheses.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to reducing streaming automatic speech recognition (ASR) model latency using self-alignment.

Background Art

[0002] Automatic speech recognition (ASR), which is the process of obtaining an audio input and transcribing it into text, is a very important technology used by mobile devices and other devices. Generally, ASR attempts to provide an accurate transcription of what a person has said by obtaining an audio input (e.g., speech) and transcribing the audio input into text. Modern ASR models have continued to improve in both accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay between a user's speech and the transcription) based on the continuous development of deep neural networks. When using current ASR systems, it is required that the ASR system decode speech in a streaming manner that is either equivalent to real time or, in some cases, faster than real time and also accurate. However, in a streaming end-to-end model that optimizes sequence likelihood without latency constraints, such a model learns to improve the model's predictions by using context from the more distant future, resulting in a large delay between the audio input and the predicted text.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Means for Solving the Problems

[0004] One aspect of the present disclosure provides a streaming speech recognition model that receives a sequence of acoustic frames as input and is configured to generate, for each of a plurality of time steps, a high-order feature representation for a corresponding acoustic frame within the sequence of acoustic frames. The streaming speech recognition model also includes a label encoder that receives, as input, a sequence of non-blank symbols output by a final softmax layer and is configured to generate a dense representation for each of a plurality of time steps. The streaming speech recognition model also includes a joint network that receives, as input, the high-order feature representation generated by the audio encoder for each of a plurality of time steps and the dense representation generated by the label encoder for each of a plurality of time steps, and is configured to generate, for each of a plurality of time steps, a probability distribution over possible speech recognition hypotheses at the corresponding time step. Here, the streaming speech recognition model is trained to reduce prediction latency for each training batch by using self-alignment to encourage an alignment path one frame to the left of the reference forced alignment frame at each time step.

[0005] Implementations of the present disclosure may include one or more of any of the following features. In some implementations, the streaming speech recognition model includes a Transformer-Transducer model. In these implementations, the audio encoder may include a stack of Transformer layers, each Transformer layer including a normalization layer, a masked multi-head attention layer with relative position encoding, a residual connection, a stacking / unstacking layer, and a feed-forward layer. Here, the stacking / unstacking layer may be configured to adjust the frame rate of the corresponding Transformer layer to adjust the processing time by the Transformer-Transducer model during training and inference. In some examples, the label encoder includes a stack of Transformer layers, each Transformer layer including a normalization layer, a masked multi-head attention layer with relative position encoding, a residual connection, a stacking / unstacking layer, and a feed-forward layer.

[0006] In some cases, the label encoder may include a bigram embedding lookup decoder model. In some examples, the streaming speech recognition model includes one of a Recurrent Neural-Transducer (RNN-T) model, a Transformer-Transducer model, a Convolutional Network-Transducer (ConvNet-Transducer) model, or a Conformer-Transducer model. Training the streaming speech recognition model to reduce prediction latency using self-alignment may include using self-alignment without using an external aligner model to constrain the alignment of the decoding graph. In some implementations, the streaming speech recognition model is executed on a user device or a server. In some examples, each acoustic frame in a sequence of acoustic frames includes a dimensional feature vector.

[0007] Another aspect of the present disclosure provides a computer-implemented method for causing data processing hardware to perform operations for training a streaming speech recognition model to use self-alignment to reduce prediction latency when executed on the data processing hardware. The operations include receiving, as input to the streaming speech recognition model, a sequence of acoustic frames corresponding to an utterance. The streaming speech recognition model is configured to learn an alignment probability between the sequence of acoustic frames and an output sequence of label tokens. The operations also include generating, as output from the streaming speech recognition model, a speech recognition result for the utterance. The speech recognition result includes generating an output sequence of label tokens using a decoding graph. The operations also include generating a speech recognition model loss based on the speech recognition result and a ground truth transcription of the utterance. The operations also include obtaining, from the decoding graph, a reference forced alignment path including reference forced alignment frames, and identifying, from the decoding graph, one frame to the left of each reference forced alignment frame in the reference forced alignment path. The operations also include summing label transition probabilities based on the identified frames to the left of each forced alignment frame in the reference forced alignment path, and updating the streaming speech recognition model based on the sum of the label transition probabilities and the speech recognition model loss.

[0008] Implementations of the present disclosure may include one or more of any of the following features. In some implementations, the operation includes generating, by an audio encoder of a streaming speech recognition model, a high-order feature representation for a corresponding acoustic frame in a sequence of acoustic frames at each of a plurality of time steps, receiving, as an input to a label encoder of the streaming speech recognition model, a sequence of non-blank symbols output by a final softmax layer, generating, by the label encoder, a dense representation at each of the plurality of time steps, receiving, as an input to a joint network of the streaming speech recognition model, the high-order feature representation generated by the audio encoder at each of the plurality of time steps and the dense representation generated by the label encoder at each of the plurality of time steps, and generating, by the joint network, a probability distribution over possible speech recognition hypotheses at a corresponding time step at each of the plurality of time steps. In some examples, the label encoder includes a stack of transformer layers, and each transformer layer includes a normalization layer, a masked multi-head attention layer with relative position encoding, a residual connection, a stacking / unstacking layer, and a feed-forward layer. The label encoder may include a bigram embedding lookup decoder model.

[0009] In some implementations, the streaming speech recognition model includes a Transformer-Transducer model. The audio encoder may include a stack of transformer layers, and each transformer layer includes a normalization layer, a masked multi-head attention layer with relative position encoding, a residual connection, a stacking / unstacking layer, and a feed-forward layer. Here, the stacking / unstacking layer may be configured to change the frame rate of a corresponding transformer layer to adjust the processing time by the Transformer-Transducer model during training and inference.

[0010] In some implementations, the streaming speech recognition model includes one of a recurrent neural - transducer (RNN - T) model, a transformer - transducer model, a convolutional network - transducer (ConvNet - transducer) model, or a conformer - transducer model. The streaming speech recognition model may be executed on a user device or a server. In some examples, the operation further includes training a streaming speech recognition model to use self - alignment to reduce prediction latency without using an external aligner model to constrain the alignment of the decoding graph.

[0011] Details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0013] Like reference numerals in the various drawings indicate like elements.

[0014] An automatic speech recognition (ASR) system aims to achieve not only high quality / accuracy (e.g., low word error rate (WER)) but also low latency (e.g., short delay between a user's utterance and the resulting transcription). Recently, end-to-end (E2E) ASR models have been widely used to achieve state-of-the-art performance in terms of accuracy and latency. In contrast to conventional hybrid ASR systems that include separate acoustic, pronunciation, and language models, E2E models apply a sequence-to-sequence approach to jointly learn acoustic and language modeling in a single neural network that is trained end-to-end from training data, e.g., utterance-transcription pairs. Here, an E2E model refers to a model whose architecture is entirely composed of neural networks. A complete neural network functions without external components and / or manually designed components (e.g., finite state transducers, dictionaries, or text normalization modules). Also, when training E2E models, these models generally do not require bootstrapping from decision trees nor time alignment from separate systems.

[0015] When using an ASR system, there may be a requirement for the ASR system to decode speech in a streaming manner that corresponds to displaying a description of the speech in real time or, in some cases, faster than real time when the user is speaking. As an example, when the ASR system is displayed on a user computing device such as a mobile phone that experiences direct user interaction, an application running on the user device that uses the ASR system (e.g., a digital assistant application) needs to stream speech recognition such that words, word pieces, and / or individual characters are displayed on the screen as they are spoken. Also, the user of the user device may have a low tolerance for latency. For example, when a user speaks a query that requests the digital assistant to retrieve detailed information from a calendar application to check their schedule, the user expects the digital assistant to provide a response communicating the retrieved detailed information as quickly as possible. Due to such low tolerance, the ASR system works hard on the user device to minimize the impact of latency and inaccuracy, which may negatively affect the user experience.

[0016] One form of the sequence-to-sequence model called the Recurrent Neural Network Transducer (RNN-T) does not use an attention mechanism and, unlike other sequence-to-sequence models that generally need to process the entire sequence (audio waveform) to generate an output (e.g., text), the RNN-T processes input samples continuously and streams output symbols. This is particularly attractive for real-time communication. For example, speech recognition by RNN-T may output characters one by one in response to speech. Here, the RNN-T uses a feedback loop that sends the symbol predicted by the model back to the RNN-T itself to predict the next symbol. Decoding the RNN-T involves beam search by a single neural network rather than a large decoder graph, so the RNN-T may scale to a fraction of the size of server-based speech recognition models. The size reduction allows the entire RNN-T to be deployed on-device and may operate offline (i.e., without a network connection), thus avoiding reliability issues with the communication network. The RNN-T model that uses Long Short-Term Memory (LSTM) to provide a sequence encoder is generally suitable for providing a streaming transcription function for recognizing conversational queries (e.g., "Please set a timer", "Don't forget to buy milk", etc.) and applications affected by latency, but has limited ability to pre-read audio text, thereby still lagging behind state-of-the-art conference models (e.g., server-based models with separate AM, PM, and LM) and attention-based sequence-to-sequence models (e.g., Listen Attend Spell (LAS)) in terms of quality (e.g., speech recognition accuracy often measured by Word Error Rate (WER)).

[0017] In recent years, the Transformer-Transducer (T-T) and Conformer-Transducer (C-T) model architectures have been introduced to further improve the RNN-T model architecture by replacing the LSTM layers in the audio encoder and / or prediction network with their respective Transformer layers or Conformer layers. Generally, the T-T and C-T model architectures can access future audio frames (e.g., right context) when calculating self-attention in their respective Transformer layers or Conformer layers. Therefore, the T-T and C-T model architectures can utilize the future right context to operate in a non-streaming transcription mode and improve the speech recognition performance when the latency constraint is relaxed. That is, the duration of the prediction delay is proportional to the amount of future-accessed audio frames. However, similar to RNN-T, the T-T and C-T model architectures can also operate in a streaming transcription mode where self-attention depends only on past acoustic frames (e.g., left context).

[0018] Streaming speech recognition models, such as transducer models (e.g., RNN-T, T-T, and C-T), optimize sequence likelihood without latency constraints, and thus these models learn to improve predictions by using context from the more distant future, resulting in a large delay between the audio input and the predicted text. Recent techniques for reducing prediction latency include constrained alignment techniques that penalize word boundaries based on audio alignment information obtained from an external alignment model by masking alignment paths that exceed a predetermined threshold latency. This technique is effective in reducing the latency of streaming end-to-end models, but requires a highly accurate external alignment model to minimize WER degradation, which can make the model training steps more complex. Other techniques that blindly reduce latency by selecting the most efficient direction in the RNN-T decoding graph often select directions that are not optimal for all audio inputs due to the lack of alignment information, which can further worsen the trade-off relationship between latency and WER.

[0019] To mitigate the drawbacks associated with simply blindly reducing latency by using an external alignment model or selecting the most efficient direction from a decoding graph, the implementations herein are directed to reducing prediction latency in a streaming speech recognition model by using self-alignment. In particular, self-alignment does not require the use of an external alignment model and does not blindly optimize latency. Instead, it selects the optimal low-latency direction for reducing latency by utilizing a criterion forced alignment learned from a trained speech recognition model. The criterion forced alignment may include a Viterbi forced alignment. That is, self-alignment always identifies the path in the decoding graph that is one frame to the left of the Viterbi forced alignment at each time step. Self-alignment has advantages over existing methods for constraining latency. First, since no external alignment model is required for self-alignment, the training complexity for self-alignment is much lower than that of supervised methods. Second, self-alignment has a minimal impact on ASR training by constraining only the most probable alignment path. In contrast, other methods affect many alignment paths by masking the alignment path or changing the weights for label transition probabilities. Since the latency constraint regularization term opposes the main ASR loss, it is important to minimize the intervention on the main loss in optimizing the latency-performance trade-off. Self-alignment only regularizes the path by pushing a single path to the left direction.

[0020] FIG. 1 is an example of an audio environment 100. In the audio environment 100, the way a user 104 interacts with a computing device such as the user device 10 may be via voice input. The user device 10 (commonly also referred to as device 10) is configured to capture voice (e.g., streaming audio data) from one or more users 104 within the audio environment 100. Here, the streaming audio data may refer to an audible query, a command for the device 10, or an utterance 106 by the user 104 that serves as an audible communication captured by the device 10. The voice-responsive system of the device 10 may process the query or command by answering the query and / or executing / completing the command by one or more downstream applications.

[0021] The user device 10 may correspond to any computing device associated with the user 104 and can receive audio data. Some examples of the user device 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smartwatches), smart home appliances, Internet of Things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, and the like. The user device 10 includes data processing hardware 12 and memory hardware 14 that communicates with the data processing hardware 12, and stores instructions that cause the data processing hardware 12 to perform one or more operations when executed by the data processing hardware 12. The user device 10 further includes an audio system 16 having a voice capture device (e.g., a microphone) 16, 16a for capturing an utterance 106 in the acoustic environment 100 and converting it into an electrical signal, and a voice output device (e.g., a speaker) 16, 16b for communicating an audible audio signal (e.g., as output audio data from the device 10). Although the user device 10 implements a single voice capture device 16a in the illustrated example, the user device 10 may implement an array of voice capture devices 16a without departing from the scope of the present disclosure, and one or more of the capture devices 16a in the array may communicate with the audio system 16 without being physically present on the user device 10.

[0022] In the audio environment 100, the automatic speech recognition (ASR) system 118 transducer model 200 exists on the user device 10 of the user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system executed in a cloud-computing environment) that communicates with the user device 10 via the network 40. The user device 10 and / or the remote computing device 60 also include an audio subsystem 108 configured to receive an utterance 106 by the user 104 captured by the audio capture device 16a and convert the utterance 106 into a corresponding digital format related to an input acoustic frame 110 that can be processed by the ASR system 118. In the illustrated example, the user emits each utterance 106, and the audio subsystem 108 converts the utterance 106 into a corresponding audio data (e.g., acoustic frame) 110 input to the ASR system 118. Thereafter, the transducer model 200 receives the audio data 110 corresponding to the utterance 106 as input and generates / predicts a corresponding transcription 120 (e.g., recognition result / hypothesis) of the utterance 106 as output. The transducer model 200 provides a streaming speech recognition model result that cannot access look-ahead audio and thus provides a streaming transcription function in real time while the user 104 is emitting the utterance 106. For example, a digital assistant application 50 executed on the user device 10 may need to stream speech recognition such that words, word pieces, and / or individual characters are displayed on the screen when uttered.

[0023] The user device 10 and / or the remote computing device 60 also execute a user interface generator 107 configured to present the representation of the transcription 120 of the utterance 106 to the user 104 of the user device 10. As will be described in detail below, the user interface generator 107 may display the partial speech recognition result 120a in a streaming manner during time 1 and then display the final speech recognition result 120b during time 2. In some configurations, the transcription 120 output from the ASR system 118 is processed by, for example, a natural language understanding (NLU) module executed on the user device 10 or the remote computing device 60 to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executed on any combination of the user device 10 or the remote computing device 60) may convert the transcription into synthetic speech for audible output by the user device 10 and / or another device.

[0024] In the illustrated example, the user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) of the user device 10 that uses the ASR system 118. For example, FIG. 1 shows the user 104 communicating with the digital assistant application 50, and the digital assistant application 50 displaying a digital assistant interface 18 on the screen of the user device 10, showing a conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, "What time is the concert tonight?" This question from the user 104 is an utterance 106 that is captured by the audio capture device 16a and processed by the audio system 16 of the user device 10. In this example, the audio system 16 receives the utterance 106 and converts it into an acoustic frame 110 that is input to the ASR system 118.

[0025] Continuing with the example, while the transducer model 200 receives the acoustic frame 110 corresponding to the utterance 106 when the user 104 speaks, it encodes the acoustic frame 110 and then decodes the encoded acoustic frame 110 into the partial speech recognition result 120a. During time 1, the user interface generator 107 presents, via the digital assistant interface 18, the representation of the partial speech recognition result 120a of the utterance 106 to the user 104 of the user device 10 in a streaming manner, whereby words, word pieces, and / or individual characters are displayed on the screen as they are spoken. In some examples, the first look-ahead audio context is equal to zero.

[0026] During time 2, the user interface generator 107 presents, via the digital assistant interface 18, the representation of the final speech recognition result 120b of the utterance 106 to the user 104 of the user device 10. The final speech recognition result 120b may simply be the partial speech recognition result 120a when the user has finished speaking. In some cases, the ASR system 118 may include another speech recognition to rescore the partial speech recognition result and / or use an external language model. In some cases, the same transducer model 200 may process the audio again after the user has finished speaking and instead utilize the right look-ahead audio context to generate the final speech recognition result 120b. The present disclosure is directed to limiting the latency in the streaming partial speech recognition result 120a output by the transducer model 200 regardless of how the final speech recognition result 120b is obtained.

[0027] In the example shown in FIG. 1, the digital assistant application 50 may respond to questions presented by the user 104 using natural language processing. Natural language processing generally refers to the process of interpreting written language (e.g., partial speech recognition result 120a and / or final speech recognition result 120b) and determining whether the written language prompts some action. In this example, the digital assistant application 50 uses natural language processing to recognize that the question from the user 104 is a question regarding the user's schedule, and more specifically, a question regarding a concert on the user's schedule. By recognizing such detailed information through natural language processing, the automated assistant returns a response 19 to the user's query, in this case, the response 19 presents "The venue opens at 6:30 p.m. and the concert starts at 8:00." In some configurations, natural language processing is performed on a remote server 60 that communicates with the data processing hardware 12 of the user device 10.

[0028] Referring to FIG. 2, the transducer model 200 may enable end-to-end (E2E) speech recognition by incorporating an acoustic model, a pronunciation model, and a language model into a single neural network, in which case neither a dictionary nor a separate text normalization component is required. Various structures and optimization mechanisms can improve accuracy and shorten the model training time. In the illustrated example, the transducer model 200 includes a Transformer-Transducer (T-T) model architecture, and the Transformer-Transducer (T-T) model architecture complies with the latency constraints associated with interactive applications. The T-T model 200 has a small computational footprint and fewer memory requirements than conventional ASR architectures, such that the T-T model architecture is suitable for performing speech recognition on the entire user device 10 (e.g., communication with the remote server 60 is not required). The T-T model 200 includes an audio encoder 210, a label encoder 220, and a joint network 230. The audio encoder 210 is generally similar to the acoustic model (AM) in a conventional ASR system and includes a neural network having a plurality of Transformer layers. For example, the audio encoder 210 reads a sequence x = (x1, x2, ..., x T ) of d-dimensional feature vectors (e.g., acoustic frames 110 (FIG. 1)), where

Number

[0029] Similarly, the label encoder 220 may also include a neural network of transformer layers or a lookup table embedding model, and the lookup table embedding model, similar to the language model (LM), has been output by the final softmax layer 240 so far for the sequence y0, ..., y of non-blank symbols 242 ui-1 is processed as a dense representation 222 (e.g., shown as Ih u that encodes the predicted label history). In an implementation where the label encoder 220 includes a neural network of transformer layers, each transformer layer may include a normalization layer, a masked multi-head attention layer with relative position encoding, a residual connection, a feed-forward layer, and a dropout layer. In these implementations, the label encoder 220 may include two transformer layers. In an implementation where the label encoder 220 includes a lookup table embedding model with a bi-gram label context, the embedding model is configured to learn a d-dimensional weight vector for each possible bigram label context, where d is the dimension of the outputs of the audio encoder 210 and the label encoder 220. In some examples, the total number of parameters in the embedding model is N 2 ×d, where N is the vocabulary size for the labels. Here, the learned weight vectors are then used as embeddings of the bigram label contexts in the T-T model 200 to generate fast label encoder 220 execution times.

[0030] Finally, in the T-T model architecture, the representations generated by the audio encoder 210 and the label encoder 220 are combined by the joint network 230 using a dense layer J u,t and then the joint network 230 predicts an alignment distribution (e.g., alignment probability 232) over the next output symbol as follows. Pr(z u,t |x,t,y1,…, y u-1) (1) In the above equation, x is the audio input, y is the ground truth label sequence, and z is the alignment belonging to y. In other words, the joint network 230 generates, at each output step (e.g., time step), a probability distribution 232 over possible speech recognition hypotheses. Here, "possible speech recognition hypotheses" corresponds to a set of output labels (also called "speech units") each represented by a grapheme (e.g., symbol / character) or word piece in the specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 characters in the English alphabet and one label for the space. Thus, the joint network 230 may output a set of values indicating the likelihood of occurrence of each of a predetermined set of output labels. This set of values can be a vector (e.g., one-hot vector) and can represent a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and optionally punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels can include word pieces and / or whole words in addition to or instead of graphemes. The output distribution of the joint network 230 can include posterior probability values for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output z of the joint network 230 u,t can include 100 different probability values, one for each output label. Then, the probability distribution can be used to select scores (e.g., by the softmax layer 240) in a beam search process and assign them to candidate orthographic elements (e.g., graphemes, word pieces, and / or words) to determine the transcription 120.

[0031] The softmax layer 240 may select, using any technique, the output label / symbol having the highest probability in the distribution as the next output symbol 242 predicted by the T-T model 200 in the corresponding output step. Thus, the set of output symbols 242 predicted by the T-T model 200 may collectively be referred to as the output sequence of label tokens 242. In this way, the T-T model 200 does not make a conditional independence assumption, and the prediction of each symbol is conditioned on not only the acoustics but also the sequence of labels that have been output so far.

[0032] To determine the log conditional probability of y given the audio input x, sum all the alignment distributions corresponding to y as follows. [Number] In the above equation, the mapping K removes the blank symbol in z. This log total alignment probability of Equation 2 includes a target loss function that can be efficiently calculated using the forward-backward algorithm as follows. Pr(y|x)=α(T,U) (3) α(t,u)= α(t-1,u-1)Pr(φ|t-1,u)+α(t,u-1)Pr(y u |t,u-1) (4) In the above equation, Pr(φ|t-1,u) and Pr(y u |t,u-1) are the blank probability and the label probability, respectively, and T and U are the audio sequence length and the label sequence length.

[0033] FIG. 2 shows a transducer model 200 including a T-T model architecture, although the transducer model 200 may include an RNN-T model architecture, a convolutional neural network - transducer (CNN - transducer) model architecture, a convolutional network transducer (ConvNet - transducer) model, or a conformer - transducer model architecture without departing from the scope of the present disclosure. An exemplary CNN - transducer model architecture is described in detail in "Contextnet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context," https: / / arxiv.org / abs / 2005.03191, the contents of which are hereby incorporated by reference in their entirety. An exemplary conformer - transducer model architecture is described in detail in "Conformer: Convolution - augmented transformer for speech recognition," https: / / arxiv.org / abs / 2005.08100, the contents of which are hereby incorporated by reference in their entirety.

[0034] The transducer model 200 is trained on a training dataset of audio data corresponding to utterances paired with corresponding transcriptions. The training of the transducer model 200 may be performed on the remote server 60, and the trained transducer model 200 may be pushed to the user device 10. The transducer model 200 is trained using a cross-entropy error based on Viterbi forced alignment. The alignment delay includes the delay between the input audio frame and the decoded output label that has been stream-processed. Conventional models train an alignment model by dialogue using the re-aligned labels, so that an accurate alignment can be learned after multiple iterations. The T-T model or C-T model that accesses future frames when calculating self-attention in each transformer layer or conformer layer may include an alignment delay that matches the conventional model. However, in the transducer model in streaming mode where self-attention depends only on past frames, an alignment delay occurs.

[0035] Embodiments herein are directed to reducing prediction latency in the streaming transducer model 200 by using self-alignment. In particular, self-alignment does not require the use of an external alignment model and does not blindly optimize the delay. Instead, it selects an optimal low-latency direction to reduce the delay by utilizing the reference forced alignment learned from the trained speech recognition model. The reference forced alignment may include Viterbi forced alignment. That is, self-alignment always identifies the path in the decoding graph that is one frame to the left of the Viterbi forced alignment at each time step.

[0036] Figure 3 shows a plot of the decoding graph 300 for the transducer model 200 having a T-T model architecture for the output sequence (Figure 2) "I like it" of the label token 242. The x-axis represents each acoustic frame at each time step, and the y-axis represents the output label token 242 (Figure 2). The circles and arrows that are not thick solid lines represent each token not included in the alignment path described later. The constrained alignment path 310 (represented by thick circles and thick arrows as shown in Figure 3) includes a word boundary threshold equal to 2. The forced alignment path 320 learned from the trained transducer model 200 (for example, represented by dotted circles and dotted arrows as shown in Figure 3) (also called the reference forced alignment path 320) and the left alignment path 330 (for example, represented by dashed circles) include 1 frame to the left of each frame of the forced alignment path 320. During the training of the transducer model 200, for each training batch, self-alignment always promotes the left alignment path 330 (represented by dashed circles and dashed arrows as shown in Figure 3) by pushing the model's forced alignment path 320 to the left. The training loss may be expressed as follows.

Number

[0037] Figure 4 shows an exemplary transformer layer 400 between multiple transformer layers of the audio encoder 210. Here, during each time step, the initial transformer layer 400 receives the corresponding acoustic frame 110 as input and generates the corresponding output representation / embedding 450 that is received as input by the next transformer layer 400. That is, each transformer layer 400 following the initial transformer layer 400 may receive the input embedding 450 corresponding to the output representation / embedding generated as output by the immediately preceding transformer layer 400. The final transformer layer 400 (e.g., the last transformer layer in the final stack 320) generates a higher-order feature representation 202 (e.g., related to Figure 2 and represented by ah t ).

[0038] The input to the label encoder 220 (Figure 2) may include a vector (e.g., a one-hot vector) indicating the sequence of non-blank symbols y0, ..., y that has been output by the final softmax layer 240 up to that point. ui-1 Thus, when the label encoder 220 includes a transformer layer, the initial transformer layer may receive the input embedding 111 by passing the one-hot vector through a lookup table.

[0039] Each transformer layer 400 of the audio encoder 210 includes a normalization layer 404, a masked multi-head attention layer 406 with relative position encoding, a residual connection 408, a stacking / unstacking layer 410, and a feed-forward layer 412. The masked multi-head attention layer 406 with relative position encoding provides a flexible way to control the amount (i.e., duration) of look-ahead audio context used by the T-T model 200. Specifically, after the normalization layer 404 normalizes the acoustic frames 110 and / or the input embeddings 111, the masked multi-head attention layer 406 projects the input to a certain value for all heads. Then, the masked multi-head layer 406 may mask the attention scores to the previous frames of the current acoustic frame 110 to generate an output conditioned only on the previous acoustic frames 110. Next, the weighted average values for all heads are concatenated and passed to the dense layer 2 416, where the residual connection 414 is added to the normalized input and the output of the dense layer 416 to form the final output of the multi-head attention layer 406 with relative position encoding. The residual connection 408 is added to the output of the normalization layer 404 by the adder 430 and provided as an input to each of the masked multi-head attention layer 406 or the feed-forward layer 412. The frame rate can be changed for each transformer layer 400 using the stacking / unstacking layer 410 to accelerate training and inference.

[0040] The feed-forward layer 412 applies the normalization layer 404 and then is applied to the dense layer 1 420, the normalization linear layer (ReLu) 418, and the dense layer 2 416 in sequence. ReLu 418 is used as an activation for the output of the dense layer 1 420. Similar to the multi-head attention layer 406 with relative position encoding, the residual connection 414 from the output of the normalization layer 404 is added to the output of the dense layer 2 416 by the adder 430.

[0041] FIG. 5 shows an exemplary configuration flowchart of operations for a method 500 of training a streaming speech recognition model to reduce prediction latency using self-alignment. The method includes, in operation 502, receiving a sequence of acoustic frames 110 corresponding to utterance 106 as input to a streaming speech recognition model (e.g., a transducer model) 200. The streaming speech recognition model 200 is configured to learn an alignment probability 232 between the sequence of acoustic frames 110 and an output sequence of label tokens 242. The method 500 includes, in operation 504, generating a speech recognition result 120 for utterance 106 including the output sequence of label tokens 242 using a decoding graph 300 as an output from the streaming speech recognition model 200. The method 500 includes, in operation 506, generating a speech recognition model loss based on the speech recognition result 120 and the ground truth transcription of utterance 106.

[0042] The method 500 includes, in operation 508, obtaining a reference forced alignment path 320 from the decoding graph 300. The method 500 includes, in operation 510, identifying, from the decoding graph 300, one frame to the left of each reference forced alignment frame in the reference forced alignment path 320. The method 500 includes, in operation 512, summing label transition probabilities based on the identified frames to the left of each forced alignment frame. The method 500 includes, in operation 514, updating the streaming speech recognition model 200 based on the sum of the label transition probabilities and the speech recognition model loss.

[0043] FIG. 6 is a schematic diagram of an exemplary computing device 600 that may be used to implement the systems and methods described herein. Computing device 600 is intended to represent various forms of digital computers, such as a laptop, desktop, workstation, personal digital assistant, server, blade server, mainframe, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only, and are not intended to limit the implementations of the invention described and / or claimed herein.

[0044] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 connected to memory 620 and high-speed expansion port 650, and a low-speed interface / controller 660 connected to low-speed bus 670 and storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and may be mounted on a common motherboard or in other manners as required. Processor 610 can process instructions for execution within computing device 600, including instructions stored in memory 620 or on storage device 630 that include graphical information for a graphical user interface (GUI) to be displayed on an external input / output device such as display 680 coupled to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memory, as required. Also, multiple computing devices 600 may be connected, with each device performing a part of the necessary operations (e.g., a server bank, a group of blade servers, or a multi-processor system).

[0045] Memory 620 stores information non - transiently within computing device 600. Memory 620 may be a computer - readable medium, a volatile memory unit, or a non - volatile memory unit. The non - transient memory 620 may be a physical device used to store a program (e.g., a sequence of instructions) or data (e.g., program state information) temporarily or persistently so that it can be used by computing device 600. Examples of non - volatile memory include, but are not limited to, flash memory and read - only memory (ROM) / programmable read - only memory (PROM) / erasable programmable read - only memory (EPROM) / electrically erasable programmable read - only memory (EEPROM) (commonly used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase - change memory (PCM), and disks or tapes.

[0046] Storage device 630 can provide a mass storage device for computing device 600. In some implementations, storage device 630 is a computer - readable medium. In various different implementations, storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid - state memory device, or an array of devices including devices within a storage area network or other configurations. In additional implementations, a computer program product is actually embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more of the methods as described above. The information carrier is a computer or machine - readable medium such as memory 620, storage device 630, or memory on processor 610.

[0047] The high-speed controller 640 manages bandwidth-intensive operations for the computing device 600, while the low-speed controller 660 manages lower-bandwidth-intensive operations. Such an allocation of duties is merely exemplary. In some implementations, the high-speed controller 640 is coupled to the memory 620, the display 680 (e.g., via a graphics processor or accelerator), and the high-speed expansion port 650. The high-speed expansion port 650 may accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and the low-speed expansion port 690. The low-speed expansion port 690 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), and may be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., via a network adapter.

[0048] The computing device 600 may be implemented in several different ways as shown. For example, the computing device 600 may be implemented as a standard server 600a, or multiple times within a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0049] Various implementations of the systems and techniques described herein can be realized in digital electrical and / or optical circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations within one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor. The programmable processor may be dedicated or general purpose and may be coupled to receive and transmit data and instructions between a memory system, at least one input device, and at least one output device.

[0050] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application", an "app", or a "program". Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, document processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0051] A non-transitory memory may be a physical device used to temporarily or persistently store a program (e.g., a sequence of instructions) or data (e.g., program state information) for use by a computing device. The non-transitory memory may be a volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (commonly used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disks or tapes.

[0052] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in high-level procedural languages and / or object-oriented programming languages, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0053] The processes and logical flows described in this specification, also referred to as data processing hardware, can be executed by one or more programmable processors that execute one or more computer programs to act on input data and generate output. The processes and logical flows can also be executed by special purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Processors suitable for executing computer programs include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. In general, a processor receives instructions and data from a read only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. In general, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD ROM disks. The processor and the memory can be assisted by, or incorporated in, special purpose logic circuitry.

[0054] To enable interaction with a user, one or more aspects of the present disclosure can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, and optionally a keyboard and a pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to enable interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input from the user can be received in any form, including acoustic input, voice input, or tactile input. Also, the computer can interact with the user by sending a document to the device being used by the user and receiving a document from that device, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0055] Some implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims.

Description of the Reference Numerals

[0056] 1 or 2 hours 10 User devices 12 Data processing hardware 14 Memory hardware 16 Audio system 16a Voice input device 16b Voice output device 18 Digital assistant interface 19 Response 50 Digital assistant application 60 Remote computing device 100 Acoustic environment 104 User 106 Utterance 107 User interface generator 108 Audio subsystem 110 Acoustic frame, audio data 111 Input embedding 118 ASR system 120 Transcription 120a Partial speech recognition result 120b Final speech recognition result 200 T-T model, transducer model 202 Higher-order feature representation 210 Audio encoder 220 Label encoder 230 Joint network 232 Alignment probability 240 Softmax layer 242 Output symbol, non-blank symbol, label token 300 Decoding graph 310 Alignment path 320 Forced alignment path 330 Left alignment path 400 Transformer layer 404 Normalization layer 406 Masked multi-head attention layer 408, 414 Residual connection 410 Stacking / unstacking layer 412 Feed-forward layer 416, 420 Dense layer 418 Normalized linear layer 430 Adder 450 Output representation / embedding 600 Computing device 600a Server 600b Laptop computer 600c Rack server system 610 Processor 620 Memory 630 Storage Device 640 High-Speed Interface / Controller 650 High-Speed Expansion Port 660 Low-Speed Interface / Controller 670 Low-Speed Bus 680 Display 690 Low-Speed Expansion Port

Claims

1. A streaming speech recognition model (200), comprising: An audio encoder (210) configured to: Receive a sequence of acoustic frames (110) as input, and Generate a high-order feature representation (202) for a corresponding acoustic frame (110) in the sequence of acoustic frames (110) at each of a plurality of time steps; An audio encoder (210) configured as such; A label encoder (220) configured to: Receive a sequence of non-blank symbols (242) output by a final softmax layer (240) as input, and Generate a dense representation (222) at each of the plurality of time steps; A label encoder (220) configured as such; A joint network (230) configured to: Receive, at each of the plurality of time steps, the high-order feature representation (202) generated by the audio encoder (210) and the dense representation (222) generated by the label encoder (220) at each of the plurality of time steps as input, and Generate a probability distribution (232) over possible speech recognition hypotheses at a corresponding time step at each of the plurality of time steps; A joint network (230) configured as such, wherein the streaming speech recognition model (200) is trained to reduce prediction latency by using self-alignment to encourage an alignment path one frame to the left of a reference forced alignment frame at each time step for each training batch.

2. The speech recognition model (200) according to claim 1, wherein the streaming speech recognition model (200) comprises a Transformer-Transducer model.

3. The audio encoder (210) comprises a stack of Transformer layers (400), and each Transformer layer (400) comprises: A normalization layer (404); A masked multi-head attention layer (406) with relative position encoding; A residual connection (408); A stacking / unstacking layer (410); and A feed-forward layer (412).

4. The stacking / unstacking layer (410) is configured to adjust the processing time by the transformer-transducer model during training and inference by changing the frame rate of the corresponding transformer layer (400), the speech recognition model (200) according to claim 3.

5. The label encoder (220) comprises a stack of transformer layers (400), each transformer layer (400) comprising a normalization layer (404), a masked multi-head attention layer (406) with relative position encoding, a residual connection (408), a stacking / unstacking layer (410), and a feed-forward layer (412), the speech recognition model (200) according to any one of claims 2 to 4.

6. The label encoder (220) comprises a bigram embedding lookup decoder model, the speech recognition model (200) according to any one of claims 1 to 5.

7. The streaming speech recognition model (200) comprises one of a recurrent neural - transducer (RNN-T) model, a transformer-transducer model, a convolutional network - transducer (ConvNet-transducer) model, or a conformer-transducer model, the speech recognition model (200) according to any one of claims 1 to 6.

8. Training the streaming speech recognition model (200) to reduce prediction latency using self - alignment involves using self - alignment without constraining the alignment of the decoding graph (300) using an external aligner model, the speech recognition model (200) according to any one of claims 1 to 7.

9. The streaming speech recognition model (200) is executed on a user device (10) or a server (60), the speech recognition model (200) according to any one of claims 1 to 8.

10. Each acoustic frame (110) in the sequence of acoustic frames (110) comprises a dimensional feature vector, the speech recognition model (200) according to any one of claims 1 to 9.

11. A computer-implemented method (500) for causing the data processing hardware (12) to perform operations for training a streaming speech recognition model (200) to reduce prediction latency using self-alignment when executed on the data processing hardware (12), the operations comprising: Receiving, as an input to the streaming speech recognition model (200), a sequence of acoustic frames (110) corresponding to an utterance (106), wherein the streaming speech recognition model (200) is configured to learn an alignment probability between the sequence of acoustic frames (110) and an output sequence of label tokens (242); Generating, as an output from the streaming speech recognition model (200), a speech recognition result (120) for the utterance (106) using a decoding graph (300), wherein the speech recognition result (120) includes an output sequence of label tokens (242); Generating a speech recognition model loss based on the speech recognition result (120) and a ground truth transcription of the utterance (106); Obtaining a reference forced alignment path (320) including reference forced alignment frames from the decoding graph (300); Identifying, from the decoding graph (300), one frame to the left of each reference forced alignment frame in the reference forced alignment path (320); Summing label transition probabilities based on the identified frames to the left of each forced alignment frame in the reference forced alignment path (320); Updating the streaming speech recognition model (200) based on the sum of the label transition probabilities and the speech recognition model loss. Claim 12 The operations further comprise: Generating, by an audio encoder (210) of the streaming speech recognition model (200), a high-order feature representation (202) for a corresponding acoustic frame (110) in the sequence of acoustic frames (110) at each of a plurality of time steps. Receiving, as input to the label encoder (220) of the streaming speech recognition model (200), a sequence of non-blank symbols (242) output by the final softmax layer; Generating, by the label encoder (220), a dense representation (222) at each of the plurality of time steps; Receiving, as input to the joint network (230) of the streaming speech recognition model (200), the high-order feature representation (202) generated by the audio encoder (210) at each of the plurality of time steps and the dense representation (222) generated by the label encoder (220) at each of the plurality of time steps; Further comprising generating, by the joint network (230), at each of the plurality of time steps, a probability distribution (232) over possible speech recognition hypotheses at the corresponding time step, the computer-implemented method (500) according to claim 11.

13. The label encoder (220) comprises a stack of transformer layers (400), each transformer layer (400) comprising a normalization layer (404); a masked multi-head attention layer (406) with relative position encoding; a residual connection (408); a stacking / unstacking layer (410); and a feed-forward layer (412), the computer-implemented method (500) according to claim 12.

14. The label encoder (220) comprises a bigram embedding lookup decoder model, the computer-implemented method (500) according to claim 12 or 13.

15. The streaming speech recognition model (200) comprises a Transformer-Transducer model, the computer-implemented method (500) according to any one of claims 11 to 14.

16. The audio encoder (210) comprises a stack of transformer layers (400), each transformer layer (400) comprising a normalization layer (404); a masked multi-head attention layer (406) with relative position encoding; a residual connection (408); a stacking / unstacking layer (410); and a feed-forward layer (412), the computer-implemented method (500) according to claim 15.

17. The stacking / unstacking layer (410) is configured to change the frame rate of the corresponding transformer layer (400) to adjust the processing time by the transformer-transducer model during training and inference, the computer-implemented method (500) according to claim 16.

18. The streaming speech recognition model (200) is a recurrent neural - transducer (RNN-T) model, a transformer - transducer model, a convolutional network - transducer (ConvNet - transducer) model, or a conformer - transducer model, and includes one of them, the computer-implemented method (500) according to any one of claims 11 to 17.

19. The streaming speech recognition model (200) is executed on a user device (10) or a server (60), the computer-implemented method (500) according to any one of claims 11 to 18.

20. The operation further includes training the streaming speech recognition model (200) to shorten prediction latency using self - alignment without using an external aligner model to constrain the alignment of the decoding graph (300), the computer-implemented method (500) according to any one of claims 11 to 19.

Images