Reducing the latency of streaming ASR models using self-alignment

Abstract

To provide a speech recognition model and computer implementation method that reduce streaming automatic speech recognition (ASR) model latency using self alignment.SOLUTION: A streaming speech recognition model 200 includes: an audio encoder that receives a sequence of acoustic frames 110 and generates higher-order feature representations 202 for a corresponding acoustic frame in the sequence of acoustic frames; a label encoder 220 that receives a sequence of non-blank symbols 242 output by a final softmax layer 240 and generates a dense representation 222; and a joint network 230 that receives, as input, the higher-order feature representations generated by the audio encoder and the dense representation generated by the label encoder, and generates a probability distribution 232 over possible speech recognition hypotheses.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This disclosure relates to reducing streaming automatic speech recognition (ASR) model latency using self-alignment. [Background technology]

[0002] Automatic speech recognition (ASR), the process of taking audio input and transcribing it into text, is a crucial technology used by mobile devices and other devices. Generally, ASR attempts to provide an accurate transcription of what a person says by taking audio input (e.g., utterances) and transcribing that audio input into text. Modern ASR models continue to improve in both accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay between the user's utterance and the transcription) based on the ongoing development of deep neural networks. When using ASR systems today, it is required that the ASR system decodes utterances in a streaming manner that is fast, or in some cases, real-time equivalent, or even faster, but also accurate. However, streaming end-to-end models that optimize sequence likelihood without latency constraints suffer from a significant delay between the audio input and the predicted text, due to the fact that such models learn to improve their predictions by using more distant future context. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] U.S. Patent Application No. 17 / 210465 [Overview of the project] [Means for solving the problem]

[0004] One aspect of the present disclosure provides a streaming speech recognition model, comprising an audio encoder configured to receive a sequence of acoustic frames as input and, in each of a plurality of time steps, generate a higher-order feature representation of the corresponding acoustic frames in the sequence of acoustic frames. The streaming speech recognition model also comprises a label encoder configured to receive a sequence of non-blank symbols output by a final softmax layer as input and, in each of a plurality of time steps, generate a dense representation. The streaming speech recognition model also comprises a joint network configured to receive the higher-order feature representations generated by the audio encoder and the dense representations generated by the label encoder in each of a plurality of time steps as input and, in each of a plurality of time steps, generate a probability distribution over possible speech recognition hypotheses in the corresponding time step. Here, the streaming speech recognition model is trained to reduce predictive delay by using self-alignment to prompt an alignment path one frame to the left of a reference forced alignment frame in each time step for each training batch.

[0005] Implementations of this disclosure may include one or more of the following features. In some implementations, the streaming speech recognition model includes a transformer-transducer model. In these implementations, the audio encoder may include a stack of transformer layers, each transformer layer including a normalization layer, a masked multi-head attention layer with relative position coding, a residual connection, a stacking / unstacking layer, and a feedforward layer. The stacking / unstacking layer may be configured to adjust the processing time by the transformer-transducer model between training and inference by changing the frame rate of the corresponding transformer layer. In some examples, the label encoder includes a stack of transformer layers, each transformer layer including a normalization layer, a masked multi-head attention layer with relative position coding, a residual connection, a stacking / unstacking layer, and a feedforward layer.

[0006] In some cases, the label encoder may include a bigram embedded lookup decoder model. In some examples, the streaming speech recognition model includes one of the following: a recurrent neural-transducer (RNN-T) model, a transformer-transducer model, a convolutional network-transducer (ConvNet-transducer) model, or a conformer-transducer model. Training the streaming speech recognition model to reduce prediction latency using self-alignment may include using self-alignment without constraining the alignment of the decoding graph using an external aligner model. In some implementations, the streaming speech recognition model runs on a user device or a server. In some examples, each acoustic frame in a sequence of acoustic frames includes a dimensional feature vector.

[0007] Another aspect of the present disclosure provides a computer implementation method for causing data processing hardware to perform an operation to train a streaming speech recognition model to reduce predictive delay using self-alignment when executed on data processing hardware. The operation includes receiving a sequence of acoustic frames corresponding to an utterance as input to the streaming speech recognition model. The streaming speech recognition model is configured to learn alignment probabilities between the sequence of acoustic frames and the output sequence of label tokens. The operation also includes generating a speech recognition result for the utterance as output from the streaming speech recognition model. The speech recognition result includes generating an output sequence of label tokens using a decoding graph. The operation also includes generating a speech recognition model loss based on the speech recognition result and a ground truth transcription of the utterance. The operation also includes obtaining a reference forced alignment path from the decoding graph, which includes reference forced alignment frames, and identifying from the decoding graph the frame to the left of each reference forced alignment frame in the reference forced alignment path. The operation also includes summing the label transition probabilities based on the identified frames to the left of each forced alignment frame in the reference forced alignment path, and updating the streaming speech recognition model based on the sum of label transition probabilities and the speech recognition model loss.

[0008] Implementations of the present disclosure may include one or more of the following features. In some implementations, the operation further includes: generating higher-order feature representations of corresponding acoustic frames in a sequence of acoustic frames at each of several time steps by an audio encoder of the streaming speech recognition model; receiving a sequence of non-blank symbols output by a final softmax layer as input to a label encoder of the streaming speech recognition model; generating dense representations at each of several time steps by the label encoder; receiving the higher-order feature representations generated by the audio encoder and the dense representations generated by the label encoder at each of several time steps as input to a joint network of the streaming speech recognition model; and generating probability distributions across possible speech recognition hypotheses at each of several time steps by the joint network. In some examples, the label encoder includes a stack of transformer layers, each transformer layer including a normalization layer, a masked multi-head attention layer with relative position coding, residual connections, stacking / unstacking layers, and a feedforward layer. The label encoder may also include a bigram embedded lookup decoder model.

[0009] In some implementations, the streaming speech recognition model includes a transformer-transducer model. The audio encoder may include a stack of transformer layers, each transformer layer comprising a normalization layer, a masked multi-head attention layer with relative position coding, residual connectivity, a stacking / unstacking layer, and a feedforward layer. Here, the stacking / unstacking layer may be configured to adjust the processing time by the transformer-transducer model between training and inference by changing the frame rate of the corresponding transformer layer.

[0010] In some implementations, the streaming speech recognition model includes one of the following: a recurrent neural-transducer (RNN-T) model, a transformer-transducer model, a convolutional network-transducer (ConvNet-transducer) model, or a conformer-transducer model. The streaming speech recognition model may run on a user device or a server. In some examples, the operation further includes training the streaming speech recognition model to reduce prediction latency using self-alignment without using an external aligner model to constrain the alignment of the decoding graph.

[0011] Details of one or more implementations of this disclosure are described in the accompanying drawings and the following description. Other embodiments, features, and advantages will become apparent from the description and drawings, as well as the claims. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic diagram of an audio environment that implements a transducer model for performing streaming speech recognition. [Figure 2] This is a schematic diagram of an exemplary transducer model architecture. [Figure 3] This is a plot of an exemplary decoding graph showing the self-alignment path and the forced alignment path. [Figure 4] This is a schematic diagram of an exemplary transformer architecture. [Figure 5] This is a flowchart illustrating an exemplary configuration of how self-alignment is used to reduce streaming ASR model latency. [Figure 6] This is a schematic diagram of an exemplary computing device that may be used to implement the systems and methods described herein. [Modes for carrying out the invention]

[0013] The same reference numeral in various drawings indicates the same element.

[0014] Automatic speech recognition (ASR) systems aim to achieve not only quality / accuracy (e.g., low word error rate (WER)) but also low latency (e.g., short delay between user utterances and resulting transcriptions). Recently, end-to-end (E2E) ASR models have become widely used to achieve state-of-the-art performance in terms of accuracy and latency. In contrast to traditional hybrid ASR systems that include separate acoustic, pronunciation, and language models, E2E models apply a sequence-to-sequence method to learn acoustic and language models together in a single neural network that is trained end-to-end from training data, e.g., utterance-transcription pairs. Here, an E2E model refers to a model whose architecture consists entirely of a neural network. A complete neural network functions without external and / or manually designed components (e.g., finite-state transducers, dictionaries, or text normalization modules). Also, when training E2E models, these models generally do not require bootstrapping from decision trees or time synchronization from separate systems.

[0015] When using an ASR system, there may be a requirement for the ASR system to decode voiced speech in a streaming manner that corresponds to displaying a description of the voiced speech in real-time or, in some cases, faster then real-time, when the user is speaking. As an example, when an ASR system is displayed on a user computing device that experiences direct user interaction, such as a mobile phone, an application that runs on the user device and uses the ASR system (e.g., a digital assistant application) needs to stream speech recognition so that words, word pieces, and / or individual characters are displayed on the screen as they are spoken. Also, the user of the user device may have a low tolerance for latency. For example, when the user speaks a query that requests the digital assistant to retrieve detailed information from a calendar application in order to check their schedule, the user expects the digital assistant to provide a response that conveys the retrieved detailed information as quickly as possible. Due to such low tolerance, the ASR system works hard on the user device to minimize the impact of latency and inaccuracy, which may negatively affect the user experience.

[0016] One form of sequence-to-sequence model called a recurrent neural network transducer (RNN-T) does not use an attention mechanism and, unlike other sequence-to-sequence models that generally need to process the entire sequence (audio waveform) to generate output (e.g., a sentence), RNN-T processes input samples sequentially and streams output symbols. This is particularly attractive for real-time communication. For example, speech recognition with an RNN-T may output one character at a time in response to an utterance. Here, the RNN-T predicts the next symbol using a feedback loop that sends the symbol predicted by the model back to the RNN-T itself. Decoding an RNN-T involves beam search by a single neural network rather than a large decoder graph, so RNN-Ts can scale to a fraction of the size of a server-based speech recognition model. The reduction in size means that the entire RNN-T may be deployed on a device and may even be able to operate offline (i.e., without a network connection), thus avoiding reliability issues with communication networks. RNN-T models, which utilize long-shorter-term memory (LSTM) to provide sequence encoders, are generally well-suited for providing streaming transcription capabilities and latency-sensitive applications that recognize conversational queries (e.g., "Please set the timer," "Don't forget to buy the milk"), but their limited ability to pre-fetch audio-text means they still lag behind state-of-the-art conferencing models (e.g., server-based models with separate AM, PM, and LM) and attention-based sequence-to-sequence models (e.g., ListenAttendSpelling (LAS)) in terms of quality (e.g., speech recognition accuracy, often measured by word error rate (WER)).

[0017] In recent years, Transformer-Transducer (T-T) and Conformer-Transducer (C-T) model architectures have been introduced to further improve the RNN-T model architecture by replacing the LSTM layers in the audio encoder and / or prediction network with their respective Transformer layers or Conformer layers. Generally, the T-T and C-T model architectures can access future audio frames (e.g., right context) when calculating self-attention in their respective Transformer layers or Conformer layers. Therefore, the T-T and C-T model architectures can utilize the future right context to operate in a non-streaming transcription mode and improve the speech recognition performance when the latency constraint is relaxed. That is, the duration of the prediction delay is proportional to the amount of future-accessed audio frames. However, like RNN-T, the T-T and C-T model architectures can also operate in a streaming transcription mode where self-attention depends only on past acoustic frames (e.g., left context).

[0018] Streaming speech recognition models, such as transducer models (e.g., RNN-T, TT, and CT), optimize sequence likelihood without latency constraints, and therefore, these models learn to improve predictions by using context further into the future, resulting in significant latency between audio input and predicted text. Recent techniques for reducing prediction latency include constrained alignment techniques that penalize word boundaries based on audio alignment information obtained from an external alignment model by masking alignment paths that exceed a given threshold latency. While this technique is effective in reducing latency for streaming end-to-end models, it requires a highly accurate external alignment model to minimize WER degradation, which can further complicate the model training step. Other techniques that blindly reduce latency by selecting the most efficient direction in the RNN-T decoding graph often select a direction that is not optimal for all audio inputs due to a lack of alignment information, thereby further worsening the latency-WER trade-off.

[0019] To mitigate the drawbacks of simply reducing latency blindly by using an external alignment model or selecting the most efficient direction from the decoding graph, the implementations herein aim to reduce predicted latency in a streaming speech recognition model by using self-alignment. In particular, self-alignment does not require the use of an external alignment model and does not blindly optimize latency; instead, it utilizes a reference forced alignment learned from a trained speech recognition model to select the optimal low-latency direction for reducing latency. The reference forced alignment may include Viterbi forced alignment. That is, self-alignment always identifies a path in the decoding graph one frame to the left of the Viterbi forced alignment at each time step. Self-alignment has advantages over existing methods for constraining latency. First, since self-alignment does not require an external alignment model, the complexity of training self-alignment is much lower than that of supervised methods. Second, self-alignment has minimal impact on ASR training by constraining only the most probable alignment path. In contrast, other methods affect many alignment paths by masking them or changing the weights of the label transition probabilities. Since the delay-constrained regularization term conflicts with the primary ASR loss, it is important to minimize intervention on the primary loss in optimizing the delay-performance trade-off. Self-alignment simply regularizes a single path by pushing it to the left.

[0020] Figure 1 shows an example of a voice environment 100. In the voice environment 100, the method by which a user 104 interacts with a computing device such as a user device 10 may be via voice input. The user device 10 (also commonly referred to as device 10) is configured to take in voice (e.g., streaming audio data) from one or more users 104 within the voice environment 100. Here, streaming audio data may refer to audible queries, commands for device 10, or utterances 106 by user 104 acting as audible communications to be taken in by device 10. The voice-enabled system of device 10 may process queries or commands by responding to queries and / or by having one or more downstream applications execute / complete commands.

[0021] User device 10 may correspond to any computing device associated with user 104 that can receive audio data. Some examples of user device 10 include, but are not limited to, mobile devices (e.g., cell phones, tablets, laptops), computers, wearable devices (e.g., smartwatches), smart home appliances, Internet of Things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. User device 10 includes data processing hardware 12 and memory hardware 14 that communicates with the data processing hardware 12 and stores instructions that cause the data processing hardware 12 to perform one or more actions when executed by the data processing hardware 12. User device 10 further includes an audio system 16 having voice acquisition devices (e.g., microphones) 16, 16a for capturing speech 106 in the voice environment 100 and converting it into electrical signals, and voice output devices (e.g., speakers) 16, 16b for communicating audible audio signals (e.g., as output audio data from device 10). In the illustrated example, the user device 10 implements a single audio acquisition device 16a, but the user device 10 may implement an array of audio acquisition devices 16a without departing from the scope of this disclosure, and one or more acquisition devices 16a in the array may communicate with the audio system 16 without physically being present on the user device 10.

[0022] In the voice environment 100, the Automatic Speech Recognition (ASR) system 118 transducer model 200 resides on the user device 10 of user 104 and / or on a remote computing device 60 (for example, one or more remote servers of a distributed system running in a cloud computing environment) that communicates with the user device 10 via the network 40. The user device 10 and / or the remote computing device 60 also include an audio subsystem 108 configured to receive utterances 106 by user 104 captured by a speech capture device 16a and convert the utterances 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by the ASR system 118. In the illustrated example, the user utters each utterance 106, and the audio subsystem 108 converts the utterances 106 into corresponding audio data (for example, an acoustic frame) 110 that is input to the ASR system 118. Subsequently, the transducer model 200 receives audio data 110 corresponding to the utterance 106 as input and generates / predicts a corresponding transcription 120 (e.g., recognition result / hypothesis) of the utterance 106 as output. The transducer model 200 provides streaming speech recognition model results that do not have access to look-ahead audio and therefore provides real-time streaming transcription functionality as the user 104 utters the utterance 106. For example, a digital assistant application 50 running on a user device 10 may need to stream speech recognition so that words, word pieces, and / or individual characters are displayed on the screen as they are spoken.

[0023] User device 10 and / or remote computing device 60 also run a user interface generator 107 configured to present a representation of the transcription 120 of the utterance 106 to the user 104 of user device 10. As described in detail below, the user interface generator 107 may display a partial speech recognition result 120a in a streaming manner for time 1, and then display the final speech recognition result 120b for time 2. In some configurations, the transcription 120 output from the ASR system 118 is processed by a natural language understanding (NLU) module running, for example, on user device 10 or remote computing device 60, to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (for example, running on any combination of user device 10 or remote computing device 60) may convert the transcription into synthesized speech for audible output by user device 10 and / or another device.

[0024] In the illustrated example, user 104 interacts with a program or application 50 (for example, a digital assistant application 50) on user device 10 that uses the ASR system 118. For example, Figure 1 shows user 104 communicating with digital assistant application 50, with the digital assistant application 50 displaying a digital assistant interface 18 on the screen of user device 10, illustrating a conversation between user 104 and digital assistant application 50. In this example, user 104 asks digital assistant application 50, "What time is tonight's concert?" This question from user 104 is an utterance 106 that is captured by the speech capture device 16a and processed by the audio system 16 of user device 10. In this example, the audio system 16 receives the utterance 106 and converts it into an acoustic frame 110 that is input to the ASR system 118.

[0025] Continuing with the example, the transducer model 200 encodes the acoustic frame 110 corresponding to the utterance 106 as user 104 speaks, and then decodes the encoded acoustic frame 110 into a partial speech recognition result 120a. During time 1, the user interface generator 107 presents the representation of the partial speech recognition result 120a of the utterance 106 to user 104 on user device 10 via the digital assistant interface 18 in a streaming manner, thereby displaying words, word pieces, and / or individual characters on the screen as they are spoken. In some examples, the first look-ahead audio context is equal to zero.

[0026] During time 2, the user interface generator 107 presents the user 104 of the user device 10 with a representation of the final speech recognition result 120b of the utterance 106 via the digital assistant interface 18. The final speech recognition result 120b may simply be the partial speech recognition result 120a at the time the user finishes speaking. In some cases, the ASR system 118 may include another speech recognition and / or use an external language model to rescore the partial speech recognition result. In some cases, the same transducer model 200 may process the audio again after the user finishes speaking and instead use a right-look-ahead audio context to generate the final speech recognition result 120b. This disclosure is not concerned with how the final speech recognition result 120b is obtained, but rather with the limitations of latency in the streaming partial speech recognition result 120a output by the transducer model 200.

[0027] In the example shown in Figure 1, the digital assistant application 50 may use natural language processing to respond to a question posed by the user 104. Natural language processing generally refers to the process of interpreting written language (e.g., partial speech recognition result 120a and / or final speech recognition result 120b) and determining whether the written language prompts any action. In this example, the digital assistant application 50 uses natural language processing to recognize that the question from user 104 is about the user's schedule, and more specifically, about a concert on the user's schedule. By recognizing such details through natural language processing, the automated assistant returns a response 19 to the user's query, in this case the response 19 presents: "The venue opens at 6:30 p.m. and the concert starts at 8:00 p.m." In some configurations, natural language processing is performed on a remote server 60 that communicates with the data processing hardware 12 of the user device 10.

[0028] Referring to Figure 2, the transducer model 200 may enable end-to-end (E2E) speech recognition by integrating the acoustic model, pronunciation model, and language model into a single neural network, in which case neither dictionaries nor separate text normalization components are required. Various structures and optimization mechanisms can improve accuracy and reduce model training time. In the illustrated example, the transducer model 200 includes a transformer-transducer (TT) model architecture, which adheres to latency constraints associated with interactive applications. The TT model 200 has a small computational footprint and lower memory requirements than conventional ASR architectures, making it suitable for performing speech recognition across the entire user device 10 (e.g., communication with a remote server 60 is not required). The TT model 200 includes an audio encoder 210, a label encoder 220, and a joint network 230. The audio encoder 210 is roughly similar to the acoustic model (AM) in conventional ASR systems and includes a neural network with multiple transformer layers. For example, the audio encoder 210 takes a sequence of d-dimensional feature vectors (e.g., acoustic frame 110 (Figure 1)) x = (x1, x2, ..., x T ) read here

number

[0029] Similarly, the label encoder 220 may also include a transformer layer neural network or a lookup table embedding model, which, like the language model (LM), processes the sequence y0, ..., y0 of the non-blank symbols 242 that have been output by the final softmax layer 240. ui-1 A dense representation 222 (for example, Ih) encodes the predicted label history. u It is processed as (represented as). In implementations where the label encoder 220 includes a neural network of transformer layers, each transformer layer may include a normalization layer, a masked multi-head attention layer with relative position coding, residual connections, a feedforward layer, and a dropout layer. In these implementations, the label encoder 220 may include two transformer layers. In implementations where the label encoder 220 includes a lookup table embedding model with a bi-gram label context, the embedding model is configured to learn a d-dimensional weight vector for each possible bigram label context, where d is the dimension of the output of the audio encoder 210 and the label encoder 220. In some examples, the total number of parameters in the embedding model is N 2 ×d is where N is the vocabulary size for the labels. Here, the learned weight vector is then used as an embedding of the bigram label context in the TT model 200 to generate the execution time of the fast label encoder 220.

[0030] Finally, in the TT model architecture, the representations generated by the audio encoder 210 and the label encoder 220 are in a dense layer J. u,t These are combined by the joint network 230. The joint network 230 then predicts the alignment distribution (e.g., alignment probability 232) over the following output symbols, as shown in the following equation. Pr(z u,t |x,t,y1,…, y u-1) (1) In the above equation, x is the audio input, y is the grand true label sequence, and z is the alignment belonging to y. In other words, the joint network 230 generates a probability distribution 232 over possible speech recognition hypotheses at each output step (e.g., time step). Here, the "possible speech recognition hypotheses" correspond to a set of output labels (also called "speech units") each represented by a grapheme (e.g., symbol / character) or word piece in the specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, for example, one label for each of the 26 characters in the English alphabet and one label specifying a space. Thus, the joint network 230 may output a set of values indicating the likelihood of occurrence of each of a predetermined set of output labels. This set of values can be a vector (e.g., a one-hot vector) and can indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and optionally punctuation and other symbols), but the set of output labels is not limited as such. For example, the set of output labels can include word pieces and / or whole words in addition to or instead of graphemes. The output distribution of the joint network 230 can include posterior probability values for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output z of the joint network 230 u,t can include 100 different probability values, one for each output label. The probability distribution can then be used to select scores in a beam search process (e.g., by the softmax layer 240) and assign them to candidate orthographic elements (e.g., graphemes, word pieces, and / or words) to determine the transcription 120.

[0031] The softmax layer 240 may, using any technique, select the output label / symbol with the highest probability in the distribution as the next output symbol 242 predicted by the TT model 200 at the corresponding output step. Thus, the set of output symbols 242 predicted by the TT model 200 is sometimes collectively referred to as the output sequence of label tokens 242. In this way, the TT model 200 does not make a conditional independence assumption, and the prediction of each symbol is conditional not only on acoustics but also on the sequence of labels that have been output so far.

[0032] To determine the log-conditional probability of y given an audio input x, we sum all alignment distributions corresponding to y, as shown in the following equation.

number

[0033] Figure 2 shows transducer model 200 including a TT model architecture, but transducer model 200 may include, without departing from the scope of this disclosure, an RNN-T model architecture, a convolutional neural network-transducer (CNN-transducer) model architecture, a convolutional network-transducer (ConvNet-transducer) model, or a conformer-transducer model architecture. An exemplary CNN-transducer model architecture is described in detail in "Contextnet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context," https: / / arxiv.org / abs / 2005.03191, which is incorporated herein by reference in its entirety. An exemplary conformer-transducer model architecture is described in detail in "Conformer: Convolution-augmented transformer for speech recognition," https: / / arxiv.org / abs / 2005.08100, which is incorporated herein by reference in its entirety.

[0034] The transducer model 200 is trained on a training dataset of audio data corresponding to utterances paired with corresponding transcriptions. Training of the transducer model 200 may be performed on a remote server 60, and the trained transducer model 200 may be pushed to the user device 10. The transducer model 200 is trained using cross-entropy error based on Viterbi forced alignment. Alignment delay includes the delay between the input audio frame and the stream-processed decoded output label. Conventional models train the alignment model through dialogue using realigned labels, allowing them to learn accurate alignment after multiple iterations. TT or CT models, which access future frames when computing self-attention in each transformer or conformer layer, may include alignment delays consistent with conventional models. However, transducer models in streaming mode, where self-attention relies only on past frames, will have alignment delays.

[0035] The implementation described herein aims to reduce the predicted latency in a streaming transducer model 200 by using self-alignment. In particular, self-alignment does not require the use of an external alignment model and does not blindly optimize latency, but instead uses a reference forced alignment learned from a trained speech recognition model to select the optimal low-latency direction for reducing latency. The reference forced alignment may include Viterbi forced alignment. That is, self-alignment always identifies the path in the decoding graph one frame to the left of the Viterbi forced alignment at each time step.

[0036] Figure 3 shows a plot of the decoding graph 300 for transducer model 200 having a TT model architecture for the output sequence of label token 242 (Figure 2) "I like it". The x-axis represents the respective acoustic frame at each time step, and the y-axis represents the output label token 242 (Figure 2). Circles and arrows that are not thick solid lines represent the respective tokens that are not included in the alignment paths described below. The constrained alignment path 310 (represented by thick circles and thick arrows as shown in Figure 3) includes a word boundary threshold equal to 2. The forced alignment path 320 (represented by dotted circles and dotted arrows as shown in Figure 3, for example) (also called the reference forced alignment path 320) and the left alignment path 330 (represented by dashed circles, for example) learned from the trained transducer model 200 include one frame to the left of every frame in the forced alignment path 320. During the training of the transducer model 200, for each training batch, self-alignment always pushes the model's forced alignment path 320 to the left, thereby facilitating the left alignment path 330 (represented by the dashed circle and dashed arrow as shown in Figure 3). The training loss may be expressed as follows:

number

[0037] Figure 4 shows an exemplary transformer layer 400 between multiple transformer layers of the audio encoder 210. Here, between each time step, the initial transformer layer 400 receives the corresponding acoustic frame 110 as input and generates the corresponding output representation / embedding 450 which is received as input by the next transformer layer 400. That is, each transformer layer 400 following the initial transformer layer 400 may receive an input embedding 450 corresponding to the output representation / embedding generated as output by the previous transformer layer 400. The final transformer layer 400 (e.g., the last transformer layer in the final stack 320) generates a higher-order feature representation 202 (e.g., ah in relation to Figure 2) for the corresponding acoustic frame 110 in each of the multiple time steps. t It generates (represented by).

[0038] The input to the label encoder 220 (Figure 2) is the sequence of non-blank symbols y0, ..., y that has been output by the final softmax layer 240 up to that point. ui-1 It may include a vector (for example, a one-hot vector) that indicates the value. Therefore, when the label encoder 220 includes a transformer layer, the initial transformer layer may receive the input embedding 111 by passing the one-hot vector through a lookup table.

[0039] Each transformer layer 400 of the audio encoder 210 includes a normalization layer 404, a masked multi-head attention layer 406 with relative position coding, a residual connection 408, a stacking / unstacking layer 410, and a feedforward layer 412. The masked multi-head attention layer 406 with relative position coding provides a flexible way to control the amount of look-ahead audio context (i.e., duration) used by the TT model 200. Specifically, after the normalization layer 404 normalizes the acoustic frame 110 and / or the input embedding 111, the masked multi-head attention layer 406 projects the input to a certain value for all heads. The masked multi-head layer 406 may then mask the attention score to the preceding frame of the current acoustic frame 110 to produce an output conditioned only on the previous acoustic frame 110. Next, the weighted average values ​​for all heads are concatenated and passed to a dense layer 416, where residual connections 414 are added to the normalized input and the output of the dense layer 416 to form the final output of the multi-head attention layer 406 with relative position coding. Residual connections 408 are added to the output of the normalization layer 404 by an adder 430 and provided as input to either the masked multi-head attention layer 406 or the feedforward layer 412, respectively. Stacking / unstacking layers 410 can be used to change the frame rate for each transformer layer 400 to accelerate training and inference.

[0040] The feedforward layer 412 applies the normalization layer 404, and then sequentially applies it to the dense layer 1 420, the normalized linear layer (ReLU) 418, and the dense layer 2 416. The ReLU 418 is used as an activation for the output of dense layer 1 420. Similar to the multi-head attention layer 406 with relative position coding, the residual connection 414 of the output from the normalization layer 404 is added to the output of dense layer 2 416 by the adder 430.

[0041] Figure 5 shows a flowchart illustrating an exemplary configuration of operation for Method 500, which trains a streaming speech recognition model to reduce predictive delay using self-alignment. The Method includes, in operation 502, receiving a sequence of acoustic frames 110 corresponding to an utterance 106 as input to a streaming speech recognition model (e.g., a transducer model) 200. The streaming speech recognition model 200 is configured to learn alignment probabilities 232 between the sequence of acoustic frames 110 and the output sequences of label tokens 242. Method 500 includes, in operation 504, generating a speech recognition result 120 for the utterance 106, including the output sequences of label tokens 242, using a decoding graph 300 as output from the streaming speech recognition model 200. Method 500 includes, in operation 506, generating a speech recognition model loss based on the speech recognition result 120 and the ground truth transcription of the utterance 106.

[0042] Method 500 includes, in operation 508, obtaining a reference forced alignment path 320 from the decoding graph 300. Method 500 includes, in operation 510, identifying one frame to the left of each reference forced alignment frame in the reference forced alignment path 320 from the decoding graph 300. Method 500 includes, in operation 512, summing the label transition probabilities based on the identified frame to the left of each forced alignment frame. Method 500 includes, in operation 514, updating the streaming speech recognition model 200 based on the sum of label transition probabilities and the speech recognition model loss.

[0043] Figure 6 is a schematic diagram of an exemplary computing device 600 that may be used to implement the systems and methods described herein. The computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the implementations of the invention described and / or claimed herein.

[0044] The computing device 600 includes a processor 610, memory 620, storage device 630, a high-speed interface / controller 640 connected to memory 620 and high-speed expansion port 650, and a low-speed bus 670 and a low-speed interface / controller 660 connected to storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and may be mounted on a common motherboard or in other ways as needed. The processor 610 can process instructions executed within the computing device 600, including instructions stored in memory 620 or on storage device 630 for displaying graphical information about a graphical user interface (GUI) on an external input / output device such as a display 680 coupled to the high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used with multiple memories and multiple types of memory as needed. Alternatively, multiple computing devices 600 may be connected so that each device performs some of the necessary operations (for example, a server bank, a group of blade servers, or a multiprocessor system).

[0045] Memory 620 stores information non-temporarily within the computing device 600. Memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporarily, memory 620 may be a physical device used to temporarily or persistently store a program (e.g., a sequence of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and disks or tapes.

[0046] The storage device 630 can provide a mass storage device for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 may be an array of devices including a floppy disk device, a hard disk device, an optical disk device, or a tape device, flash memory or other similar solid-state memory device, or a storage network or other configuration. In additional implementations, the computer program product is actually embodied in the information carrier. The computer program product includes instructions that, when executed, perform one or more of the methods described above. The information carrier is a computer or machine-readable medium such as memory 620, the storage device 630, or memory on the processor 610.

[0047] The high-speed controller 640 manages bandwidth-intensive operations for the computing device 600, while the low-speed controller 660 manages lower bandwidth-intensive operations. Such duty cycle allocations are illustrative only. In some implementations, the high-speed controller 640 is coupled to memory 620, display 680 (e.g., via a graphics processor or accelerator), and high-speed expansion port 650. The high-speed expansion port 650 may accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to storage device 630 and low-speed expansion port 690. The low-speed expansion port 690 may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet) and may be coupled to one or more input / output devices such as a keyboard, pointing device, scanner, or networking device such as a switch or router via a network adapter.

[0048] The computing device 600 may be implemented in several different ways, as shown in the illustration. For example, the computing device 600 may be implemented as a standard server 600a, or multiple times within a group of such servers 600a, or as a laptop computer 600b, or as part of a rack server system 600c.

[0049] Various implementations of the systems and techniques described herein can be realized in digital electrical and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor. The programmable processor may be dedicated or general-purpose and may be coupled to receive and transmit data and instructions to and from a storage system, at least one input device, and at least one output device.

[0050] A software application (i.e., a software resource) can refer to computer software that causes a computing device to perform tasks. In some cases, a software application may be called an “application,” “app,” or “program.” Exemplary applications include, but are not limited to, system diagnostic applications, system administration applications, system maintenance applications, document processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0051] Non-temporary memory may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) temporarily or persistently for use by computing devices. Non-temporary memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), phase-change memory (PCM), and disks or tapes.

[0052] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural languages ​​and / or object-oriented programming languages, as well as / or assembly / machine languages. In this specification, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transient computer-readable medium, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0053] The processes and logic flows described herein, also known as data processing hardware, can be executed by one or more programmable processors that perform functions by executing one or more computer programs to act on input data and produce outputs. Processes and logic flows can also be executed by special-purpose logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Suitable processors for executing computer programs include, as an example, both general-purpose and dedicated microprocessors, and any one or more processors in any type of digital computer. Generally, processors receive instructions and data from read-only memory or random-access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operablely coupled to receive data from or transfer data to one or more mass storage devices, or both. However, a computer does not have to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory can be assisted by or integrated into dedicated logic circuits.

[0054] To enable interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen, and optionally a keyboard and pointing device, such as a mouse or trackball, to which the user can provide input to the computer. Interaction with the user may also be enabled using other types of devices. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic input, voice input, or haptic input. The computer may also interact with the user by sending documents to and receiving documents from devices used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0055] Several implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of this disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of Symbols]

[0056] 1-2 hours 10 User Devices 12. Data Processing Hardware 14. Memory Hardware 16 Audio Systems 16a Audio Capture Device 16b Audio output device 18 Digital Assistant Interface 19 Response 50 Digital Assistant Applications 60 Remote Computing Devices 100 Audio Environment 104 users 106 utterances 107 User Interface Generator 108 Audio Subsystem 110 acoustic frames, audio data 111 entered embeds 118 ASR System 120 Transcriptions 120a Partial speech recognition result 120b Final speech recognition result 200 TT model, transducer model 202 Higher-Order Feature Representations 210 Audio Encoders 220 Label Encoder 230 Joint Network 232 Alignment Probabilities 240 Softmax layer 242 Output symbols, non-blank symbols, label tokens 300 Decoding Graph 310 Alignment Path 320 Forced Alignment Path 330 Left alignment path 400 Transformer Layers 404 Normalization layer 406 Masked multi-head attention layer 408, 414 residual connection 410 stacking / unstacking layers 412 Feedforward Layers 416, 420 Dense layer 418 Normalized linear layer 430 Adder 450 Output representation / embedding 600 Computing Devices 600a Server 600b Laptop Computer 600c Rack Server System 610 Processor 620 memory 630 Storage Devices 640 High-Speed ​​Interfaces / Controllers 650 high-speed expansion ports 660 Low-Speed ​​Interface / Controller 670 Slow Bus 680 displays 690 Low-Speed ​​Expansion Ports

Claims

1. A computer implementation method that runs on data processing hardware, The computer implementation method causes the data processing hardware to perform a plurality of operations, and the plurality of operations The operation of receiving a sequence of acoustic frames corresponding to the utterance paired with a ground truth transcription of the utterance, An operation to process a sequence of acoustic frames in order to generate a decoding graph for a sequence of label tokens using a streaming speech recognition model, wherein the output sequence of label tokens includes speech recognition results for the utterance. An operation to determine the speech recognition model loss based on the speech recognition result and the ground truth transcription of the utterance, An operation to obtain a forced alignment path including forced alignment frames from the decoding graph, the operation including identifying the left frame for each reference forced alignment frame in the reference forced alignment path, An operation to determine the training loss based on the sum of the speech recognition model loss and the label transition probabilities based on the forced alignment path, The operation of training the streaming speech recognition model based on the aforementioned training loss and Computer implementation methods, including those mentioned above.

2. The aforementioned multiple operations, The audio encoder of the streaming speech recognition model performs the operation of generating a higher-order feature representation for the corresponding acoustic frame in the sequence of acoustic frames at each of a plurality of time steps, The aforementioned streaming speech recognition model receives a sequence of non-blank symbols output by the final softmax layer as input to the label encoder, The label encoder performs the operation of generating a dense representation in each of the plurality of time steps, The operation of receiving, as input to the joint network of the streaming speech recognition model, the higher-order feature representation generated by the audio encoder at each of the multiple time steps, and the dense representation generated by the label encoder at each of the multiple time steps, The joint network performs the operation of generating a probability distribution over the speech recognition hypothesis in each of the plurality of time steps, and The computer implementation method according to claim 1, further comprising:

3. The label encoder includes a stack of transformer layers, and each transformer layer is Normalized layer and, A masked multi-head attention layer with relative position coding, Residual connection and, Stacking / unstacking layers, Feedforward layer and The computer implementation method according to claim 2, including the method described in claim 2.

4. The computer implementation method according to claim 2, wherein the label encoder includes a bigram embedded lookup decoder model.

5. The computer implementation method according to claim 2, wherein the audio encoder includes a plurality of multi-head attention layers.

6. The audio encoder includes a stack of transformer layers, and each transformer layer is Normalized layer and, A masked multi-head attention layer with relative position coding, Residual connection and, Stacking / unstacking layers, Feedforward layer and The computer implementation method according to claim 2, including the method described in claim 2.

7. The computer implementation method according to claim 1, wherein the streaming speech recognition model includes a recurrent neural network transducer (RNN-T) model architecture.

8. The computer implementation method according to claim 1, wherein the data processing hardware resides in the server.

9. The computer implementation method according to claim 8, wherein the trained streaming speech recognition model is configured to run on a user device after the streaming speech recognition model has been trained.

10. The computer implementation method according to claim 1, wherein training the streaming speech recognition model based on the training loss includes training the streaming speech recognition model based on the training loss without using an external aligner model to constrain the alignment of the decoding graph.

11. It is a system, Data processing hardware and The memory hardware that communicates with the aforementioned data processing hardware The memory hardware, when executed in the data processing hardware, records instructions that cause the data processing hardware to perform a plurality of operations, and the plurality of operations The operation of receiving a sequence of acoustic frames corresponding to the utterance paired with a ground truth transcription of the utterance, An operation to process a sequence of acoustic frames in order to generate a decoding graph for a sequence of label tokens using a streaming speech recognition model, wherein the output sequence of label tokens includes speech recognition results for the utterance. An operation to determine the speech recognition model loss based on the speech recognition result and the ground truth transcription of the utterance, An operation to obtain a forced alignment path including forced alignment frames from the decoding graph, the operation including identifying the left frame for each reference forced alignment frame in the reference forced alignment path, An operation to determine the training loss based on the sum of the speech recognition model loss and the label transition probabilities based on the forced alignment path, The operation of training the streaming speech recognition model based on the aforementioned training loss and A system that includes this.

12. The aforementioned multiple operations, The audio encoder of the streaming speech recognition model performs the operation of generating a higher-order feature representation for the corresponding acoustic frame in the sequence of acoustic frames at each of a plurality of time steps, The aforementioned streaming speech recognition model receives a sequence of non-blank symbols output by the final softmax layer as input to the label encoder, The label encoder performs the operation of generating a dense representation in each of the plurality of time steps, The operation of receiving, as input to the joint network of the streaming speech recognition model, the higher-order feature representation generated by the audio encoder at each of the multiple time steps, and the dense representation generated by the label encoder at each of the multiple time steps, The joint network performs the operation of generating a probability distribution over the speech recognition hypothesis in each of the plurality of time steps, and The system according to claim 11, further comprising:

13. The label encoder includes a stack of transformer layers, and each transformer layer is Normalized layer and, A masked multi-head attention layer with relative position coding, Residual connection and, Stacking / unstacking layers, Feedforward layer and The system according to claim 12, including the system described in claim 12.

14. The system according to claim 12, wherein the label encoder includes a bigram embedded lookup decoder model.

15. The system according to claim 12, wherein the audio encoder includes a plurality of multi-head attention layers.

16. The audio encoder includes a stack of transformer layers, and each transformer layer is Normalized layer and, A masked multi-head attention layer with relative position coding, Residual connection and, Stacking / unstacking layers, Feedforward layer and The system according to claim 12, including the system described in claim 12.

17. The system according to claim 11, wherein the streaming speech recognition model includes a recurrent neural network transducer (RNN-T) model architecture.

18. The system according to claim 11, wherein the data processing hardware resides in the server.

19. The system according to claim 18, wherein the trained streaming speech recognition model is configured to run on a user device after the streaming speech recognition model has been trained.

20. The system according to claim 11, wherein training the streaming speech recognition model based on the training loss includes training the streaming speech recognition model based on the training loss without using an external aligner model to constrain the alignment of the decoding graph.

Images