Reducing streaming asr model latency with self-alignment

Abstract

A streaming speech recognition model (200) includes an audio encoder (210) configured to receive a sequence of acoustic frames (110) and generate a high-level feature representation (202) for a respective acoustic frame in the sequence of acoustic frames. The streaming speech recognition model also includes a label encoder (220) configured to receive a sequence of non-blank symbols (242) output by a final softmax layer (240) and generate a dense representation (222). The streaming speech recognition model also includes a joint network (230) configured to receive the high-level feature representation generated by the audio encoder and the dense representation generated by the label encoder and generate a probability distribution (232) over possible speech recognition hypotheses. Here, the streaming speech recognition model is trained using self-alignment to reduce prediction latency by encouraging alignment paths that go left by 1 frame from a reference forced alignment frame.

Description

Technical Field

[0001] This disclosure relates to using self-alignment to reduce latency in streaming automatic speech recognition (ASR) models. Background Technology

[0002] Automatic speech recognition (ASR), the process of taking audio input and transcribing it into text, has become a vital technology widely used in mobile devices and other applications. Typically, ASR attempts to provide an accurate transcription of what a person is saying by taking audio input (e.g., spoken words) and transcribing it into text. Based on the continuous development of deep neural networks, modern ASR models have continuously improved in both accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay between the user speaking and the transcription). Using ASR systems today requires them to decode speech in a streaming manner, corresponding to real-time or even faster, but also accurate. However, streaming end-to-end models that optimize sequence likelihood without any latency constraints suffer from high latency between the audio input and the predicted text because these models learn to improve their predictions by using more future context. Summary of the Invention

[0003] One aspect of this disclosure provides a streaming speech recognition model, including an audio encoder configured to: receive a sequence of acoustic frames as input; and at each of a plurality of time steps, generate a high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The streaming speech recognition model further includes a label encoder configured to receive a sequence of non-whitespace symbols output by a final softmax layer as input; and generate a dense representation at each of the plurality of time steps. The streaming speech recognition model also includes a joint network configured to: receive the high-order feature representation generated by the audio encoder at each of the plurality of time steps and the dense representation generated by the label encoder at each of the plurality of time steps as input; and at each of the plurality of time steps, generate a probability distribution over possible speech recognition hypotheses at the corresponding time step. Here, self-alignment is used to train the streaming speech recognition model to reduce prediction latency by encouraging an alignment path of one frame to the left of a reference forced alignment frame at each time step for each training batch.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the streaming speech recognition model may include a transducer model. In some implementations, the audio encoder may include a stack of transducer layers, wherein each transducer layer includes: a normalization layer; a masked multi-head attention layer with relative position encoding; a residual connection; a stacked / destacking layer; and a feedforward layer. Here, the stacked / destacking layer may be configured to change the frame rate of the corresponding transducer layer to adjust the processing time of the transducer model during training and inference. In some examples, the label encoder includes a stack of transducer layers, wherein each transducer layer includes: a normalization layer; a masked multi-head attention layer with relative position encoding; a residual connection; a stacked / destacking layer; and a feedforward layer.

[0005] Optionally, the label encoder may include a bigram embedding lookup decoder model. In some examples, the streaming speech recognition model includes one of the following: a recurrent neural transducer (RNN-T) model; a transformer-transducer model; a convolutional network-transducer (ConvNet-Transducer) model; or a Conformer-transducer model. Training the streaming speech recognition model using self-alignment to reduce prediction latency may include using self-alignment without using any external aligner model to constrain the alignment of the decoded graph. In some implementations, the streaming speech recognition model is executed on a user device or server. In some examples, each acoustic frame in the sequence of acoustic frames includes a dimensional feature vector.

[0006] Another aspect of this disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for training a streaming speech recognition model using self-alignment to reduce prediction latency. The operations include receiving a sequence of acoustic frames corresponding to a utterance as input to the streaming speech recognition model. The streaming speech recognition model is configured to learn alignment probabilities between the sequence of acoustic frames and an output sequence of tag tokens. The operations further include generating a speech recognition result for the utterance as output from the streaming speech recognition model. The speech recognition result includes an output sequence of tag tokens using a decoded graph. The operations further include generating a speech recognition model loss based on the speech recognition result and a true transcription of the utterance. The operations further include obtaining a reference forced alignment path including reference forced alignment frames from the decoded graph; and identifying one frame to the left of each reference forced alignment frame in the reference forced alignment path from the decoded graph. The operations further include summing tag transition probabilities based on the identified frames to the left of each forced alignment frame in the reference forced alignment path; and updating the streaming speech recognition model based on the sum of the tag transition probabilities and the speech recognition model loss.

[0007] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the operation further includes generating, at each of a plurality of time steps, a high-order feature representation for a corresponding acoustic frame in a sequence of acoustic frames by an audio encoder of the streaming speech recognition model; receiving a sequence of non-whitespace symbols output by a final softmax layer as input to a tag encoder of the streaming speech recognition model; generating a dense representation by the tag encoder at each of the plurality of time steps; receiving the high-order feature representation generated by the audio encoder at each of the plurality of time steps and the dense representation generated by the tag encoder at each of the plurality of time steps as input to a joint network of the streaming speech recognition model; and generating a probability distribution on possible speech recognition hypotheses at each of the plurality of time steps by the joint network. In some examples, the tag encoder includes a stack of transformer layers, wherein each transformer layer includes: a normalization layer; a masked multi-head attention layer with relative position encoding; a residual connection; a stacking / destacking layer; and a feedforward layer. The tag encoder may include a binary embedding lookup decoder model.

[0008] In some implementations, the streaming speech recognition model includes a transducer model. The audio encoder may include a stack of transducer layers, wherein each transducer layer includes: a normalization layer; a masked multi-head attention layer with relative position encoding; a residual connection; a stacked / destacking layer; and a feedforward layer. Here, the stacked / destacking layer may be configured to change the frame rate of the corresponding transducer layer to adjust the processing time of the transducer model during training and inference.

[0009] In some implementations, the streaming speech recognition model includes one of the following: a recurrent neural transducer (RNN-T) model; a transformer-transducer model; a convolutional network-transducer (ConvNet-Transducer) model; or a Conformer-transducer model. The streaming speech recognition model can be executed on a user device or a server. In some examples, the operation further includes using self-alignment to train the streaming speech recognition model to reduce prediction latency, without using any external aligner model to constrain the alignment of the decoded graph.

[0010] Details of one or more embodiments of this disclosure are set forth in the following drawings and description. Other aspects, features, and advantages will become apparent from the description and drawings and from the claims. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of a speech environment that implements a transducer model to perform streaming speech recognition.

[0012] Figure 2 This is a schematic diagram of the example transducer model architecture.

[0013] Figure 3 This is a diagram depicting example decoding diagrams of self-aligned paths and forced alignment paths.

[0014] Figure 4 This is a schematic diagram of an example converter architecture.

[0015] Figure 5 This is a flowchart of an example layout for a method to reduce latency in streaming ASR models by utilizing self-alignment.

[0016] Figure 6 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described in this paper.

[0017] In the various figures, the same reference numerals indicate the same elements. Detailed Implementation

[0018] Automatic speech recognition (ASR) systems focus on providing not only quality / accuracy (e.g., low word error rate (WER)) but also low latency (e.g., a short delay between the user speaking and the occurrence of the transcription). Recently, end-to-end (E2E) ASR models have gained prominence for achieving state-of-the-art performance in both accuracy and latency. Compared to conventional hybrid ASR systems that include separate acoustic, articulation, and language models, E2E models apply a sequence-to-sequence approach to jointly learn acoustic and language modeling in a single neural network, which is trained end-to-end on training data (e.g., utterance-transcription pairs). Here, an E2E model refers to a model whose architecture is entirely built from neural networks. These fully neural networks operate without external and / or manually designed components (e.g., finite-state transducers, dictionaries, or text normalization modules). Additionally, when training E2E models, these models typically do not require bootstrapping from decision trees or temporal alignment from separate systems.

[0019] When using ASR systems today, it may be necessary for the ASR system to decode speech in a streaming manner, which corresponds to displaying a description of the speech in real time or even faster than real time as the user speaks. To illustrate, when an ASR system is displayed on a user computing device (such as a mobile phone) experiencing direct user interaction, applications running on the user device and using the ASR system (such as digital assistant applications) may require speech recognition to be streaming, so that words, chunks of words, and / or individual characters appear on the screen as soon as they are spoken. Additionally, users of the user device may also have a low tolerance for latency. For example, when a user utters a query requesting a digital assistant to retrieve details of an upcoming appointment from a calendar application, the user expects the digital assistant to provide a response conveying the retrieved details as quickly as possible. Due to this low tolerance, ASR systems strive to operate on the user device in a way that minimizes the impact of latency and inaccuracies that could adversely affect the user experience.

[0020] One form of sequence-to-sequence model, known as a recurrent neural network transducer (RNN-T), does not employ an attention mechanism. Unlike other sequence-to-sequence models that typically require processing the entire sequence (e.g., an audio waveform) to produce an output (e.g., a sentence), RNN-T processes input samples continuously and streams output symbols, a feature particularly attractive for real-time communication. For example, speech recognition using RNN-T can output characters one after another as they are spoken. Here, RNN-T uses feedback loops that feed back the symbols predicted by the model to themselves to predict the next symbol. Because decoding an RNN-T involves beam searching through a single neural network rather than a large decoder graph, RNN-T can scale to a fraction of the size of server-based speech recognition models. With this reduction in size, RNN-T can be deployed entirely on a device and can operate offline (i.e., without a network connection); thus avoiding the unreliability issues associated with communication networks. While well-suited for providing streaming transcription capabilities that typically recognize conversational queries (e.g., “set a timer”, “remind me to buy milk”, etc.) and for latency-sensitive applications, RNN-T models that utilize Long Short-Term Memory (LSTM) to provide sequence encoders have limited look-ahead capabilities for audio context, thus still lagging behind large, state-of-the-art conventional models (e.g., server-based models with separate AM, PM, and LM) and attention-based sequence-to-sequence models (e.g., Listen-Attention-Spelling (LAS)) in terms of quality (e.g., speech recognition accuracy, typically measured by word error rate (WER)).

[0021] Recently, Transformer-Transducer (TT) and Conformer-Transducer (CT) model architectures have been introduced to further improve RNN-T model architectures by replacing the LSTM layers at the audio encoder and / or prediction network with corresponding layers in the transformer or Conformer layers. Typically, TT and CT model architectures are able to access future audio frames (e.g., the right field) when computing self-attention in their respective transformer or Conformer layers. Therefore, TT and CT model architectures can utilize future right fields to operate in non-streaming transcription mode to improve speech recognition performance when latency constraints are relaxed. That is, there exists a duration of prediction latency proportional to the amount of future audio frames accessed. However, similar to RNN-T, TT and CT model architectures can also operate in streaming transcription mode, where self-attention depends only on past acoustic frames (e.g., the left field).

[0022] Streaming speech recognition models (such as transducer models (e.g., RNN-T, TT, and CT)) optimize sequence likelihood without any latency constraints and therefore suffer from high latency between the audio input and the predicted text, as these models learn to improve their predictions by using more future context. Recent approaches to reduce prediction latency include constrained alignment techniques, which penalize word boundaries by masking alignment paths that exceed a predetermined threshold latency, based on audio alignment information obtained from an external alignment model. While this technique is effective in reducing latency in streaming end-to-end models, it requires a high-precision external alignment model to minimize WER degradation, further complicating the model training process. Other techniques for blindly reducing latency by selecting the most efficient direction in the RNN-T decoding graph often choose directions that are not optimal for all audio inputs due to a lack of alignment information, which can further degrade the latency-WER tradeoff.

[0023] To mitigate the drawbacks associated with using external alignment models or simply blindly reducing latency by selecting the most efficient direction from the decoded graph, this paper's implementation focuses on reducing prediction latency in streaming speech recognition models using self-alignment. Notably, self-alignment does not require any external alignment model or blind latency optimization; instead, it leverages a reference-forced alignment learned from the trained speech recognition model to select the optimal low-latency direction for latency reduction. The reference-forced alignment can include Viterbi-forced alignment. That is, self-alignment will always locate the path one frame to the left of the Viterbi-forced alignment at each time step in the decoded graph. Self-alignment offers advantages over existing schemes for constraining latency. First, the training complexity of self-alignment is significantly lower than teacher-assisted schemes because it does not require an external alignment model. Second, self-alignment minimizes the impact on ASR training by constraining only the most probable alignment paths. Conversely, other schemes influence many alignment paths by masking them or altering the weights on their label transition probabilities. Since the latency constraint regularization term always conflicts with the main ASR loss, minimal intervention in the main loss is crucial for optimizing the latency and performance tradeoffs. Self-alignment regularizes a single path simply by pushing it to the left.

[0024] Figure 1This is an example of a voice environment 100. In voice environment 100, user 104 interacts with a computing device such as user device 10 via voice input. User device 10 (often also referred to as device 10) is configured to capture sound (e.g., streaming audio data) from one or more users 104 within voice environment 100. Here, streaming audio data may refer to spoken utterances 106 of user 104, which serve as audible queries, commands, or audible communications captured by device 10. The voice-enabled system of device 10 can respond to queries or commands by answering queries and / or having commands executed / fulfilled by one or more downstream applications.

[0025] User equipment 10 may correspond to any computing device associated with user 104 and capable of receiving audio data. Some examples of user equipment 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smartwatches), smart appliances, Internet of Things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. User equipment 10 includes data processing hardware 12 and memory hardware 14 that communicates with and stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. User equipment 10 further includes an audio system 16 having audio capture devices (e.g., microphones) 16, 16a for capturing spoken words 106 within the speech environment 100 and converting them into electrical signals, and voice output devices (e.g., speakers) 16, 16b for transmitting audible audio signals (e.g., as output audio data from device 10). Although user equipment 10 implements a single audio capture device 16a in the example shown, user equipment 10 may implement an array of audio capture devices 16a without departing from the scope of this disclosure, whereby one or more capture devices 16a in the array may not physically reside on user equipment 10, but communicate with audio system 16.

[0026] In the speech environment 100, the transducer model 200 of the Automated Speech Recognition (ASR) system 118 resides on the user equipment 10 of user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system executing in a cloud computing environment) that communicates with user equipment 10 via network 40. User equipment 10 and / or remote computing device 60 also include an audio subsystem 108 configured to receive utterance 106 spoken by user 104 and captured by audio capture device 16a, and to convert utterance 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by the ASR system 118. In the example shown, the user speaks the corresponding utterance 106, and the audio subsystem 108 converts utterance 106 into corresponding audio data (e.g., acoustic frame) 110 for input to the ASR system 118. Subsequently, transducer model 200 receives audio data 110 corresponding to utterance 106 as input and generates / predicts a corresponding transcription 120 (e.g., recognition result / hypothesis) of utterance 106 as output. Transducer model 200 provides streaming speech recognition results without access to prospective audio, thus providing streaming transcription capabilities in real time as user 104 is speaking utterance 106. For example, a digital assistant application 50 executing on user device 10 may require speech recognition to be streaming, so that words, chunks of words, and / or individual characters appear on the screen as soon as they are spoken.

[0027] User device 10 and / or remote computing device 60 also execute user interface generator 107, which is configured to present a representation of the transcription 120 of utterance 106 to user 104 of user device 10. As described in more detail below, user interface generator 107 may stream partial speech recognition results 120a during time 1 and subsequently display the final speech recognition result 120b during time 2. In some configurations, the transcription 120 output from ASR system 118 is processed, for example, by a natural language understanding (NLU) module executing on user device 10 or remote computing device 60 to execute a user command / query specified by utterance 106. Additionally or alternatively, a text-to-speech system (not shown), (e.g., executing on any combination of user device 10 or remote computing device 60), may convert the transcription into synthesized speech for audible output by user device 10 and / or another device.

[0028] In the example shown, user 104 interacts with a program or application 50 (e.g., digital assistant application 50) on user device 10 using ASR system 118. For example, Figure 1The text depicts a conversation between user 104 and digital assistant application 50, with digital assistant application 50 displaying digital assistant interface 18 on the screen of user device 10. In this example, user 104 asks digital assistant application 50, “What time is the concert tonight?” This question from user 104 is a spoken word 106 captured by audio capture device 16a and processed by audio system 16 of user device 10. In this example, audio system 16 receives spoken word 106 and converts it into acoustic frames 110 for input to ASR system 118.

[0029] Continuing this example, when an acoustic frame 110 corresponding to utterance 106 spoken by user 104 is received, transducer model 200 encodes the acoustic frame 110 and then decodes the encoded acoustic frame 110 into a partial speech recognition result 120a. During time 1, user interface generator 107 presents a representation of the partial speech recognition result 120a of utterance 106 to user 104 of user device 10 via digital assistant interface 18, such that words, chunks of words, and / or individual characters appear on the screen as soon as they are spoken. In some examples, the first look-ahead audio context is equal to 0.

[0030] During time 2, the user interface generator 107 presents a representation of the final speech recognition result 120b of utterance 106 to the user 104 of the user device 10 via the digital assistant interface 18. Once the user finishes speaking, the final speech recognition result 120b can simply be the partial speech recognition result 120a. Optionally, the ASR system 118 may include another speech recognition to re-evaluate the partial speech recognition result and / or use an external language model. Optionally, once the user finishes speaking, the same transducer model 200 can process the audio again, but instead utilize the right look-ahead audio context to generate the final speech recognition result 120b. This disclosure does not concern how the final speech recognition result 120b is obtained, but focuses on limiting the latency in the streaming partial speech recognition result 120a output by the transducer model 200.

[0031] exist Figure 1In the example shown, digital assistant application 50 can use natural language processing (NLP) to respond to a question posed by user 10. NLP typically refers to the process of interpreting written language (e.g., partial speech recognition result 120a and / or final speech recognition result 120b) and determining whether the written language prompts any action. In this example, digital assistant application 50 uses NLP to recognize that the question from user 10 relates to the user's schedule, and more specifically, to a concert on the user's schedule. By recognizing these details using NLP, the automated assistant returns a response 19 to the user's query, where response 19 states "Venue doors open at 6:30 PM and concert starts at 8 PM". In some configurations, NLP occurs on a remote server 60 communicating with the data processing hardware 12 of user device 10.

[0032] refer to Figure 2 Transducer model 200 can provide end-to-end (E2E) speech recognition by integrating acoustic, articulation, and language models into a single neural network, without requiring a dictionary or separate text normalization component. Various architectures and optimization mechanisms can provide increased accuracy and reduced model training time. In the example shown, transducer model 200 includes a transducer-transducer (TT) model architecture that adheres to latency constraints associated with interactive applications. Compared to conventional ASR architectures, TT model 200 offers a small computational footprint and utilizes fewer memory requirements, making the TT model architecture suitable for performing speech recognition entirely on user device 10 (e.g., without needing to communicate with remote server 60). TT model 200 includes an audio encoder 210, a tag encoder 220, and a joint network 230. The audio encoder 210, which is broadly similar to the acoustic model (AM) in a conventional ASR system, includes a neural network with multiple transducer layers. For example, the audio encoder 210 reads a sequence of d-dimensional feature vectors (e.g., acoustic frames 110 ( Figure 1 x = (x1, x2, ..., x T ),in, Furthermore, a higher-order feature representation 202 is generated at each time step. This higher-order feature representation 202 is denoted as ah1,...,ah T An example converter-transducer model architecture is described in US Application No. 17 / 210,465, filed on March 23, 2021, the contents of which are incorporated herein by reference in their entirety.

[0033] Similarly, the tag encoder 220 may also include a neural network or lookup table embedding model with a transformer layer, which, like the language model (LM), takes the sequence y0,...,y242 of non-whitespace symbols 242 output so far by the final Softmax layer 240. ui-1 The processing involves encoding a dense representation 222 (e.g., represented as Ih) that encodes the predicted label history. u In embodiments of the tag encoder 220 that include a neural network with transformer layers, each transformer layer may include a normalization layer, a masked multi-head attention layer with relative position encoding, residual connections, feedforward layers, and dropout layers. In these embodiments, the tag encoder 220 may include two transformer layers. In embodiments where the tag encoder 220 includes a lookup table embedding model with a binary tag context, the embedding model is configured to learn a d-dimensional weight vector for each possible binary tag context, where d is the dimension of the outputs of the audio encoder 210 and the tag encoder 220. In some examples, the total number of parameters in the embedding model is N. 2 ×d, where N is the vocabulary size for the labels. Here, the learned weight vector is then used as an embedding of the binary label context in the TT model 200 to produce the running time of the fast label encoder 220.

[0034] Finally, utilizing the TT model architecture, the representations generated by the audio encoder 210 and the tag encoder 220 are processed by the joint network 230 using dense layers J. u,t The network is then combined. The joint network 230 then predicts the alignment distribution (e.g., alignment probability 232) on the next output symbol as follows.

[0035] Pr(z u,t |x,t,y1,…,y u-1 (1)

[0036] Where x is the audio input, y is the sequence of ground truth labels, and z is the alignment belonging to y. In other words, the joint network 230 generates a probability distribution 232 over possible speech recognition hypotheses at each output step (e.g., time step). Here, a “possible speech recognition hypothesis” corresponds to a set of output labels (also called “speech units”), each output label representing a morpheme (e.g., symbol / character) or chunk of word in a particular natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, such as a label for each of the 26 letters of the English alphabet, and a label specifying a space. Thus, the joint network 230 may output a set of values ​​indicating the probability of occurrence of each output label in the predetermined set of output labels. This set of values ​​can be a vector (e.g., a one-hot vector) and can indicate a probability distribution over the set of output labels. In some cases, the output labels are morphemes (e.g., a single character and possibly punctuation marks and other symbols), but the set of output labels is not limited to this. For example, in addition to or in place of morphemes, the set of output labels can include chunks of word and / or entire words. The output distribution of the joint network 230 can include the posterior probability value of each output label among the different output labels. Therefore, if there are 100 different output labels representing different characters or other symbols, the output z of the joint network 230 will be... u,t It can include 100 different probability values, with one probability value used for each output label. The probability distribution can then be used to select scores during beam search (e.g., via a Softmax layer 240) and assign scores to candidate orthographic elements (e.g., graphemes, chunks, and / or words) to determine transcription 120.

[0037] The Softmax layer 240 can employ any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol 242 predicted by the TT model 200 at the corresponding output step. Therefore, the set of output symbols 242 predicted by the TT model 200 can also be collectively referred to as the output sequence of label tokens 242. In this way, the TT model 200 does not make a conditional independence assumption; instead, the prediction of each symbol is conditioned not only on acoustics but also on the label sequence output so far.

[0038] To determine the log-conditional probability of y given an audio input x, all alignment distributions corresponding to y are summed as follows.

[0039]

[0040] Here, mapping K removes the whitespace symbols in z. The log-total alignment probability in Equation 2 includes the target loss function, which can be efficiently computed using the following forward-backward algorithm.

[0041] Pr(y|x)=α(T,U) (3)

[0042] α(t,u)=α(t-1,u-1)Pr(φ|t-1,u)+α(t,u-1)Pr(y u |t,u-1) (4)

[0043] Wherein, Pr(φ|t-1,u) and Pr(y u |t,u-1) are the blank probability and the label probability, respectively, and T and U are the audio sequence length and the label sequence length, respectively.

[0044] Although Figure 2 A transducer model 200 including a TT model architecture is described, but without departing from the scope of this disclosure, the transducer model 200 may also include an RNN-T model architecture, a convolutional neural network-transducer (CNN-transducer) model architecture, a convolutional network-transducer (ConvNet-transducer) model, or a Conformer-transducer model architecture. An exemplary CNN-transducer model architecture is described in detail in “Contextnet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context” https: / / arxiv.org / abs / 2005.03191, the contents of which are incorporated herein by reference in their entirety. An exemplary Conformer-transducer model architecture is described in detail in “Convolution-augmented transformer for speech recognition” https: / / arxiv.org / abs / 2005.08100, the contents of which are incorporated herein by reference in their entirety.

[0045] Transducer model 200 is trained on a training dataset corresponding to audio data of spoken utterances paired with corresponding transcriptions. Training transducer model 200 can occur on a remote server 60, and the trained transducer model 200 can be pushed to user device 10. Transducer model 200 is trained using a cross-entropy loss based on Viterbi-forced alignment. Alignment latency includes the delay between the input audio frame and the decoded output label of the streaming. Since the conventional model iteratively trains an alignment model with realigned labels, it is able to learn accurate alignment after multiple iterations. TT or CT models accessing future frames when self-attention is computed at the corresponding transformer or Conformer layer can include alignment latency consistent with the conventional model. However, its self-attention depends only on the transducer model in the streaming mode of past frames having experienced excessive alignment latency.

[0046] The implementation described in this paper addresses reducing prediction latency in a streaming transducer model 200 by using self-alignment. Notably, self-alignment does not require the use of any external alignment model or blind latency optimization; instead, it leverages a reference-forced alignment learned from a trained speech recognition model to select the optimal low-latency direction for latency reduction. The reference-forced alignment may include a Viterbi-forced alignment. That is, self-alignment will always locate the path one frame to the left of the Viterbi-forced alignment at each time step in the decoding graph.

[0047] Figure 3 The output sequence for tag token 242 is shown. Figure 2 The decoding diagram of transducer model 200 of the TT model architecture for the phrase "I like it" is shown in Figure 300. The x-axis represents the corresponding acoustic frame at each time step, while the y-axis represents the output tag token 242. Figure 2 Non-bold solid circle and arrow indicate tokens not included in any alignment path described below. (For example, by...) Figure 3 The bold circles and bold arrows shown indicate that the constrained alignment path 310 includes a word boundary threshold equal to 2. (For example, by...) Figure 3 The forced alignment path 320 (also called the reference forced alignment path 320), indicated by the dotted circles and dotted arrows shown, is learned from the trained transducer model 200, and the left alignment path 330 (e.g., indicated by the dashed circles) comprises one frame to the left of each frame of the forced alignment path 320. During the training of the transducer model 200, for each training batch, self-alignment is encouraged by continuously pushing the model's forced alignment path 320 to the left (e.g., by the dotted circles). Figure 3 The dashed circle and dashed arrow indicate the left alignment path 330. The training loss can be represented as follows.

[0048]

[0049] Where λ is the weighting factor for the left-hand alignment likelihood, and t u It is the frame index of the left alignment at the u-th tag / token.

[0050] Figure 4 An example transformer layer 400 among multiple transformer layers of the audio encoder 210 is shown. Here, during each time step, the initial transformer layer 400 receives the corresponding acoustic frame 110 as input and generates a corresponding output representation / embedding 450 that is received as input by the next transformer layer 400. That is, each transformer layer 400 after the initial transformer layer 400 can receive an input embedding 450 corresponding to the output representation / embedding generated by the immediately preceding transformer layer 400 as output. The final transformer layer 400 (e.g., the last transformer layer in the final stack 320) generates (see reference ) for the corresponding acoustic frame 110 at each of the multiple time steps. Figure 2 , such as by ah t The higher-order feature shown is represented as 202.

[0051] Tag Encoder 220 ( Figure 2 The input can include a vector (e.g., a one-hot vector) indicating the sequence of non-blank symbols y0,...,y0 so far, output by the final Softmax layer 240. ui-1 Therefore, when the tag encoder 220 includes a transformer layer, the initial transformer layer can receive the input embedding 111 by having the one-hot vector pass through a lookup table.

[0052] Each transformer layer 400 of the audio encoder 210 includes a normalization layer 404, a masked multi-head attention layer 406 with relative position encoding, a residual connection 408, a stacking / destacking layer 410, and a feedforward layer 412. The masked multi-head attention layer 406 with relative position encoding provides a flexible way to control the amount (i.e., duration) of the look-forward audio context used by the TT model 200. Specifically, after the normalization layer 404 normalizes the acoustic frame 110 and / or the input embedding 111, the masked multi-head attention layer 406 projects the input onto the values ​​of all heads. Thereafter, the masked multi-head layer 406 can mask the attention scores to the left of the current acoustic frame 110 to produce an output conditioned only on the previous acoustic frame 110. The weighted average of all heads is then concatenated and passed to a dense layer 2 416, where a residual connection 414 is added to the normalized input and output of the dense layer 416 to form the final output of the multi-head attention layer 406 with relative position encoding. The residual connection 408 is added to the output of the normalization layer 404 via adder 430 and is provided as input to either the masked multi-head attention layer 406 or the feedforward layer 412. The stacked / destacking layer 410 can be used to change the frame rate of each transformer layer 400 to accelerate training and inference.

[0053] Feedforward layer 412 applies normalization layer 404, followed by dense layer 1 420, rectified linear layer (ReLU) 418, and dense layer 2 416. ReLU 418 is used as the activation for the output of dense layer 1 420. Similar to multi-head attention layer 406 with relative position encoding, residual connection 414 from the output of normalization layer 404 is added to the output of dense layer 2 416 via adder 430.

[0054] Figure 5 The flowchart includes an example arrangement of operations for a method 500 for training a streaming speech recognition model using self-alignment to reduce prediction latency. At operation 502, the method includes receiving a sequence of acoustic frames 110 corresponding to utterance 106 as input to a streaming speech recognition model (e.g., a transducer model) 200. The streaming speech recognition model 200 is configured to learn alignment probabilities 232 between the sequence of acoustic frames 110 and the output sequence of tag tokens 242. At operation 504, the method 500 includes generating a speech recognition result 120 for utterance 106, including the output sequence of tag tokens 242, as output from the streaming speech recognition model 200 using a decoding graph 300. At operation 506, the method 500 includes generating a speech recognition model loss based on the speech recognition result 120 and the actual transcription of utterance 106.

[0055] At operation 508, method 500 includes obtaining a reference forced alignment path 320 from the decoded graph 300. At operation 510, method 500 includes identifying one frame to the left of each reference forced alignment frame in the reference forced alignment path 320 from the decoded graph 300. At operation 512, method 500 includes summing the label transition probabilities based on the identified frames to the left of each forced alignment frame. At operation 514, method 500 includes updating the streaming speech recognition model 200 based on the sum of the label transition probabilities and the speech recognition model loss.

[0056] Figure 6 This is a schematic diagram of an example computing device 600 that can be used to implement the systems and methods described in this document. The computing device 600 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the embodiments of the invention described and / or claimed in this document.

[0057] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 connected to the memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 connected to a low-speed bus 670 and the storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and may be mounted on a common motherboard or otherwise, depending on the circumstances. Processor 610 is capable of processing instructions for execution within computing device 600, including instructions stored in memory 620 or storage device 630, to display graphical information of a graphical user interface (GUI) on an external input / output device, such as a display 680 coupled to high-speed interface 640. In other embodiments, multiple processors and / or multiple buses may be used with multiple memories and multiple memory types, depending on the circumstances. Furthermore, multiple computing devices 600 can be connected, with each device providing a necessary part of the operation (e.g., as a server group, a set of blade servers, or a multiprocessor system).

[0058] Memory 620 stores information non-temporarily within computing device 600. Memory 620 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-temporarily stored memory 620 may be a physical device for storing programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and optical or magnetic tape.

[0059] Storage device 630 provides mass storage for computing device 600. In some embodiments, storage device 630 is a computer-readable medium. In various embodiments, storage device 630 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory, or other similar solid-state storage device or array of devices, including devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 620, storage device 630, or memory on processor 610.

[0060] High-speed controller 640 manages bandwidth-intensive operations of computing device 600, while low-speed controller 660 manages less bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 640 is coupled to memory 620, display 680 (e.g., via a graphics processor or accelerator), and high-speed expansion port 650, which can accept various expansion cards (not shown). In some embodiments, low-speed controller 660 is coupled to storage device 630 and low-speed expansion port 690. Low-speed expansion port 690, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices (such as switches or routers), for example, via a network adapter.

[0061] As shown in the figure, the computing device 600 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 600a or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack-mounted server system 600c.

[0062] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, which includes at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0063] A software application (i.e., a software resource) can refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0064] Non-transitory memory can be a physical device used to store programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and optical or magnetic tape.

[0065] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0066] The processes and logic described in this specification can be executed by one or more programmable processors (also known as data processing hardware) that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic can also be executed by special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)). By way of example, processors suitable for executing computer programs include both general-purpose and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from read-only memory or random access memory, or both. Essential components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or the computer will be operatively coupled to receive data from or transfer data to such mass storage devices, or both. However, a computer does not necessarily need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory can be supplemented by or incorporated into a dedicated logic circuit system.

[0067] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having: a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen; and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form (including acoustic input, voice input, or tactile input). Additionally, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on the user's client device in response to a request received from a web browser.

[0068] Many embodiments have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A system comprising data processing hardware and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that cause the data processing hardware to execute a streaming speech recognition model, the streaming speech recognition model comprising: An audio encoder, the audio encoder being configured to: It receives a sequence of acoustic frames representing the discourse as input; as well as At each of the multiple time steps, a high-order feature representation of the corresponding acoustic frame in the sequence of the acoustic frames is generated; Tag encoder, the tag encoder being configured to: It receives a sequence of non-whitespace symbols output from the final softmax layer as input; as well as A dense representation is generated at each of the plurality of time steps; and A federated network, which is configured as follows: The high-order feature representation generated by the audio encoder at each of the plurality of time steps and the dense representation generated by the tag encoder at each of the plurality of time steps are received as inputs; as well as At each of the plurality of time steps, a probability distribution is generated based on the possible speech recognition hypotheses at the corresponding time step. The streaming speech recognition model is trained using self-alignment to reduce prediction latency by: Using the decoding graph, the speech recognition result of the utterance is obtained based on the probability distribution of possible speech recognition hypotheses generated by the joint network at each of the plurality of time steps; A reference forced alignment path is obtained from the decoded graph, the reference forced alignment path including reference forced alignment frames; Identify one frame to the left of each reference forced alignment frame in the reference forced alignment path from the decoded graph; Based on the frames to the left of each reference forced alignment frame in the identified reference forced alignment path, sum the label transition probabilities; and The streaming speech recognition model is updated based on the summation of the label transition probabilities.

2. The system of claim 1, wherein, The streaming speech recognition model includes a converter-transducer model.

3. The system according to claim 2, wherein, The audio encoder comprises a stack of transformer layers, each transformer layer including: Normalization layer; A masked multi-head attention layer with relative position encoding; Residual connectivity; Stacking / destacking layers; and Feedforward layer.

4. The system according to claim 3, wherein, The stacking / destacking layers are configured to change the frame rate of the corresponding transformer layer to adjust the processing time of the transformer-transducer model during training and inference.

5. The system according to claim 2, wherein, The tag encoder comprises a stack of transformer layers, each transformer layer including: Normalization layer; A masked multi-head attention layer with relative position encoding; Residual connectivity; Stacking / destacking layers; and Feedforward layer.

6. The system according to claim 1, wherein, The tag encoder includes a binary embedding lookup decoder model.

7. The system according to claim 1, wherein, The streaming speech recognition model includes one of the following: Recursive Neural Transducer (RNN-T) Model; Converter-transducer model; Convolutional Network-Transducer (ConvNet-Transducer) model; or Conformer model.

8. The system according to claim 1, wherein, Using self-alignment to train the streaming speech recognition model to reduce prediction latency includes: using self-alignment without using any external aligner model to constrain the alignment of the decoding graph.

9. The system according to claim 1, wherein, The streaming speech recognition model is executed on a user device or server.

10. The system according to any one of claims 1-9, wherein, Each acoustic frame in the sequence of acoustic frames includes a dimensional feature vector.

11. A computer-implemented method, when executed on data processing hardware, causing the data processing hardware to perform operations for training a streaming speech recognition model using self-alignment to reduce prediction latency, the operations comprising: The streaming speech recognition model receives a sequence of acoustic frames corresponding to the speech as input, and the streaming speech recognition model is configured to learn alignment probabilities between the sequence of acoustic frames and the output sequence of tag tokens. The decoding graph is used to generate a speech recognition result for the utterance as the output from the streaming speech recognition model, the speech recognition result including an output sequence of tag tokens; The speech recognition model loss is generated based on the actual transcription of the utterance and the speech recognition result. The reference forced alignment path, including the reference forced alignment frame, is obtained from the decoded graph; The decoding graph identifies one frame to the left of each reference forced alignment frame in the reference forced alignment path; The label transition probabilities are summed based on the identified frames to the left of each forced alignment frame in the reference forced alignment path; as well as The streaming speech recognition model is updated based on the summation of the label transition probabilities and the speech recognition model loss.

12. The computer-implemented method according to claim 11, wherein, The operation further includes: The audio encoder of the streaming speech recognition model generates a high-order feature representation of the corresponding acoustic frame in the sequence of acoustic frames at each of multiple time steps. The sequence of non-whitespace symbols output by the final softmax layer is received as input to the label encoder of the streaming speech recognition model; The label encoder generates a dense representation at each of the plurality of time steps; The high-order feature representation generated by the audio encoder at each of the plurality of time steps and the dense representation generated by the label encoder at each of the plurality of time steps are received as inputs to the joint network of the streaming speech recognition model; and The joint network generates a probability distribution on possible speech recognition hypotheses at each of the plurality of time steps.

13. The computer-implemented method according to claim 12, wherein, The tag encoder comprises a stack of transformer layers, each transformer layer including: Normalization layer; A masked multi-head attention layer with relative position encoding; Residual connectivity; Stacking / destacking layers; and Feedforward layer.

14. The computer-implemented method according to claim 12, wherein, The tag encoder includes a binary embedding lookup decoder model.

15. The computer-implemented method according to claim 12, wherein, The streaming speech recognition model includes a converter-transducer model.

16. The computer-implemented method according to claim 15, wherein, The audio encoder comprises a stack of transformer layers, each transformer layer including: Normalization layer; A masked multi-head attention layer with relative position encoding; Residual connectivity; Stacking / destacking layers; and Feedforward layer.

17. The computer-implemented method according to claim 16, wherein, The stacking / destacking layers are configured to change the frame rate of the corresponding transformer layer to adjust the processing time of the transformer-transducer model during training and inference.

18. The computer-implemented method according to claim 11, wherein, The streaming speech recognition model includes one of the following: Recursive Neural Transducer (RNN-T) Model; Converter-transducer model; Convolutional Network-Transducer (ConvNet-Transducer) model; or Conformer model.

19. The computer-implemented method according to claim 11, wherein, The streaming speech recognition model is executed on a user device or server.

20. The computer-implemented method according to any one of claims 11-19, wherein, The operation further includes: using self-alignment to train the streaming speech recognition model to reduce prediction latency, without using any external aligner model to constrain the alignment of the decoded graph.

Images