Fast emitting low latency streaming asr with sequence level emission regularization

Abstract

A computer-implemented method (400) for training a streaming speech recognition model (200) includes receiving a sequence of acoustic frames (122) as input to the streaming speech recognition model. The streaming speech recognition model is configured to learn alignment probabilities (206) between the sequence of acoustic frames and an output sequence of vocabulary tokens (204). The vocabulary tokens include a plurality of label tokens and blank tokens. At each output step, the method includes determining a first probability (264) of transmitting one of the label tokens and determining a second probability (266) of transmitting a blank token. The method also includes generating alignment probabilities at the sequence level based on the first and second probabilities. The method further includes applying a tuning parameter (282) to the alignment probabilities at the sequence level to maximize the first probability of transmitting one of the label tokens.

Description

Technical Field

[0001] This disclosure relates to the use of fast-transmit, low-latency streaming ASR with sequence-level transmit regularization. Background Technology

[0002] Automatic speech recognition (ASR) attempts to provide an accurate transcription of what a person is saying by taking audio input and transcribing it into text. Streaming ASR models aim to transcribe each word in the audio input as quickly and accurately as possible. End-to-end (E2E) recurrent neural network converter (RNN-T) models have gained immense popularity for streaming ASR models. These streaming ASR models learn to make optimal predictions by using future context of the audio input, which leads to a significant delay between the user speaking and the transcription generation. Some methods manipulate the probability of transcription to reduce the amount of delay. However, while manipulating the probability of transcription has provided some success in reducing the latency of streaming ASR models, this success comes at the cost of suffering severe accuracy regression. Summary of the Invention

[0003] One aspect of this disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for training a streaming speech recognition model. The operations include receiving a sequence of acoustic frames as input to the speech recognition model. The streaming speech recognition model is configured to learn alignment probabilities between the sequence of acoustic frames and an output sequence of vocabulary tokens. The vocabulary tokens include a plurality of label tokens and blank tokens. At each of a plurality of output steps, the operations include determining a first probability of transmitting one of the label tokens and determining a second probability of transmitting a blank token. The operations also include generating alignment probabilities at the sequence level based on the first probability of transmitting one of the label tokens and the second probability of transmitting a blank token at each output step. The operations further include applying tuning parameters to the alignment probabilities at the sequence level to maximize the first probability of transmitting one of the label tokens.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, a first probability of firing one of the tag tokens at a corresponding step corresponds to the probability of firing one of the tag tokens after a previous firing of the corresponding tag token. A second probability of firing a blank token at a corresponding step may correspond to the probability of firing a blank tag at a step immediately preceding the corresponding step or firing a blank tag after one of the tag tokens. Optionally, the first and second probabilities may define the forward variables of the forward-backward propagation algorithm.

[0005] In some examples, the operation further includes generating an alignment grid comprising multiple nodes, defined as a matrix with T columns of nodes and U rows of nodes. Here, each column in the T columns corresponds to a corresponding step in a plurality of output steps, and each row in the U rows corresponds to a label representing a sequence of acoustic frames in text. In these examples, at each node position in the alignment grid matrix, the operation may further include: determining a forward probability for predicting subsequent nodes adjacent to the corresponding node, the forward probability including a first probability and a second probability; and determining a backward probability from the subsequent nodes adjacent to the corresponding node to include the corresponding node in the output sequence of vocabulary tokens. Generating alignment probabilities at the sequence level may include aggregating the forward and backward probabilities for all nodes at each corresponding step of the alignment grid.

[0006] In some implementations, when training the streaming speech recognition model, tuning parameters are applied to the alignment probabilities at the sequence level to balance the loss and regularization loss at the streaming speech recognition model. The tuning parameters can be applied independently of any speech-word alignment information. In some examples, the emission of a blank token at one of the output strides is not penalized. Optionally, the streaming speech recognition model may include at least one of the following: a recurrent neural-transducer (RNN-T) model, a transformer-transducer model, a convolutional network-transducer model (ConvNet-Transducer model), or a conformer-transducer model. The streaming speech recognition model may include a recurrent neural-transducer (RNN-T) model. The streaming speech recognition model may include a conformer-transducer model. In some implementations, after training the streaming speech recognition model, the trained streaming speech recognition model is executed on a user device to transcribe speech in a streaming manner. In other implementations, after training the streaming speech recognition model, the trained streaming speech recognition model is executed on a server.

[0007] Another aspect of this disclosure provides a system for training a streaming speech recognition model. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving a sequence of acoustic frames as input to the speech recognition model. The streaming speech recognition model is configured to learn alignment probabilities between the sequence of acoustic frames and an output sequence of vocabulary tokens. The vocabulary tokens include a plurality of label tokens and blank tokens. At each of a plurality of output steps, the operations include determining a first probability of transmitting one of the label tokens and determining a second probability of transmitting a blank token. The operations also include generating alignment probabilities at the sequence level based on the first probability of transmitting one of the label tokens and the second probability of transmitting a blank token at each output step. The operations further include applying tuning parameters to the alignment probabilities at the sequence level to maximize the first probability of transmitting one of the label tokens.

[0008] Implementations of this disclosure may include one or more of the following optional features. In some implementations, a first probability of firing one of the tag tokens at a corresponding step corresponds to the probability of firing one of the tag tokens after a previous firing of the corresponding tag token. A second probability of firing a blank token at a corresponding step may correspond to the probability of firing a blank tag at a step immediately preceding the corresponding step or firing a blank tag after one of the tag tokens. Optionally, the first and second probabilities may define the forward variables of the forward-backward propagation algorithm.

[0009] In some examples, the operation further includes generating an alignment grid comprising multiple nodes, defined as a matrix with T columns of nodes and U rows of nodes. Here, each column in the T columns corresponds to a corresponding step in a plurality of output steps, and each row in the U rows corresponds to a label representing a sequence of acoustic frames in text. In these examples, at each node position in the alignment grid matrix, the operation may further include: determining a forward probability for predicting subsequent nodes adjacent to the corresponding node, the forward probability including a first probability and a second probability; and determining a backward probability from the subsequent nodes adjacent to the corresponding node to include the corresponding node in the output sequence of vocabulary tokens. Generating alignment probabilities at the sequence level may include aggregating the forward and backward probabilities for all nodes at each corresponding step of the alignment grid.

[0010] In some implementations, when training the streaming speech recognition model, tuning parameters are applied to the alignment probabilities at the sequence level to balance the loss and regularization loss at the streaming speech recognition model. The tuning parameters can be applied independently of any speech-word alignment information. In some examples, the emission of a blank token at one of the output strides is not penalized. Optionally, the streaming speech recognition model may include at least one of the following: a recurrent neural transducer (RNN-T) model, a transformer-transducer model, a convolutional network-transducer (ConvNet-Transducer) model, or a conformer-transducer model. The streaming speech recognition model may include a recurrent neural transducer (RNN-T) model. The streaming speech recognition model may include a conformer-transducer model. In some implementations, after training the streaming speech recognition model, the trained streaming speech recognition model is executed on a user device to transcribe speech in a streaming manner. In other implementations, after training the streaming speech recognition model, the trained streaming speech recognition model is executed on a server.

[0011] Details of one or more implementations of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will be apparent from the description and drawings, as well as from the claims. Attached Figure Description

[0012] Figure 1 It is an example system, such as a speech recognition system.

[0013] Figure 2A This is a schematic diagram of a recurrent neural network-transformer (RNN-T) speech recognition model.

[0014] Figure 2B This is a schematic diagram of training a language model for a speech recognition system.

[0015] Figure 3A This is a schematic diagram of an example aligned grid.

[0016] Figure 3B It is certain Figure 3A A schematic diagram of the forward probability of the aligned grid.

[0017] Figure 3C It is certain Figure 3A A schematic diagram of the backward probability of the aligned grid.

[0018] Figure 4 This is a flowchart illustrating an example setup of a method for training a streaming speech recognition model.

[0019] Figure 5 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0020] Similar reference symbols in the various figures indicate similar elements. Detailed Implementation

[0021] Streaming Automatic Speech Recognition (ASR) aims to emit each hypothetical word as quickly and accurately as possible. However, reducing the emission latency of each hypothetical word (i.e., the delay between the user speaking and the occurrence of the text) while maintaining accuracy is challenging. Some methods regularize or penalize emission latency by manipulating the probability prediction per token or per frame in the transducer model. While penalizing emission latency by manipulating the probability per token or per frame has successfully reduced emission latency, these methods suffer significant accuracy regression. To improve the accuracy of streaming speech recognition results, this paper implements a method for training sequence-level streaming speech recognition models. Specifically, the transducer model is trained to reduce emission latency without suffering accuracy regression. Emission latency represents the time interval between the end of the user's speech and the transcription occurrence of the last word spoken by the user.

[0022] Now for reference Figure 1 Example voice environment 100 includes an Automatic Speech Recognition (ASR) system 130 residing on user equipment 102 of user 10 and / or residing on a remote computing device 160 (e.g., one or more servers of a distributed system executing in a cloud computing environment) communicating with user equipment 102 via network 150. Although user equipment 102 is described as a mobile computing device (e.g., a smartphone), user equipment 102 can correspond to any type of computing device, such as, but not limited to, tablet devices, laptop / desktop computers, wearable devices, digital assistant devices, smart speakers / displays, smart appliances, automotive infotainment systems, or Internet of Things (IoT) devices. User equipment 102 includes data processing hardware 104 and memory hardware 106 that communicates with data processing hardware 104 and stores instructions that, when executed by data processing hardware 104, cause data processing hardware 104 to perform one or more operations.

[0023] User equipment 102 also includes an audio system 116 having: audio capture devices (e.g., microphones) 116, 116a for capturing spoken words 12 within the speech environment 100 and converting them into electrical signals; and voice output devices (e.g., speakers) 116, 116b for conveying audible audio signals (e.g., as output audio data from user equipment 102). Although user equipment 102 implements a single audio capture device 116a in the illustrated example, user equipment 102 may implement an array of audio capture devices 116a without departing from the scope of this disclosure, whereby one or more capture devices 116a in the array may not physically reside on user equipment 102 but communicate with audio system 116.

[0024] User equipment 102 includes an audio subsystem 120 configured to receive utterance 12 spoken by user 10 (e.g., captured by one or more microphones 116a) and convert the utterance 12 into a corresponding digital format associated with an input acoustic frame 122 that can be processed by ASR system 130. In the example shown, user 10 utters the corresponding utterance 12 in the English natural language phrase “What song is playing now?”, and audio subsystem 120 converts utterance 12 into a corresponding sequence of acoustic frames 122 for use as input to ASR system 130. ASR system 130 then receives the acoustic frame 122 corresponding to utterance 12 as input and generates / predicts a corresponding transcription (e.g., recognition result / hypothesis) 132 of utterance 12 as output. The time interval between user 10 stopping conversation 124 (e.g., end of speech (EOS) 124) and the last token of transcription (e.g., end of transcription 134) represents transmission delay 136.

[0025] In the example shown, user device 102 and / or remote computing device 160 also execute a user interface generator 140 configured to present a representation of the transcription 132 of utterance 12 to user 10 of user device 102. In some configurations, the transcription 132 output from ASR system 130 is processed, for example, by a natural language understanding (NLU) module executing on user device 102 or remote computing device 160 to execute user commands. Additionally or alternatively, a text-to-speech system (e.g., executing on any combination of user device 102 or remote computing device 160) can convert the transcription 132 into synthesized speech for audible output by another device. For example, the original utterance 12 could correspond to a message that user 10 is sending to a friend, where the transcription 132 is converted into synthesized speech for audible output to the friend to listen to the message expressed in the original utterance 12. Figure 1 As shown in A, example speech environments 100 and 100a use ASR system 130 to generate transcription 132.

[0026] ASR system 130 includes a streaming ASR model 200, which is configured to reduce the emission delay 136 between EOS 124 and the end of transcription 134. Training process 201 ( Figure 2B The ASR model 200 is trained to encourage it to emit features of transcription 132 instead of emitting blanks without penalizing blank emission. In some examples, the ASR model 200 maximizes the probability of emitting character transcriptions at the sequence level rather than at the per-frame or per-token level.

[0027] Figure 2AThe illustration shows an example ASR model 200 including a Recurrent Neural Network-Transformer (RNN-T) model architecture that adheres to latency constraints associated with interactive applications. The RNN-T model 200 offers a small computational footprint and utilizes less memory compared to traditional ASR architectures, making it suitable for performing speech recognition entirely on user device 102 (e.g., without requiring communication with a remote server). Although Figure 2A ASR model 200, including an RNN-T model structure, is described. However, ASR model 200 may also include other types of transducer models without departing from the scope of this disclosure, such as transformer-transducer model structures, convolutional neural network transducer (CNN-Transducer) model structures, convolutional network transducer (ConvNet-Transducer) models, or synthesizer-transducer model architectures. An example transformer-transducer model structure is described in detail below: “TransformerTransducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss”, https: / / arxiv.org / pdf / 2002.02562.pdf, the contents of which are incorporated herein by reference in their entirety. The example CNN-transformer model architecture is described in detail in the following document: “Contextnet: Improving Convolutional Neural Networks for Automatic Speech Recognition with GlobalContext”, https: / / arxiv.org / abs / 2005.03191, the contents of which are incorporated herein by reference in their entirety. The example conformer-transformer model architecture is also described in detail in the following document: “Conformer: Convolution-augmented transformer for speech recognition”, https: / / arxiv.org / abs / 2005.08100, the contents of which are incorporated herein by reference in their entirety.

[0028] Figure 2AThe RNN-T model 200 includes an encoder network 210, a prediction network 220, and a joint network 230. The encoder network 210, roughly analogous to the acoustic model (AM) in a conventional ASR system, comprises a recurrent network of stacked long short-term memory (LSTM) layers. For example, the encoder reads a d-dimensional feature vector (e.g., acoustic frame 122...). Figure 1 The sequence x = (x1, x2, ..., xn) is given by the given sequence x = (x1, x2, ..., xn). T ),in Furthermore, higher-order feature representations are generated at each time step. These higher-order feature representations are recorded as follows:

[0029] Similarly, prediction network 220 is also an LSTM network, which, like a language model (LM), takes the sequence of non-blank tokens y0, ..., y0 from the output of the final Softmax layer 240. ui-1 Process into dense representation Finally, utilizing the RNN-T model structure, the representations generated by the encoder and prediction networks 210 and 220 are combined by the joint network 230. Then, the joint network predicts... This serves as the distribution over the next output token. In other words, the joint network 230 generates a probability distribution over the possible speech recognition hypothesis at each output step (e.g., time step). Here, the "possible speech recognition hypothesis" corresponds to a set of label tokens 204, each label token representing a symbol / character of a specified natural language. For example, when the natural language is English, the set of label tokens may include twenty-seven (27) symbols, such as a label token for each of the 26 letters of the English alphabet and a label for a space. Thus, the joint network 230 can output a set of values ​​indicating the probability of occurrence of each of the predetermined set of label tokens. This set of values ​​may be a vector and may indicate the probability distribution over the set of label tokens. In some cases, the label tokens are morphemes (e.g., a single character, and potential punctuation marks and other symbols), but the set of label tokens is not limited to this. For example, in addition to or instead of morphemes, the set of label tokens may include word pieces and / or entire words. The output distribution of the joint network 230 may include the posterior probability value of each of the different label tokens. Therefore, if there are 100 different tag tokens representing different characters or other symbols, then the output y of the joint network 230 will be... i This can include 100 different probability values, one probability value for each tag token. The probability distribution can then be used in the beam search process used to determine transcription 132 (e.g., via Softmax layer 240) to select scores and assign them to candidate text elements (e.g., graphemes, word slices, and / or words).

[0030] The Softmax layer 240 can employ any technique to select the tag token with the highest probability in the distribution as the next output symbol predicted by the converter model 200 at the corresponding output step. In this way, the RNN-T model 200 does not make a conditional independence assumption; instead, the prediction of each tag token is conditioned not only on the acoustics but also on the sequence of tag tokens emitted so far. The RNN-T model 200 does assume that the tag tokens are independent of future acoustic frames 110, which allows the RNN-T model to be used in a streaming manner.

[0031] In some examples, the encoder network 210 of the RNN-T model 200 consists of eight 2048-dimensional LSTM layers, each followed by a 540-dimensional projection layer. The prediction network 220 can have two 2048-dimensional LSTM layers, each followed by a 540-dimensional projection layer. Finally, the joint network 230 can also have 540 hidden units. The softmax layer 240 can consist of a uniform set of word pieces or glyphs generated using all unique word pieces or glyphs from multiple training datasets.

[0032] Figure 2B The diagram illustrates the training process 201 for training the ASR model 200. Training the converter-based streaming ASR model 200 aims to minimize the transmit delay 136 without suffering accuracy regression. The converter-based streaming ASR model 200 can be interchangeably referred to as "converter model 200". The converter model 200 receives a sequence of acoustic frames 122 from the audio subsystem 120 and is configured to learn the sequence of acoustic frames 122 (e.g., x = (x1, x2, ..., x...)). T The output sequence of the vocabulary token 204 (e.g., y = (y1, y2, ..., y3)) and the output sequence of the vocabulary token 204 (e.g., y = (y1, y2, ..., y4)). U Alignment probability 206 between )). Vocabulary token 204 is the output element of converter model 200, which includes multiple tag tokens 204, 204a ( Figure 3A ) and blank tokens 204, 204b Figure 3ATag token 204a is a textual representation of utterance 12, which may include morphemes (e.g., single characters, and potentially punctuation marks and other symbols), word pieces, and / or entire words. Whitespace token 204b is a textual representation of utterance 12 for whitespace / empty output. Therefore, the output sequence of lexical tokens includes a sequence of tag tokens 204a and whitespace tokens 204b to represent the transcription of utterance 12. In some examples, lexical tokens 204 represent character words, where each tag token 204a represents an alphabetic character (i.e., AZ) and the whitespace token 204b represents a whitespace space. In other examples, lexical tokens 204 represent word piece words, where each tag token 204a represents one or more alphabetic characters, and the whitespace token 204b represents a whitespace space. Lexical tags 204 may also represent punctuation marks and other symbols. Lexical tokens 204 may include any combination of character words, word piece words, and / or punctuation marks and other symbols.

[0033] Alignment probability 206 refers to the probability that converter model 200 emits the corresponding output sequence of vocabulary token 204 from all possible output sequences of vocabulary token 204. To learn alignment probability 206, converter model 200 expands the output sequence using blank token 204b. Training converter model 200 aims to maximize the log probability of the conditional distribution represented by:

[0034]

[0035] In equation (1), This indicates the converter loss, and 'a' represents the alignment grid. This represents the ground truth output sequence of tag token 204a, where x represents the sequence of acoustic frames 122, and... This represents a function that removes the blank token 204b from the alignment grid a. In some implementations, the training process 201 applies a grid generator 250, a forward probability layer 260, a backward probability layer 270, an alignment model 280, and a loss model 290. The grid generator 250 is configured to generate an alignment grid 300 for the sequence of acoustic frames 122 input to the ASR model 200. The grid generator 250 may consist of an encoder 210, a prediction network 220, and a joint network 230 of the converter model 200. The alignment grid 300 includes a matrix of lexical tokens 204 (e.g., label tokens 204a and blank tokens 204b) corresponding to the sequence of acoustic frames 122 for utterance 12. At each output step, the converter model 200 emits one of the label tokens 204a or the blank token 204b from the alignment grid 300.

[0036] Now for reference Figures 3A to 3CThe alignment grid 300 comprises a matrix with multiple nodes 254, 254a-n. Each node 254 of the multiple nodes represents one of the word tokens 204. That is, the alignment grid 300 comprises U rows of nodes 254, each row corresponding to a tag token 204a representing a portion of the sequence of acoustic frames 122 in text. Additionally, the alignment grid 300 comprises T columns of nodes 254, each column corresponding to an output step from multiple output steps. The converter model 200 emits one of the tag tokens 204a or a blank token 204b at each output step. The number of T columns of nodes 254 depends on the amount of output step required to emit all the tag tokens 204a of the corresponding sequence of acoustic frames 122. The grid generator 250 generates the alignment grid 300 based on the sequence of acoustic frames 122 for utterance 12. Now refer to Figure 2B and 3A The alignment grid 300 includes five (5) rows of nodes 254 and seven (7) columns of nodes 254 for the utterance 12 “HELLO”. Each row of the alignment grid 300 has a tag token 204a representing the letter character of the word “HELLO”. Here, the alignment grid 300 includes seven columns because the converter model 200 requires seven (7) output steps to transmit the utterance 12 “HELLO”. The alignment grid 300 may include any number of T columns and U rows required to represent the corresponding sequence of acoustic frames 122.

[0037] Alignment grid 300 provides a matrix of vocabulary tokens 204 to converter model 200 to generate an output sequence of vocabulary tokens 204. Specifically, converter model 200 determines at each node 254 in alignment grid 300 whether to fire one of the tag tokens 204a or a blank token 204b. Therefore, at each output step, converter model 200 either fires one of the tag tokens 204a (e.g., an up arrow) or a blank token 204b (e.g., a right arrow). Converter model 200 continues outputting vocabulary tokens 204 until the last tag token 204a is fired. Once converter model 200 fires the last tag token 204a, the output sequence of vocabulary tokens 204 is complete. Grid generator 250 sends alignment grid 300 and multiple nodes 254 to forward probabilistic layer 260 and backward probabilistic layer 270.

[0038] Forward probability layer 260 and backward probability layer 270 are configured to determine the probability of transmitting one of the tag tokens 204a or a blank token 204b. Forward probability layer 260 determines the probability of transmitting word token 204 based on the probability of each token and / or each frame. That is, forward probability layer 260 determines the probability of transmitting word token 204 based solely on word token 204 or frames. Backward probability layer 270 determines the probability of transmitting word token 204 based on each sequence probability. Therefore, backward probability layer 270 considers previously transmitted word tokens 204 when determining which word token 204 to transmit next. In summary, forward probability layer 260 and backward probability layer 270 determine which word token 204 to transmit based on the probability of each token / each frame and each sequence.

[0039] Specifically, the forward probability layer 260 determines the probability of firing one of the tag tokens 204a or a blank token 204b at subsequent nodes 254, 254S of the alignment grid 300. That is, the forward probability layer 260 determines from the corresponding node 254 of the alignment grid 300 the probability of firing one of the tag tokens 204a or a blank token 204b at subsequent node 254S. Here, subsequent node 254S is adjacent to the corresponding node 254. Subsequent node 254S may be to the right of the corresponding node 254 indicating the firing of a blank token 204b (e.g., node (T+1, U)), or above the corresponding node 254 indicating the firing of one of the tag tokens 204a (e.g., node (T, U+1)).

[0040] The forward probabilistic layer 260 determines the probability of issuing the word token 204 based on the forward probability 262. The forward probability 262 is expressed as:

[0041]

[0042] In Equation 2, α(t, u) represents the forward probability 262. Let t represent the label token 204a, b(t, u) represent the blank token 204b, t represent the column of the corresponding node 254, and u represent the row of the corresponding node 254.

[0043] For example, now refer to Figure 3BThe forward probability layer 260 determines forward probabilities 262 from the corresponding node 254 (e.g., node (T, U)) including a first probability 264 of firing one of the tag tokens 204a and a second probability 266 of firing a blank token 204b. Here, the first probability 264 represents the probability of moving from the corresponding node 254 to the subsequent node 254S (e.g., node (T, U+1)) to fire one of the tag tokens 204a. Specifically, the first probability 264 represents the probability of firing tag token 204a “L” in the next output step. Continuing the example, the second probability 266 represents the probability of moving from the corresponding node 254 to the subsequent node 254S (e.g., node (T+1, U)) to fire a blank token 204b. That is, the second probability 266 represents the probability of firing a blank token in the next output step. In some examples, the second probability 266 of firing a blank token 204b at the corresponding step corresponds to the probability of firing one of the blank tokens 204b at the step immediately preceding the corresponding step or firing a blank token 204b after a tag token.

[0044] The backward probability layer 270 is configured to determine the probability that the output sequence includes the corresponding subsequent node 254S. The backward probability layer 270 determines the probability of the output sequence including the corresponding subsequent node 254S based on the backward probability 272. The backward probability layer 270 determines the backward probability 272, represented by the following, based on all possible output sequences 202 and the proportion of all possible output sequences 202 that include the corresponding subsequent node 254S:

[0045]

[0046] β(t, u) represents the backward probability. Let t represent the tag token 204a, b(t, u) represent the blank token 204b, t represent the column of the corresponding node 254, and u represent the row of the corresponding node 254. Now refer to Figure 3C Starting from subsequent nodes 244S, the backward probability layer 270 determines a backward probability 272 for including the corresponding subsequent node 254S in the output sequence 202. In the example shown, the backward probability layer 270 determines three output sequences 202; however, it should be understood that the backward probability layer 270 can determine any number of output sequences 202. In this example, the first output sequences 202, 202a and the second output sequences 202, 202b include the corresponding subsequent node 254S, while the third output sequences 202, 202c do not include the corresponding subsequent node 254S. The backward probability layer 270 determines the backward probability 272 based on the number of output sequences 202 that include the subsequent nodes 254S from all possible output sequences 202. In this example, the backward probability 272 for the output sequence 202 to include the corresponding subsequent node 254S is one-third.

[0047] In some implementations, the backward probability layer 270 determines backward probabilities 272 for multiple subsequent nodes 254S. That is, from a given node 254, there are two possible subsequent nodes 254S, either representing the tag token 204a or the blank token 204b. In some examples, the backward probability layer 270 determines backward probabilities 272 for both subsequent nodes 254S. In other examples, the backward probability layer 270 determines backward probabilities 272 only for subsequent nodes 254S that satisfy a threshold. That is, when a first probability 264 satisfies the threshold, the backward probability layer 270 determines backward probabilities 272 only for subsequent nodes 254S representing the tag token 204a, and when a second probability 266 satisfies the threshold, the backward probability layer 270 determines backward probabilities 272 only for subsequent nodes 254S representing the blank token 204b.

[0048] In some implementations, node 254 of aligned grid 300 can represent one of the tag tokens 204a and a blank token 204b. Depending on how the output sequence 202 traverses node 254, it will be determined whether node 254 is one of the tag tokens 204a or a blank token 204b. Figure 3C As shown, the two output sequences of lexical tokens 204A and 204B include a subsequent node 254C. The first output sequence of lexical token 204A advances to a subsequent node 254S by emitting a blank token 204B (e.g., a right arrow). In this case, the subsequent node 254S represents a blank token 204b. The second output sequence of lexical token 204B advances to a subsequent node 254S by emitting one of the label tokens 204a (e.g., an up arrow). Here, the subsequent node 254S represents one of the label tokens 204a. Therefore, whether a node 254 in the alignment grid 300 represents one of the label tokens 204a or a blank token 204b depends on the output sequence of the lexical tokens 204. The forward probability layer 260 and the backward probability layer 270 send forward probabilities 262 and backward probabilities 272 to the alignment model 280, respectively. The converter model 200 may emit one or more lexical tokens 204 at the output step. For example, at an output step of T=6 for the third output sequence used for word token 202C, converter model 200 advances through three (3) tag tokens 204a corresponding to the letters “L”, “L”, and “O”. Here, at an output step of T=6, converter model 200 emits all three (3) tag tokens 204a.

[0049] Back Figure 2BThe forward probability layer 260 and the backward probability layer 270 send forward probabilities 262 and backward probabilities 272 to the alignment model 280. The alignment model 280 is configured to determine an alignment probability 206 based on the forward probabilities 262 and backward probabilities 272. That is, the alignment model 280 generates the alignment probability 206 at the sequence level by aggregating the forward probabilities 262 and backward probabilities 272 for all nodes at each corresponding output step of the alignment grid 300. The alignment model 280 determines the alignment probability 206 based on the following equation:

[0050]

[0051]

[0052] In equations 4 and 5, A t，u This represents the coordinates of the corresponding node 254 in the aligned grid 300. Therefore, P(A t，u |x) represents the probability that all complete output sequences of the vocabulary token 204 pass through the corresponding node 254, and Let α(t, u)b(t, u)β(t+1, u) represent the probability of predicting all output sequences of the word token 204 in the aligned grid 300. This represents the probability of predicting one of the tag tokens 204a. Therefore, the alignment model 280 can determine the alignment probability 206 based on each token probability (e.g., forward probability 262) and each sequence probability (e.g., backward probability 272). In some examples, the alignment model 280 sends the alignment probability 206 to the ASR system 130 to determine whether to fire one of the tag tokens 204a or a blank token 204b to generate a transcription 132 corresponding to utterance 19.

[0053] In some implementations, the training process 201 applies a loss model 290, which is configured to determine a converter loss 292 for any node 254 of the alignment grid 300. The alignment model 280 can send alignment probabilities 206 to the loss model 290 to determine the converter loss 292 at each output step. The loss function can be expressed as:

[0054]

[0055]

[0056] Transformer model 200 maximizes the log probability of all possible output sequences of vocabulary token 204, regardless of the emission delay. That is, transformer model 200 equally processes the emission of one of the labeled tokens 204a and the blank token, since the log probability (e.g., Equation 1) is maximized. Therefore, equally processing the emission of labeled tokens 204a and blank tokens 204b inevitably leads to an emission delay 136, as transformer model 200 learns to predict better by using more future context, resulting in a significant emission delay 136. That is, loss model 290 provides transformer loss 292 as feedback to alignment model 280. Alignment model 280 uses transformer loss 292 to minimize the error in the emission of vocabulary tokens 204. Therefore, because the emission of labeled tokens 204a and blank tokens 204b is processed equally, alignment model 280 inevitably introduces an emission delay 136. The implementation in this paper is aimed at training the converter model 200 to encourage the prediction of one of the label tokens 204a to be better than the blank token 204b by maximizing the probability of the label token 204a represented by the following:

[0057]

[0058]

[0059] In equations 8 and 9, This represents the maximum probability of launching one of the tag tokens 204a. Let λ represent the updated converter loss, and λ represent the tuning parameters. Therefore, the first probability 264 and the second probability 266 define the forward variables of the forward-backward propagation algorithm. Specifically, alignment model 280 uses the first probability 264 and the second probability 266 to determine the alignment probability 206 of the output sequence of the emitted vocabulary token 204, and loss model 190 uses the first probability 264 and the second probability 266 to determine the converter loss 292 to train alignment model 280. Alignment model 280 can be incorporated into... Figure 2A In the converter model 200, the joint network 230 and / or the Softmax layer 240.

[0060] The loss function (e.g., Equation 8) used for updating the converter loss applies tuning parameter 282 to maximize the probability of transmitting one of the tag tokens 204a at the sequence level. Tuning parameter 282 is configured to balance converter loss 292 and regularization loss 294. That is, tuning parameter 282 balances the accuracy of transmitting the word tokens 204 (e.g., converter loss 292) and penalizing the transmission delay 136 (e.g., regularization loss 294). Therefore, converter model 200 maximizes the probability of transmitting one of the tag tokens 204a without penalizing the probability of transmitting one of the blank tokens 204b. Tuning parameter 282 can be applied to alignment model 280 independently of any speech-word alignment information. In some implementations, tuning parameter 282 can be implemented based on an existing converter model 200, since the new regularized converter loss can be expressed as:

[0061]

[0062]

[0063] In some examples, after training the transducer model 200 using tuning parameter 282, the trained transducer model 200 is configured to execute on a remote computing device 160 during inference. In other examples, the trained transducer model 200 executes on a user device 102 during inference. Executing the trained transducer model 200 reduces the emission delay 136 of the transcription 132 of the generated utterance 12 by maximizing the probability of emitting one of the sequence-level tag tokens 204a without penalizing the probability of emitting a blank token 204b.

[0064] Figure 4 This is a flowchart illustrating an exemplary arrangement of operations for a method 400 used to train a streaming speech recognition model. In step 402, method 400 includes receiving a sequence of acoustic frames 122 as input to the speech recognition model. The streaming speech recognition model is configured to learn alignment probabilities 206 between the sequence of acoustic frames 122 and the output sequence of vocabulary tokens 204. Vocabulary tokens 204 include a plurality of tag tokens 204a and blank tokens 204b. At each of the plurality of output steps, method 400 includes determining a first probability 264 of transmitting one of the tag tokens 204a in step 404. In step 406, method 400 includes determining a second probability 266 of transmitting a blank token 204b.

[0065] In step 408, method 400 includes generating an alignment probability 206 at the sequence level based on a first probability 264 of firing one of the tag tokens 204a and a second probability 266 of firing a blank token 204b at each output step. At step 410, method 400 includes applying a tuning parameter 282 to the alignment probability 206 at the sequence level to maximize the first probability 264 of firing one of the tag tokens 204a.

[0066] Figure 5 This is a schematic diagram of an example computing device 500 that can be used to implement the systems and methods described in this document. The computing device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components, their connections and relationships, and their functions shown herein are merely illustrative and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0067] Computing device 500 includes: processor 510, memory 520, storage device 530, high-speed interface / controller 540 connected to memory 520 and high-speed expansion port 550, and low-speed interface / controller 560 connected to low-speed bus 570 and storage device 530. Each of components 510, 520, 530, 540, 550, and 560 is interconnected using various buses and can be mounted on a common motherboard or otherwise as required. Processor 510 can process instructions executed within computing device 500, including instructions stored in memory 520 or storage device 530 to display graphical information for a graphical user interface (GUI) on an external input / output device (such as a display 580 coupled to high-speed interface 540). In other implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple types of memory, as needed. Similarly, multiple computing devices 500 can be connected, with each device providing a portion of the necessary operation (e.g., as a server library, a set of blade servers, or a multiprocessor system).

[0068] Memory 520 stores information non-temporarily within computing device 500. Memory 520 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-temporarily stored memory 520 may be a physical device for storing programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0069] Storage device 530 provides massive storage for computing device 500. In some implementations, storage device 530 is a computer-readable medium. In various implementations, storage device 530 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory, or other similar solid-state storage device or array of devices, including devices located in a storage area network or other configuration. In additional implementations, the computer program product is tangibly embodied as an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.

[0070] High-speed controller 540 manages bandwidth-intensive operations for computing equipment 500, while low-speed controller 560 manages less bandwidth-intensive operations. This allocation of responsibility is merely exemplary. In some implementations, high-speed controller 540 is coupled to memory 520, display 580 (e.g., via a graphics processor or accelerator), and high-speed expansion port 550, which can accept various expansion cards (not shown). In some implementations, low-speed controller 560 is coupled to storage device 530 and low-speed expansion port 590. Low-speed expansion port 590, which may include various communication ports (such as USB, Bluetooth, Ethernet, and Wi-Fi), can be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices (such as switches or routers), for example via a network adapter.

[0071] As shown in the figure, the computing device 500 can be implemented in many different forms. For example, it can be implemented as a standard server 500a or multiple times in a group of such servers 500a, as a laptop computer 500b, or as part of a rack-mounted server system 500c.

[0072] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transfer data and instructions to the storage system, at least one input device, and at least one output device.

[0073] A software application (i.e., a software resource) can refer to computer software that instructs a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0074] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0075] The processes and logic flows described herein can be executed by one or more programmable processors (also known as data processing hardware) that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by special-purpose logic circuit systems (such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)). For example, processors suitable for executing computer programs include general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from read-only memory or random access memory, or both. Essential components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (such as disks, magneto-optical disks, or optical disks) for storing data, or the computer will be operatively coupled to receive data from or transfer data to or both of these mass storage devices. However, a computer does not necessarily need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (such as EPROM, EEPROM, and flash memory devices), magnetic disks (such as internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into a dedicated logic circuit system.

[0076] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen) for displaying information to the user; and optionally, a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form (including sound input, voice input, or tactile input). Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0077] Various implementations have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other implementations are within the scope of the following claims.

Claims

1. A computer-implemented method (400), said method, when executed on data processing hardware (104), causing said data processing hardware (104) to perform operations for training a streaming speech recognition model (200), said operations comprising: The sequence of received acoustic frames (122) is used as input to the streaming speech recognition model (200), which is configured to learn alignment probabilities (206) between the sequence of acoustic frames (122) and the output sequence of vocabulary tokens (204), the vocabulary tokens (204) including a plurality of tag tokens (204) and blank tokens (204); At each of the multiple output steps: Determine the first probability (264) of launching one of the tag tokens (204); and Determine the second probability (266) of launching the blank token (204); and The alignment probability (206) is generated at the sequence level based on the first probability (264) of firing one of the tag tokens (204) at each output step and the second probability (266) of firing the blank token (204); as well as The tuning parameter (282) is applied to the alignment probability (206) at the sequence level to maximize the first probability (264) of transmitting one of the tag tokens (204).

2. The computer-implemented method (400) of claim 1, wherein, The first probability (264) of firing one of the tag tokens (204) at the corresponding step size corresponds to the probability of firing one of the tag tokens (204) after the previous firing of the corresponding tag token (204).

3. The computer-implemented method (400) of claim 1, wherein, The second probability (266) of firing the blank token (204) at the corresponding step size corresponds to the probability of firing the blank token (204) at the step size immediately preceding the corresponding step size, after firing either the tag token (204) or the blank token (204).

4. The computer-implemented method (400) of claim 1, wherein, The first probability (264) and the second probability (266) define the forward variables of the forward-backward propagation algorithm.

5. The computer-implemented method (400) of claim 1, wherein, The operation further includes generating an alignment grid (300) comprising a plurality of nodes (254), the alignment grid (300) being defined as a matrix having T columns of nodes (254) and U rows of nodes (254), each of the T columns corresponding to a corresponding step among the plurality of output step sizes, and each of the U rows corresponding to a label representing a sequence of acoustic frames (122) in text.

6. The computer-implemented method (400) of claim 5, wherein, The operation also includes, at each node (254) position in the matrix of the alignment grid (300): Determine a forward probability (262) for predicting subsequent nodes (254) adjacent to the corresponding node (254), the forward probability (262) including the first probability (264) and the second probability (266); and The backward probability (272) of including the corresponding subsequent node (254) in the output sequence of the vocabulary token (204) is determined from the subsequent node (254) adjacent to the corresponding node (254).

7. The computer-implemented method (400) of claim 6, wherein, Generating the alignment probability (206) at the sequence level includes aggregating the forward probability (262) and the backward probability (272) for all nodes (254) at each corresponding step of the alignment grid (300).

8. The computer-implemented method (400) of claim 1, wherein, When training the streaming speech recognition model (200), the tuning parameter (282) is applied to the alignment probability (206) at the sequence level to balance the loss and regularization loss (294) at the streaming speech recognition model (200).

9. The computer-implemented method (400) of claim 1, wherein, The tuning parameters (282) are applied independently of any speech-word alignment information.

10. The computer-implemented method (400) of claim 1, wherein, The issuance of the blank token (204) at one of the output steps is not penalized.

11. The computer-implemented method (400) of claim 1, wherein, The streaming speech recognition model (200) includes at least one of the following: Recurrent Neural Transformer (RNN-T) model; Converter-converter model; Convolutional Network-Transducer (ConvNet-Transducer) model; or Matcher-converter model.

12. The computer-implemented method (400) of claim 1, wherein, After training the streaming speech recognition model (200), the trained streaming speech recognition model (200) is executed on the user device (102) to transcribe speech in a streaming manner.

13. The computer-implemented method (400) of claim 1, wherein, After training the streaming speech recognition model (200), the trained streaming speech recognition model (200) is executed on the server.

14. A system (130) for training a streaming speech recognition model (200), the system comprising: Data processing hardware (104); as well as A memory hardware (106) communicating with the data processing hardware (104), the memory hardware (106) storing instructions that, when executed by the data processing hardware (104), cause the data processing hardware (104) to perform operations, the operations including: The sequence of acoustic frames (122) is received as input to the speech recognition model (200), which is configured to learn alignment probabilities (206) between the sequence of acoustic frames (122) and the output sequence of vocabulary tokens (204), which include a plurality of tag tokens (204) and blank tokens (204). At each of the multiple output steps: Determine the first probability (264) of launching one of the tag tokens (204); and Determine the second probability (266) of launching the blank token (204); and The alignment probability (206) is generated at the sequence level based on the first probability (264) of firing one of the tag tokens (204) at each output step and the second probability (266) of firing the blank token (204); as well as The tuning parameter (282) is applied to the alignment probability (206) at the sequence level to maximize the first probability (264) of transmitting one of the tag tokens (204).

15. The system (130) of claim 14, wherein, The first probability (264) of firing one of the tag tokens (204) at the corresponding step size corresponds to the probability of firing one of the tag tokens (204) after the previous firing of the corresponding tag token.

16. The system (130) of claim 14, wherein, The second probability (266) of firing the blank token (204) at the corresponding step length corresponds to the probability of firing the blank token after firing either the tag token or the blank token at the step length immediately preceding the corresponding step length.

17. The system (130) of claim 14, wherein, The first probability (264) and the second probability (266) define the forward variables of the forward-backward propagation algorithm.

18. The system (130) of claim 14, wherein, The operation further includes generating an alignment grid (300) comprising a plurality of nodes (254), the alignment grid (300) being defined as a matrix having T columns of nodes (254) and U rows of nodes (254), each of the T columns corresponding to a corresponding step among the plurality of output step sizes, and each of the U rows corresponding to a label representing a sequence of acoustic frames (122) in text.

19. The system (130) of claim 18, wherein, The operation also includes, at each node (254) position in the matrix of the alignment grid (300): Determine a forward probability (262) for predicting subsequent nodes (254) adjacent to the corresponding node (254), the forward probability (262) including the first probability (264) and the second probability (266); and The backward probability (272) of including the corresponding subsequent node (254) in the output sequence of the vocabulary token (204) is determined from the subsequent node (254) adjacent to the corresponding node (254).

20. The system (130) of claim 19, wherein, Generating the alignment probability (206) at the sequence level includes aggregating the forward probability (262) and the backward probability (272) for all nodes (254) at each corresponding step of the alignment grid (300).

21. The system (130) of claim 14, wherein, When training the streaming speech recognition model (200), the tuning parameter (282) is applied to the alignment probability (206) at the sequence level to balance the loss and regularization loss (294) at the streaming speech recognition model (200).

22. The system (130) of claim 14, wherein, The tuning parameters (282) are applied independently of any speech-word alignment information.

23. The system (130) of claim 14, wherein, The issuance of the blank token (204) at one of the output steps is not penalized.

24. The system (130) of claim 14, wherein, The streaming speech recognition model (200) includes at least one of the following: Recurrent Neural Transformer (RNN-T) model; Converter-converter model; Convolutional Network-Transducer (ConvNet-Transducer) model; or Matcher-converter model.

25. The system (130) of claim 14, wherein, After training the streaming speech recognition model (200), the trained streaming speech recognition model (200) is executed on the user device (102) to transcribe speech in a streaming manner.

26. The system (130) of claim 14, wherein, After training the streaming speech recognition model (200), the trained streaming speech recognition model (200) is executed on the server.

Images