Recurrent neural network-transducer model for performing speech recognition

Abstract

The RNN-T model (200) includes a prediction network (300) configured to receive a sequence of non-blank symbols (301) at each time step after an initial time step. For each non-blank symbol, the prediction network is further configured to generate an embedding (306) for the corresponding non-blank symbol using a shared embedding matrix (304), assign a respective position vector (308) to the non-blank symbol, and weight the embedding in proportion to a similarity between the embedding and the respective position vector. The prediction network is further configured to generate a single embedding vector (350) at the corresponding time step. The RNN-T model further includes a joint network (230) configured to, at each of a plurality of time steps after the initial time step, receive the single embedding vector generated as output from the prediction network at the corresponding time step, and generate a probability distribution over possible speech recognition hypotheses.

Description

Technical Field

[0001] This disclosure relates to bound and reduced recurrent neural network-transducer (RNN-T) models. Background Technology

[0002] Modern Automatic Speech Recognition (ASR) systems focus not only on providing high quality (e.g., low word error rate (WER)) but also on providing low latency (e.g., short delay between the user speaking and the transcription). Furthermore, using ASR systems today requires them to decode speech in a streaming manner that corresponds to or is even faster than real-time. To illustrate, when an ASR system is deployed on a mobile phone experiencing direct user interaction, applications on that phone may require speech recognition to be streaming, so that words appear on the screen as soon as they are spoken. Here, mobile phone users may also have a low tolerance for latency. Due to this low tolerance, speech recognition strives to operate on mobile devices in a way that minimizes the impact of latency and inaccuracy that could adversely affect the user experience. Summary of the Invention

[0003] One aspect of this disclosure provides a recurrent neural network transducer (RNN-T) model, comprising: a prediction network configured to, at each of a plurality of time steps following an initial time step: receive as input a sequence of non-whitespace symbols output by a final Softmax layer. The prediction network is further configured to, for each non-whitespace symbol in the sequence of non-whitespace symbols received as input at the corresponding time step: generate an embedding for the corresponding non-whitespace symbol using a shared embedding matrix; assign a corresponding position vector to the corresponding non-whitespace symbol; and weight the embedding proportionally to the similarity between the embedding and the corresponding position vector. The prediction network is further configured to generate a single embedding vector as output at the corresponding time step, the single embedding vector being a weighted average of the weighted embeddings. The RNN-T model further comprises a joint network configured to, at each of the plurality of time steps following the initial time step: receive as input the single embedding vector generated as an output from the prediction network at the corresponding time step; and generate a probability distribution of possible speech recognition hypotheses at the corresponding time step.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the RNN-T model further includes an audio encoder configured to: receive a sequence of acoustic frames as input; and at each of the plurality of time steps, generate a high-order feature representation of the corresponding acoustic frame in the sequence of acoustic frames. Here, the joint network is further configured to receive, at each of the plurality of time steps, the high-order feature representation generated by the audio encoder at the corresponding time step as input. In some examples, weighting the embedding proportionally to the similarity between the embedding and the corresponding position vector includes: weighting the embedding proportionally to the cosine similarity between the embedding and the corresponding position vector. The sequence of non-whitespace symbols output by the final Softmax layer includes word segments. The sequence of non-whitespace symbols output by the final Softmax layer may include graphemes. Each embedding in the embeddings may include the same dimension as each position vector in the position vectors. In some implementations, the sequence of non-whitespace symbols received as input is limited to the N previous non-whitespace symbols output by the final Softmax layer. In these implementations, N can be equal to 2. Alternatively, N can be equal to 5.

[0005] In some examples, the prediction network includes a multi-head attention mechanism that shares the shared embedding matrix across each head of the multi-head attention mechanism. In these examples, at each of the plurality of time steps following the initial time step, the prediction network is configured to, at each head of the multi-head attention mechanism, and for each non-whitespace symbol in the sequence of non-whitespace symbols received as input at the corresponding time step: using the shared embedding matrix, generate an embedding for the corresponding non-whitespace symbol that is identical to the embedding generated at each of the other heads of the multi-head attention mechanism; assign a corresponding position vector to the corresponding non-whitespace symbol that is different from the corresponding position vector assigned to the corresponding non-whitespace symbol at each of the other heads of the multi-head attention mechanism; and weight the embedding proportionally to the similarity between the embedding and the corresponding position vector. Here, the prediction network is also configured to generate a weighted average of the weighted embeddings of the sequence of non-whitespace symbols as the output of the corresponding head of the multi-head attention mechanism; and to generate the single embedding vector as output at the corresponding time step by averaging the weighted averages of the outputs from the corresponding heads of the multi-head attention mechanism. In these examples, the multi-head attention mechanism may include four heads. Optionally, the prediction network may bind the dimension of the shared embedding matrix to the dimension of the output layer of the joint network.

[0006] Another aspect of this disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations. At each of a plurality of time steps following an initial time step, the operations include: receiving a sequence of non-whitespace symbols output by a final Softmax layer as input to a recurrent neural network-transducer (RNN-T) model; for each non-whitespace symbol in the sequence of non-whitespace symbols received as input at the corresponding time step: generating an embedding for the corresponding non-whitespace symbol by the prediction network using a shared embedding matrix; assigning a corresponding position vector to the corresponding non-whitespace symbol by the prediction network; and weighting the embedding by the prediction network in proportion to the similarity between the embedding and the corresponding position vector. At each of the plurality of time steps following the initial time step, the operations further include: generating a single embedding vector at the corresponding time step as an output from the prediction network; and generating a probability distribution of possible speech recognition hypotheses at the corresponding time step by a joint network of the RNN-T model using the single embedding vector generated as an output from the prediction network at the corresponding time step. The individual embedding vector is based on a weighted average of the weighted embeddings.

[0007] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the operation further includes: receiving a sequence of acoustic frames as input to an audio encoder; generating, by the audio encoder, a high-order feature representation of the corresponding acoustic frame in the sequence of acoustic frames at each of the plurality of time steps; and receiving, at the corresponding time step, the high-order feature representation generated by the audio encoder as input to the joint network. In some examples, weighting the embedding proportionally to the similarity between the embedding and the corresponding position vector includes: weighting the embedding proportionally to the cosine similarity between the embedding and the corresponding position vector.

[0008] The sequence of non-whitespace symbols output by the final Softmax layer may include wordpieces. Optionally, the sequence of non-whitespace symbols output by the final Softmax layer may include character elements. Each embedding in the embeddings may include the same dimension as each position vector in the position vectors. In some implementations, the sequence of non-whitespace symbols received as input is limited to N previous non-whitespace symbols output by the final Softmax layer. In these implementations, N may be equal to 2. Alternatively, N may be equal to 5. Optionally, the prediction network may bind the dimension of the shared embedding matrix to the dimension of the output layer of the joint network.

[0009] In some examples, the prediction network includes a multi-head attention mechanism that shares the shared embedding matrix across each head of the multi-head attention mechanism. The multi-head attention mechanism may include four heads. In these examples, at each of the plurality of time steps following the initial time step, the operation may further include, at each head of the multi-head attention mechanism, and for each non-whitespace symbol in the sequence of non-whitespace symbols received as input at the corresponding time step: the prediction network uses the shared embedding matrix to generate an embedding for the corresponding non-whitespace symbol that is identical to the embedding generated at each of the other heads of the multi-head attention mechanism; the prediction network assigns a corresponding position vector to the corresponding non-whitespace symbol that is different from the corresponding position vector assigned to the corresponding non-whitespace symbol at each of the other heads of the multi-head attention mechanism; and the prediction network weights the embedding proportionally to the similarity between the embedding and the corresponding position vector. Here, at each of the plurality of time steps following the initial time step and at each head of the multi-head attention mechanism, the operation may further include generating a weighted average of the weighted embeddings of the non-whitespace symbol sequence by the prediction network as the output of the corresponding head of the multi-head attention mechanism. Subsequently, at each of the plurality of time steps following the initial time step, the operation may further include generating the single embedding vector at the corresponding time step as the output of the prediction network by averaging the weighted averages of the outputs from the corresponding head of the multi-head attention mechanism.

[0010] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will become apparent from the specification, the drawings, and the claims. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of an example speech environment that uses a recurrent neural network transducer (RNN-T) model to transcribe speech.

[0012] Figure 2 This is a schematic diagram of the example RNN-T model architecture.

[0013] Figure 3 yes Figure 2 A schematic diagram of an example binding and simplified prediction network for the RNN-T model architecture.

[0014] Figure 4 It is a graph depicting the comparison of word error rates between bound and unbound predictions and joint networks.

[0015] Figure 5This is a flowchart illustrating an example layout of operations for a computer-implemented method of performing binding and simplification of the RNN-T model.

[0016] Figure 6 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described in this paper.

[0017] The same reference numerals in the various figures indicate the same elements. Detailed Implementation

[0018] Figure 1 This is an example of a voice environment 100. In voice environment 100, user 104 interacts with a computing device such as user device 10 via voice input. User device 10 (often also referred to as device 10) is configured to capture (e.g., stream audio data) the voice of one or more users 104 within voice environment 100. Here, streamed audio data may refer to spoken utterances 106 of user 104, which are used as audible queries, commands, or audible communications captured by device 10 for device 10. The voice-enabled system of device 10 can process queries or commands by answering queries and / or causing commands to be executed / fulfilled by one or more downstream applications.

[0019] User device 10 may correspond to any computing device associated with user 104 and capable of receiving audio data. Some examples of user device 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smartwatches), smart appliances, Internet of Things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. User device 10 includes data processing hardware 12 and memory hardware 14 that communicates with and stores instructions to the data processing hardware 12, which, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. User device 10 further includes an audio system 16 having audio capture devices (e.g., microphones) 16, 16a for capturing spoken words 106 within the speech environment 100 and converting them into electrical signals, and voice output devices (e.g., speakers) 16, 16b for transmitting audible audio signals (e.g., as output audio data from device 10). Although in the example shown, user equipment 10 implements a single audio capture device 16a, without departing from the scope of this disclosure, user equipment 10 may implement an array of audio capture devices 16a, whereby one or more capture devices 16a in the array may not physically reside on user equipment 10, but communicate with audio system 16.

[0020] In the speech environment 100, an automatic speech recognition (ASR) system 118 implementing a recurrent neural network-transducer (RNN-T) model 200 and an optional re-scoringer 180 resides on a user device 10 of user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system executing in a cloud computing environment) that communicates with user device 10 via network 40. User device 10 and / or remote computing device 60 also include an audio subsystem 108 configured to receive utterance 106 spoken by user 104 and captured by audio capture device 16a, and to convert utterance 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by the ASR system 118. In the example shown, the user speaks the corresponding utterance 106, and the audio subsystem 108 converts utterance 106 into corresponding audio data (e.g., acoustic frame) 110 for input to the ASR system 118. Subsequently, the RNN-T model 200 receives audio data 110 corresponding to utterance 106 as input and generates / predicts a corresponding transcription 120 (e.g., identification result / hypothesis) of utterance 106 as output. In the example shown, the RNN-T model 200 can perform streaming speech recognition to produce initial speech recognition results 120, 120a, and the re-scoring unit 180 can update (i.e., re-score) the initial speech recognition results 120a to produce final speech recognition results 120, 120b.

[0021] User device 10 and / or remote computing device 60 also execute user interface generator 107, which is configured to present a representation of the transcription 120 of utterance 106 to user 104 of user device 10. As described in more detail below, user interface generator 107 may stream an initial speech recognition result 120a during time 1 and subsequently display a final speech recognition result 120b during time 2. In some configurations, the transcription 120 output from ASR system 118 is processed, for example, by a natural language understanding (NLU) module executing on user device 10 or remote computing device 60 to execute a user command / query specified by utterance 106. Additionally or alternatively, a text-to-speech system (not shown), (e.g., executing on any combination of user device 10 or remote computing device 60), may convert the transcription into synthesized speech for audible output by user device 10 and / or another device.

[0022] In the example shown, user 104 interacts with a program or application 50 (e.g., digital assistant application 50) on user device 10 using ASR system 118. For example, Figure 1The illustration depicts a user 104 communicating with a digital assistant application 50, and the digital assistant application 50 displaying a digital assistant interface 18 on the screen of the user device 10 to depict the dialogue between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, “What time is the concert tonight?” This question from the user 104 is a spoken word 106 captured by an audio capture device 16a and processed by the audio system 16 of the user device 10. In this example, the audio system 16 receives the spoken word 106 and converts it into an acoustic frame 110 for input to the ASR system 118.

[0023] Continuing this example, the RNN-T model 200 encodes the acoustic frame 110 upon receiving it, corresponding to the utterance 106 spoken by user 104, and then decodes the encoded acoustic frame 110 into an initial speech recognition result 120a. During time 1, the user interface generator 107 streams a representation of the initial speech recognition result 120a of utterance 106 to user 104 of user device 10 via digital assistant interface 18, such that words, phrases, and / or individual characters appear on the screen as soon as they are spoken. In some examples, the first look-ahead audio context is zero.

[0024] During time 2, the user interface generator 107 presents to the user 104 of the user device 10 a representation of the final speech recognition result 120b of the utterance 106, which has been re-scored by the re-scoring unit 180. In some embodiments, the user interface generator 107 replaces the representation of the initial speech recognition result 120a presented at time 1 with the representation of the final speech recognition result 120b presented at time 2. Here, time 1 and time 2 may include timestamps corresponding to the times when the user interface generator 107 presents the respective speech recognition result 120. In this example, the timestamp of time 1 indicates that the user interface generator 107 presented the initial speech recognition result 120a at a time earlier than the final speech recognition result 120b. For example, since the final speech recognition result 120b is assumed to be more accurate than the initial speech recognition result 120a, the final speech recognition result 120b that is ultimately displayed as transcription 120 may correct any terms that may have been misrecognized in the initial speech recognition result 120a. In this example, the initial speech recognition result 120a, streamed from the RNN-T model 200, is displayed on the screen of user device 10 at time 1, associated with low latency, and provides user 104 with the responsiveness of his / her query being processed. The final speech recognition result 120b, output by the re-scoring unit 180 and displayed on the screen at time 2, fully utilizes additional speech recognition and / or language models to improve speech recognition quality in terms of accuracy, but with increased latency. However, since the initial speech recognition result 120a is displayed when the user speaks 106, the higher latency associated with generating and ultimately displaying the final recognition result is not noticeable to user 104.

[0025] exist Figure 1 In the example shown, digital assistant application 50 can use natural language processing (NLP) to respond to a question posed by user 104. NLP typically refers to the process of interpreting written language (e.g., initial speech recognition result 120a and / or final speech recognition result 120b) and determining whether the written language prompts any action. In this example, digital assistant application 50 uses NLP to recognize that the question from user 104 is related to the user's schedule and, more specifically, to a concert on the user's schedule. By leveraging NLP to identify these details, the automated assistant returns a response 19 to the user's query, where response 19 states, "Venue doors open at 6:30 PM and concert starts at 8 PM." In some configurations, NLP occurs on a remote server 60 communicating with the data processing hardware 12 of user device 10.

[0026] refer to Figure 2Example frame-aligned transducer model 200 includes a recurrent neural network-transducer (RNN-T) model architecture that adheres to latency constraints associated with interactive applications. RNN-T model 200 offers a small computational footprint and utilizes less memory requirements than conventional ASR architectures, making it suitable for performing speech recognition entirely on user device 102 (e.g., without communication with a remote server). RNN-T model 200 includes encoder network 210, prediction network 300, and joint network 230. Prediction network 300 and joint network 230 can collectively provide the RNN-T decoder. Encoder network 210, broadly similar to the acoustic model (AM) in a conventional ASR system, includes a recurrent network with stacked long short-term memory (LSTM) layers. For example, the encoder reads... d 110-dimensional feature vectors (e.g., acoustic frames 110) Figure 1 The sequence x = (x) 1 , x 2 , · , x T ),in, And at each time step, a higher-order feature representation is generated. This higher-order feature representation is represented as... .

[0027] Similarly, prediction network 300 is also an LSTM network, which, like the language model (LM), will be generated from the sequence of non-whitespace symbols 301 so far, output by the final Softmax layer 240. y 0 ,…, y ui-1 Processing as representation As described in more detail below, Pu represents... i 350 includes a single embedding vector. Notably, the sequence of non-whitespace symbols 301 received at prediction network 300 captures the linguistic dependencies among the non-whitespace symbols 301 predicted so far during previous time steps, assisting joint network 230 in predicting the probability of the next output symbol or whitespace symbol during the current time step. As described in more detail below, to aid in techniques for reducing the size of prediction network 300 without sacrificing the accuracy / performance of RNN-T model 200, prediction network 300 may receive a finite historical sequence of non-whitespace symbols. y ui-n , . . . , y ui-1 It is limited to the N previous non-blank symbols 301 output by the final Softmax layer 240.

[0028] Finally, utilizing the RNN-T model architecture, the representations generated by the encoder and prediction networks 210 and 300 are combined by the joint network 230. Then, the joint network predicts... This is the distribution over the next output symbol. In other words, the joint network 230 generates a probability distribution over possible speech recognition hypotheses at each output step (e.g., time step). Here, "possible speech recognition hypotheses" corresponds to a set of output labels, each output label representing a symbol / character in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, such as one label for each of the 26 letters of the English alphabet and one label for spaces. Thus, the joint network 230 can output a set of values ​​indicating the likelihood of occurrence of each label in the predetermined set of output labels. This set of values ​​can be a vector and can indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation marks and other symbols), but the set of output labels is not limited to this. For example, in addition to graphemes or in place of graphemes, the set of output labels can also include word segments and / or entire words. The output distribution of the joint network 230 can include the posterior probability value of each output label in the different output labels. Therefore, if there are 100 different output labels representing different characters or other symbols, then the output y of the joint network 230 will be... i It can include 100 different probability values, one probability value for each output label. The probability distribution can then be used to select candidate orthographic elements (e.g., graphemes, word segments, and / or words) and assign scores to them during a beam search process (e.g., by a Softmax layer 240) to determine transcription 120.

[0029] The Softmax layer 240 can employ any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200 at the corresponding output step. In this way, the RNN-T model 200 does not make a conditional independence assumption; instead, the prediction of each symbol is conditional not only on the acoustics but also on the sequence of labels output so far. The RNN-T model 200 does assume that the output symbol is independent of future acoustic frames 110, which allows the RNN-T model to be used in a streaming manner.

[0030] In some examples, the encoder network 210 of the RNN-T model 200 consists of eight 2048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. In other implementations, the encoder network 210 includes a network of convolutional enhancement transformers or transformer layers. The prediction network 220 may have two 2048-dimensional LSTM layers, each followed by a 640-dimensional projection layer and an embedding layer of 128 units. Finally, the joint network 230 may also have 640 hidden units. The softmax layer 240 may consist of a uniform set of word pieces or glyphs generated using all unique word pieces or glyphs in the training data. When the output symbols / labels include word pieces, the set of output symbols / labels may include 4,096 distinct word pieces. When the output symbols / labels include glyphs, the set of output symbols / labels may include fewer than 100 distinct glyphs.

[0031] Figure 3 The prediction network 300 of the RNN-T model 200 is shown, which receives a sequence of non-blank symbols. y ui-n ,…, y ui-1 As input, the sequence of non-whitespace symbols is restricted to the N previous non-whitespace symbols 301a-n output by the final Softmax layer 240. In some examples, N equals 2. In other examples, N equals 5; however, this disclosure is non-limiting and N can be equal to any integer. The sequence of non-whitespace symbols 301a-n indicates the initial speech recognition result 120a ( Figure 1 In some implementations, the prediction network 300 includes a multi-head attention mechanism 302 that shares a shared embedding matrix 304 across each head 302A-302H of the multi-head attention mechanism. In one example, the multi-head attention mechanism 302 includes four heads. However, the multi-head attention mechanism 302 can employ any number of heads. Notably, the multi-head attention mechanism significantly improves performance with minimal increase in model size. As described in more detail below, each head 302A-H includes its own row of position vectors 308, and instead of increasing the model size by concatenating the outputs 318A-H from all heads, the outputs 318A-H are averaged by the head averaging module 322.

[0032] Referring to the first head 302A of the multi-head attention mechanism 302, head 302A uses a shared embedding matrix 304 for sequences of non-blank symbols received as input at corresponding time steps from multiple time steps. y ui-n ,…, y ui-1Each non-whitespace symbol 301 is used to generate a corresponding embedding 306, 306a-n (e.g., It is worth noting that, since the shared embedding matrix 304 is shared across all heads of the multi-head attention mechanism 302, the other heads 302B-H all generate the same corresponding embedding 306 for each non-whitespace symbol. Head 302A also generates the same embedding 306 for the sequence of non-whitespace symbols. y ui-n ,…, y ui-1 Each corresponding non-blank symbol in the vector is assigned a corresponding position vector PV. Aa–An 308, 308Aa–An (e.g., The corresponding position vector PV 308 assigned to each non-whitespace symbol indicates its position in the history of the sequence of non-whitespace symbols (e.g., the N previous non-whitespace symbols output by the final Softmax layer 240). For example, the first position vector PV Aa It is assigned to the most recent position in history, while the last position vector PV An It is assigned to the last position in the history of the N previous non-whitespace symbols output by the final Softmax layer 240. It is worth noting that each embedding in embedding 306 can include the same dimension (i.e., dimension size) as each position vector in position vector PV 308.

[0033] In the non-blank symbol sequence 301a-n, the shared embedding matrix 304 is used. y ui-n ,…, y ui-1 Each non-whitespace symbol 301 generates a corresponding embedding that is identical across all heads 302A-H of the multi-head attention mechanism 302. Simultaneously, each head 302A-H defines a different set / row of position vectors 308. For example, the first head 302A defines the position vector PV. Aa–An The row 308Aa–An, the second head 302B defines the position vector PV. Ba–Bn 308 Ba–Bn Different rows, ..., and the H-th head 302H defines the position vector PV Ha–Hn 308 Ha–Hn Another line that is not the same.

[0034] For each non-whitespace symbol in the received sequence of non-whitespace symbols 301a-n, the first head 302A is further weighted by a weighting layer 310 with respect to the similarity between the corresponding embedding 306 and the corresponding position vector PV 308 assigned to that embedding. In some examples, the similarity may include cosine similarity (e.g., cosine distance). In the example shown, the weighting layer 310 outputs a sequence of weighted embeddings 312, 312Aa-An, each weighted embedding associated with a corresponding embedding 306 weighted proportionally to the corresponding position vector PV 308 assigned to that weighted embedding. In other words, the weighted embedding 312 output by the weighting layer 310 for each embedding 306 can correspond to the dot product between the embedding 306 and the corresponding position vector PV 308. The weighted embedding 312 can be interpreted as focusing on the embedding with respect to the similarity between the embedding and the position associated with its corresponding position vector PV 308. To improve computational speed, the prediction network 300 includes acyclic layers, and therefore, the sequences of weighted embeddings 312Aa-An are not concatenated, but instead are averaged by the weighted averaging module 316 to generate a weighted average 318A of the weighted embeddings 312Aa-An as the output from the first head 302A. This weighted average 318A of the weighted embeddings 312Aa-An is expressed by the following formula:

[0035] In equation 1, h This indicates the index of the first 302. n Indicates position in the context, and e This represents the embedding dimension. Additionally, in Equation 1, H , N and d e This includes the size of the corresponding dimension. The location vector PV 308 does not need to be trainable and can include random values. Notably, even when the weighted embedding 312 is averaged, the location vector PV 308 can potentially preserve location history information, thereby alleviating the need to provide recurrent connections at each layer of the prediction network 300.

[0036] Each of the other heads 302B-H of the multi-head attention mechanism 302 similarly performs the operations described above with respect to the first head 302A. Due to the different sets of localization vectors PV 308 defined by each head 302, the weighting layer 310 outputs a sequence of weighted embeddings 312Ba–Bn, 312Ha–Hn at each of the other heads 302B-H, which differs from the sequence of weighted embeddings 312Aa–Aa at the first head 302A. Subsequently, the weighted averaging module 316 generates a corresponding weighted average 318B-H of the corresponding weighted embeddings 312 for the sequence of non-blank symbols as the output from each of the other corresponding heads 302B-H.

[0037] In the example shown, the prediction network 300 includes a head averaging module 322 that averages the weighted averages 318A-H from the outputs of corresponding heads 302A-H. A projection layer 326 with SWISH can receive an output 324 from the head averaging module 322 corresponding to the average of the weighted averages 318A-H as input and generate a projected output 328 as output. A final layer normalization 330 can normalize the projected output 328 to provide a single embedding vector Pu at corresponding time steps across multiple time steps. i 350. The prediction network 300 generates only a single embedding vector Pu at each of multiple time steps after the initial time step. i 350.

[0038] In some configurations, the prediction network 300 does not implement the multi-head attention mechanism 302 and only performs the operations described above regarding the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-An is simply passed through the projection layer 326 and layer normalization 330 to provide a single embedding vector Pu. i 350.

[0039] Return to reference Figure 2 The joint network 230 receives a single embedding vector Pu from the prediction network 300. i 350, and receives higher-order feature representations from encoder 210. The joint network 230 generates probability distributions on possible speech recognition hypotheses at the corresponding time steps. Here, the possible speech recognition hypotheses correspond to the set of output labels, each output label representing a symbolic character in a specified natural language. The probability distribution over the possible speech recognition hypotheses is shown below. Indicates the final speech recognition result 120b ( Figure 1 The probability of the final speech recognition result 120b is determined by a single embedding vector 350 based on a sequence of non-blank symbols (e.g., the initial speech recognition result 120a). The final Softmax layer 240 receives the probability distribution of the final speech recognition result 120b and selects the output tag / symbol with the highest probability to generate the transcription.

[0040] The final speech recognition result 120b is assumed to be more accurate than the initial speech recognition result 120a because the RNN-T model 200 determines the initial speech recognition result 120a in a streaming manner and uses the previous non-whitespace symbols from the initial speech recognition result 120a to determine the final speech recognition result 120b. That is, the final speech recognition result 120b takes into account the previous non-whitespace symbols and is therefore assumed to be more accurate because the initial speech recognition result 120a does not take into account any previous non-whitespace symbols. Furthermore, the re-scoring unit 180 ( Figure 1 The initial speech recognition result 120a can be updated using the final speech recognition result 120b to provide the transcription to the user 104 via the user interface generator 170.

[0041] In some implementations, to further reduce the size of the RNN-T decoder—namely, the prediction network 300 and the joint network 230—parameter binding is applied between the prediction network 300 and the joint network 230. Specifically, for vocabulary size... |V| and embedding dimension d e The shared embedding matrix 304 at the prediction network is Meanwhile, the final hidden layer includes a dimension of 230 at the joint network. d h The feedforward projection weights from the hidden layer to the output logits will be This includes additional whitespace tokens in the vocabulary. Therefore, the feedforward layer corresponding to the last layer of the joint network 230 comprises a weight matrix. By enabling the prediction network to embed dimensions 300... d e Size and dimension of the last hidden layer of the joint network 230 d h The shared embedding matrix 304 of the combined feedforward projection weights of network 230 and prediction network 300 can share their weights for all non-whitespace symbols via a simple transpose transformation. Since the two matrices share all their values, the RNN-T decoder only needs to store the values ​​once in memory, instead of storing two individual matrices. This is achieved by increasing the embedding dimension. d e The size is set to be equal to the hidden layer dimension. d h The size of the RNN-T decoder is reduced by an amount equal to the embedding dimension. d e And vocabulary size | V The number of parameters in the product of |. This weight binding corresponds to the regularization technique.

[0042] Figure 4This is a graph 400 showing the word error rate (WER) versus the number of parameters in the RNN-T decoder. Here, Figure 4 Plot 400 illustrates the number of parameters for the WER-contrast bound RNN-T decoder 410 (shown as a solid line), the unbound RNN-T decoder 420 (shown as a dotted line), and the Long Short-Term Memory (LSTM) network 430 (shown as a dashed line). Specifically, plot 400 depicts the sizes of the prediction network 300 and the joint network 230 with and without bound outputs and embeddings. Plot 400 shows the varying embedding dimensions. d e To perform a scan on the model size. For example... Figure 4 As shown, the unbound RNN-T decoder 420 includes four metrics, including embedding dimensions of 64, 320, 640, 960, and 1280. d e Here, the bound RNN-T decoder 410 includes three metrics, including embedding dimensions of 640, 960, and 1280. In the case of the unbound RNN-T decoder 420, the final hidden layer of the joint network 230 always includes a dimension of size 640. d h (Dimensions are not shown in graph 400) d h In the case of the bound RNN-T decoder 410, graph 400 also shows the dimension of the last hidden layer of the joint network 230. d h (Dimensions are not shown in graph 400) d h The embedding dimension of the prediction network is equal to 300. d e The size of the weights makes the size and performance of the RNN-T decoder more sensitive to changes in this dimension. Therefore, the results depicted by plot 400 indicate that weight binding is more parameter-efficient, achieving better performance with fewer parameters. Additionally, for sufficiently large models using weight binding, the same word error rate is achieved as with a regular RNN-T decoder using an LSTM network 430.

[0043] Figure 5 This is a flowchart illustrating an example layout of the operation of a computer implementation of method 500 for performing a bound and simplified RNN-T model 200. At each of several time steps following the initial time step, method 500 performs operations 502-512. At operation 502, method 500 includes receiving a sequence of non-whitespace symbols 301, 301an output by the final Softmax layer 240. y ui-n ,…, yui-1 This is used as input to the prediction network 300 of the recurrent neural network-transducer (RNN-T) model 200. For each non-whitespace symbol in the sequence of non-whitespace symbols received as input during the corresponding time step, method 500 performs operations 504-508. At operation 504, method 500 includes generating an embedding 306 for the corresponding non-whitespace symbol by the prediction network 300 using a shared embedding matrix 304. At operation 506, method 500 includes converting the corresponding position vector PV by the prediction network 300. Aa–An 308, 308Aa–An are assigned to the corresponding non-blank symbols. At operation 508, method 500 includes weighting the embedding 306 proportionally to the similarity between the prediction network 300 and the embedding 306 and the corresponding location vector 308.

[0044] At operation 510, method 500 includes generating a single embedding vector 350 as the output from the prediction network 300 at the corresponding time step. Here, the single embedding vector 350 is a weighted average of the weighted embeddings 312Aa-An and 318A-H. At operation 512, method 500 includes generating a probability distribution of possible speech recognition hypotheses at the corresponding time step using the single embedding vector 350 generated by the joint network 230 of the RNN-T model 200 as the output from the prediction network 300 at the corresponding time step. .

[0045] Figure 6 This is a schematic diagram of an example computing device 600 that can be used to implement the systems and methods described herein. The computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components, their connections and relationships, and their functions shown herein are merely exemplary and are not intended to limit the implementation of the invention described and / or claimed herein.

[0046] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 connected to the memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 connected to a low-speed bus 670 and the storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and may be mounted on a common motherboard or otherwise suitably mounted. The processor 610 is capable of processing instructions for execution within computing device 600, including instructions stored in memory 620 or storage device 630, to display graphical information for a graphical user interface (GUI) on an external input / output device—such as a display 680 coupled to the high-speed interface 640. In other embodiments, multiple processors and / or multiple buses may be used in conjunction with multiple memories and multiple types of memory. Furthermore, multiple computing devices 600 may be connected, with each device providing a portion of the necessary operation (e.g., as a server library, blade server group, or multiprocessor system).

[0047] Memory 620 stores information non-transitory within computing device 600. Memory 620 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-transitory memory 620 may be a physical device for storing programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0048] Storage device 630 provides mass storage for computing device 600. In some embodiments, storage device 630 is a computer-readable medium. In various embodiments, storage device 630 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory, or other similar solid-state storage device or array of devices, including devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 620, storage device 630, or memory on processor 610.

[0049] High-speed controller 640 manages bandwidth-intensive operations for computing device 600, while low-speed controller 660 manages lower bandwidth-intensive operations. This task allocation is merely exemplary. In some embodiments, high-speed controller 640 is coupled to memory 620, display 680 (e.g., via a graphics processor or accelerator), and high-speed expansion port 650, which can accept various expansion cards (not shown). In some embodiments, low-speed controller 660 is coupled to storage device 630 and low-speed expansion port 690. Low-speed expansion port 690, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), can be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices such as switches or routers, for example, via a network adapter.

[0050] The computing device 600 can be implemented in a variety of different forms, as shown in the figure. For example, it can be implemented as a standard server 600a or multiple times in a set of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0051] Various implementations of the systems and techniques described herein can be realized in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can be included in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to a storage system, at least one input device, and at least one output device to receive and transmit data and instructions thereto.

[0052] A software application (i.e., a software resource) can refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0053] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0054] The processes and logic flows described in this specification can be executed by one or more programmable processors—also known as data processing hardware—that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). As an example, processors suitable for executing computer programs include both general-purpose microprocessors and dedicated microprocessors, as well as any one or more processors in any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are the processor that executes instructions and one or more memory devices that store the instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices—such as disks, magneto-optical disks, or optical disks—for storing data, receiving data from them, transferring data to them, or both. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory can be complemented or integrated therein by dedicated logic circuitry systems.

[0055] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having a display device—such as a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touchscreen for displaying information to a user—and optionally a keyboard and indicating device—such as a mouse or trackball—through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents to and from devices used by the user, for example, by sending web pages to a web browser on the user's client device in response to a request received from a web browser.

[0056] Many embodiments have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments fall within the scope of the appended claims.

Claims

1. A recurrent neural network-transducer RNN-T model, comprising: The audio encoder, prediction network, joint network, and finally the Softmax layer; The prediction network is configured to, at each of a plurality of time steps following the initial time step: Receive the sequence of non-whitespace symbols output by the final Softmax layer as input; For each non-whitespace symbol in the sequence of non-whitespace symbols received as input at the corresponding time step: Use the shared embedding matrix to generate the embeddings of the corresponding non-whitespace symbols; Assign the corresponding position vector to the corresponding non-whitespace symbol; and The embedding is weighted proportionally to the similarity between the embedding and the corresponding position vector; as well as At the corresponding time step, a single embedding vector is generated as output, the single embedding vector being a weighted average of the weighted embeddings; and The audio encoder is configured as follows: The sequence of acoustic frames is received as input; as well as At each of the plurality of time steps, a high-order feature representation of the corresponding acoustic frame in the sequence of acoustic frames is generated; The joint network is configured at each of the plurality of time steps following the initial time step: At the corresponding time step, the single embedding vector generated as the output from the prediction network is received as input; The higher-order feature representation generated by the audio encoder at the corresponding time step is received as input; as well as At the corresponding time step, generate the probability distribution of possible speech recognition hypotheses; and The final Softmax layer is configured to select the non-whitespace symbol with the highest probability in the probability distribution of possible speech recognition hypotheses at the corresponding time step, so that the final Softmax layer outputs it as the next non-whitespace symbol in the sequence of non-whitespace symbols.

2. The RNN-T model according to claim 1, wherein, Weighting the embedding in proportion to the similarity between the embedding and the corresponding position vector includes: weighting the embedding in proportion to the cosine similarity between the embedding and the corresponding position vector.

3. The RNN-T model according to claim 1, wherein, The sequence of non-whitespace symbols output by the final Softmax layer includes word segments.

4. The RNN-T model according to claim 1, wherein, The sequence of non-whitespace symbols output by the final Softmax layer includes glyphs.

5. The RNN-T model according to claim 1, wherein, Each embedding in the embeddings includes the same dimension as each position vector in the position vectors.

6. The RNN-T model according to claim 1, wherein, The sequence of non-blank symbols received as input is limited to the N previously non-blank symbols output by the final Softmax layer.

7. The RNN-T model according to claim 6, wherein, N equals 2.

8. The RNN-T model according to claim 6, wherein, N equals 5.

9. The RNN-T model according to claim 1, wherein, The prediction network includes a multi-head attention mechanism that shares the shared embedding matrix across each head of the multi-head attention mechanism.

10. The RNN-T model according to claim 9, wherein, The prediction network is configured to, at each of the plurality of time steps following the initial time step: At each head of the multi-head attention mechanism: For each non-whitespace symbol in the sequence of non-whitespace symbols received as input at the corresponding time step: The shared embedding matrix is ​​used to generate the corresponding non-whitespace symbol embeddings that are identical to those generated at each of the other heads of the multi-head attention mechanism; Assigning a corresponding position vector to the corresponding non-blank symbol that is different from the corresponding position vector assigned to the corresponding non-blank symbol at each of the other heads in the multi-head attention mechanism; and The embedding is weighted proportionally to the similarity between the embedding and the corresponding position vector; and The weighted average of the weighted embeddings of the sequence of non-whitespace symbols is generated as the output of the corresponding head from the multi-head attention mechanism; as well as The single embedding vector is generated as output at the corresponding time step by averaging the corresponding weighted averages of the outputs from the corresponding heads of the multi-head attention mechanism.

11. The RNN-T model according to claim 9, wherein, The multi-head attention mechanism consists of four heads.

12. The RNN-T model according to any one of claims 1 to 11, wherein, The prediction network binds the dimension of the shared embedding matrix to the dimension of the output layer of the joint network.

13. A computer-implemented method, comprising: The sequence of received acoustic frames is used as input to the audio encoder; At each of the multiple time steps following the initial time step: The high-order feature representation of the corresponding acoustic frame in the sequence of acoustic frames generated by the audio encoder; The sequence of non-whitespace symbols output by the final Softmax layer is received as input to the prediction network of the recurrent neural network transducer RNN-T model; For each non-whitespace symbol in the sequence of non-whitespace symbols received as input at the corresponding time step: The prediction network generates the embeddings of the corresponding non-whitespace symbols using a shared embedding matrix; The prediction network assigns the corresponding position vector to the corresponding non-blank symbol. as well as The embedding is weighted proportionally to the similarity between the prediction network and the embedding and the corresponding location vector; At the corresponding time step, a single embedding vector is generated as the output from the prediction network, the single embedding vector being a weighted average of the weighted embeddings; The higher-order feature representation generated by the audio encoder at the corresponding time step is received as input to the joint network of the RNN-T model; The joint network generates a probability distribution of possible speech recognition hypotheses at the corresponding time steps using the single embedding vector generated as the output from the prediction network at the corresponding time steps. as well as The final Softmax layer selects the non-whitespace symbol with the highest probability in the probability distribution of possible speech recognition hypotheses at the corresponding time step, so that the final Softmax layer outputs the next non-whitespace symbol in the sequence of non-whitespace symbols.

14. The computer-implemented method according to claim 13, wherein, Weighting the embedding in proportion to the similarity between the embedding and the corresponding position vector includes: weighting the embedding in proportion to the cosine similarity between the embedding and the corresponding position vector.

15. The computer-implemented method according to claim 13, wherein, The sequence of non-whitespace symbols output by the final Softmax layer includes word segments.

16. The computer-implemented method according to claim 13, wherein, The sequence of non-whitespace symbols output by the final Softmax layer includes glyphs.

17. The computer-implemented method according to claim 13, wherein, Each embedding in the embeddings includes the same dimension as each position vector in the position vectors.

18. The computer-implemented method according to claim 13, wherein, The sequence of non-blank symbols received as input is limited to the N previously non-blank symbols output by the final Softmax layer.

19. The computer-implemented method according to claim 13, wherein, N equals 2.

20. The computer-implemented method according to claim 13, wherein, N equals 5.

21. The computer-implemented method according to claim 13, wherein, The prediction network includes a multi-head attention mechanism that shares the shared embedding matrix across each head of the multi-head attention mechanism.

22. The computer-implemented method of claim 21, further comprising, at each of the plurality of time steps following the initial time step: At each head of the multi-head attention mechanism: For each non-whitespace symbol in the sequence of non-whitespace symbols received as input at the corresponding time step: The prediction network uses the shared embedding matrix to generate embeddings for the corresponding non-whitespace symbols that are identical to those generated at each of the other heads of the multi-head attention mechanism. The prediction network assigns a corresponding position vector to the corresponding non-blank symbol that is different from the corresponding position vector assigned to the corresponding non-blank symbol at each of the other heads of the multi-head attention mechanism. as well as The embedding is weighted proportionally to the similarity between the prediction network and the embedding and the corresponding location vector; as well as The weighted average of the weighted embeddings of the non-blank symbol sequence generated by the prediction network is used as the output of the corresponding head from the multi-head attention mechanism; as well as The single embedding vector is generated at the corresponding time step as the output from the prediction network by averaging the corresponding weighted averages of the outputs from the corresponding heads of the multi-head attention mechanism.

23. The computer-implemented method according to claim 21, wherein, The multi-head attention mechanism consists of four heads.

24. The computer-implemented method according to claim 13, wherein, The prediction network binds the dimension of the shared embedding matrix to the dimension of the output layer of the joint network.

25. An apparatus comprising: An Automatic Speech Recognition (ASR) system, comprising: The recurrent neural network-transducer (RNN-T) model according to any one of claims 1 to 12; At least one processor; and A memory that stores instructions, which, when executed, cause the at least one processor to perform the method according to any one of claims 13 to 24.

26. A computer-readable medium storing instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform the method according to any one of claims 13 to 24.

Images