Fusion of acoustic and textual representations in an automatic speech recognition system implemented as an RNN-T

Abstract

The speech recognition model includes an encoder network (220), a predictive network (230), and a joint network (210). The encoder network is configured to receive a sequence of acoustic frames (222) characterizing an input utterance and generate a high-order feature representation (224) for the corresponding acoustic frame in each of a plurality of output steps. The predictive network is configured to receive a sequence of non-blank symbols (242) output by a final softmax layer (240) and generate a dense representation (232) in each of a plurality of output steps. The joint network (210) is configured to generate a probability distribution (212) over possible speech recognition hypotheses based on the high-order feature representation and the dense representation in each of a plurality of output steps. The joint network includes a stack of gating (260) and bilinear pooling (250) to fuse the dense representation and the high-order feature representation.

Description

[Technical Field]

[0001] The present disclosure relates generally to recurrent neural network transducer (RNN-T) models, and more particularly to improving the fusion of acoustic and textual representations in RNN-T models. [Background technology]

[0002] Modern automated speech recognition (ASR) systems focus on providing not only high quality (e.g., low word error rate (WER)) but also low latency (e.g., short delay between a user's utterance and the appearance of a transcription). Moreover, when using an ASR system today, it is required that the ASR system decode the utterance in a streaming manner that corresponds to real-time or even faster than real-time. By way of example, when an ASR system is deployed on a mobile phone that receives direct user interactivity, an application on the mobile phone that uses the ASR system may require the speech recognition to be streaming, so that words appear on the screen as soon as they are spoken. Here, mobile phone users may also have low tolerance for latency. Because of this low tolerance, speech recognition strives to run on mobile devices in a manner that minimizes the impact of latency and inaccuracies that can adversely affect the user's experience. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present disclosure provides an automated speech recognition (ASR) model including an encoder network, a prediction network, and a joint network. The encoder network is configured to receive, as a first input, a sequence of acoustic frames characterizing an input utterance and generate, at each of a plurality of output steps, a high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The prediction network is configured to receive, as a second input, a sequence of non-blank symbols output by a final softmax layer and generate, at each of the plurality of output steps, a dense representation. The joint network is configured to receive, as a third input, the dense representation generated by the prediction network and the high-order feature representation generated by the audio encoder and generate, at each of the plurality of output steps, a probability distribution over possible speech recognition hypotheses. The joint network includes a combinatorial structure that stacks gating and bilinear pooling to fuse the dense representation generated by the prediction network and the high-order feature representation generated by the audio encoder.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, a regularization method is applied to the predictive network during training by recalculating the dense representation using a scaling factor and a stop gradient function with input tensors that have zero gradients. In some examples, the joint network is not a fully connected layer.

[0005] In some implementations, an audio encoder includes a stack of self-attention blocks. In these implementations, the stack of self-attention blocks may include a stack of conformer blocks or a stack of transformer blocks. In some examples, the stack of conformer blocks includes a stack of 12 encoder blocks with 8-head self-attention.

[0006] In some implementations, the prediction network includes a long short-term memory (LSTM)-based prediction network. Alternatively, the prediction network may include a V2 embedded lookup table. In some examples, the prediction network includes a stateless prediction network.

[0007] Another aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations. The operations include receiving a sequence of acoustic frames characterizing an input utterance. The operations further include, at each of a plurality of time steps, generating, by an audio encoder of a speech recognition model, a high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames, and generating, by a prediction network of the speech recognition model, a dense representation for the corresponding sequence of non-blank symbols output by a final softmax layer of the speech recognition model. The operations at each of the plurality of time steps further include generating, by a joint network of the speech recognition model, a probability distribution over possible speech recognition hypotheses that receives the high-order feature representation generated by the audio encoder and the dense representation generated by the prediction network. The joint network includes a combinatorial structure that stacks gating and bilinear pooling to fuse the dense representation generated by the prediction network and the high-order feature representation generated by the audio encoder.

[0008] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, a regularization method is applied to the predictive network during training by recalculating a dense representation using a scaling factor and a gradient stopping function with input tensors that have zero gradients. In some examples, the joint network does not include a fully connected layer.

[0009] In some implementations, an audio encoder includes a stack of self-attention blocks. In these implementations, the stack of self-attention blocks may include a stack of conformer blocks or a stack of transformer blocks. In some examples, the stack of conformer blocks includes a stack of 12 encoder blocks with 8-head self-attention.

[0010] In some implementations, the prediction network includes a long short-term memory (LSTM)-based prediction network. Alternatively, the prediction network may include a V2 embedded lookup table. In some examples, the prediction network includes a stateless prediction network.

[0011] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of an exemplary audio environment using a recurrent neural network-transducer (RNN-T) model for transcribing audio. [Figure 2] FIG. 1 is a schematic diagram of an exemplary RNN-T for improving the fusion of acoustic and textual representations. [Figure 3] FIG. 2 is a schematic diagram of an exemplary conformer block. [Figure 4]1 is a flowchart of an exemplary configuration of operations for a computer-implemented method for improving the fusion of acoustic and textual representations in an RNN-T. [Figure 5] FIG. 1 is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0013] Like reference symbols in the various drawings indicate like elements.

[0014] The recurrent neural network-transducer (RNN-T) architecture is an end-to-end solution (e.g., a single neural network model) that can be used for, among other uses, streaming automatic speech recognition (ASR) of streaming audio. The RNN-T can be part of a speech recognition model or system. To estimate output distributions across words or subwords, the RNN-T includes a joint network for fusing (i) a high-order feature representation (also commonly referred to as an acoustic representation) generated by an audio encoder with (ii) a dense representation (also commonly referred to as a text representation) generated by a predictive network based on previously decoded text using a recursive structure between previous and current text in the output text sequence. The audio encoder receives as input a sequence of acoustic frames characterizing the input utterance and, at each of multiple output steps, generates a high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The predictive network receives as input a sequence of non-blank symbols output by the final softmax layer of the RNN-T and, at each of multiple output steps, generates a dense representation. The joint network receives as input the dense representations generated by the predictive network and the high-level feature representations generated by the audio encoder at each of the multiple output steps, and generates a probability distribution over the possible speech recognition hypotheses at each of the multiple output steps. An output layer (e.g., a final softmax layer) selects as the output transcription the candidate transcription or hypothesis with the highest likelihood score that accurately represents the sequence of acoustic frames that characterize the input utterance based on the probability distribution.

[0015] More specifically, RNN-T generates a train of acoustic frames x 1:TGiven a sequence of text, ASR is performed by finding the most likely text sequence y. Following Bayes' rule, decoding may follow the maximum a posteriori rule to search through each possible hypothesized text sequence y, for example, using the following formula: P(y|x 1:T )∝p(x 1:T |y)P(y) (1) where p(x 1:T |y) is estimated by the audio encoder, and given y, x 1:T represents the likelihood of being spoken, and P(y) is estimated by a predictive network using a language model (LM) that represents the probability distribution underlying the text. 1:T ) uses a single end-to-end model (e.g., a single neural network). 1:U where U is the number of subword units in y. Then, for streaming audio data without any look-ahead frames or time shortening, the D generated by the audio encoder at time t is enc dimensional high-dimensional feature representation

[0016]

number

[0017] , the D of the u-th subword unit generated by the prediction network pred Dense representation of dimensions

[0018]

number

[0019] , and D generated by the joint network joint Blended representation of dimensions

[0020]

number

[0021] can be expressed as follows:

[0022]

number

[0023] where y0 refers to a special sentence start symbol, and k and W out are the weights of the kth node and the output layer, respectively.

[0024] In some examples, the AcousticEncoder in equation (2) includes a conformal encoder with a fixed number of look-ahead frames and a fixed time reduction rate, the PredictionNetwork in equation (3) includes a multi-layer long short-term memory (LSTM) model, and the JointNetwork in equation (4) includes a fully connected (FC) layer, where:

[0025]

number

[0026] When is neglected in equation (6), the predictive network, joint network, and output layer together form an LSTM language model (LM), sometimes called an internal LM. However, research has shown that ASR accuracy can be improved by fusing acoustic and textual representations.

[0027] Gating has been used as a technique in recursive structures to fuse information. For example, gating has been used in RNN-T to fuse acoustic and textual representations by allowing each element in the representation vector to be scaled with a different weight before being combined, e.g., via vector addition. This allows, for example, the relative fusion of the acoustic and textual representations to be adjusted. More specifically, gating can be used to combine the D generated by the joint network. joint Blended representation of dimensions

[0028]

number

[0029] A more computationally expensive, but even more powerful, technique for fusing information such as acoustic and textual representations is bilinear pooling, which combines representation vectors using a bilinear form that can be expressed as follows:

[0030]

number

[0031] Compared to gating, bilinear pooling first provides a more expressive D enc ×D pred To capture the multiplicative interactions between all possible pairs of elements in the dimensional space, we compute the cross product of the two representation vectors and then denote it as D joint Project onto a dimensional vector space.

[0032] Implementations herein are directed to combining the use of gating and bilinear pooling in an RNN-T joint network to balance and improve the fusion of a high-level feature representation (also commonly referred to as an acoustic representation) encoded from an input audio frame by an audio encoder with a dense representation (also commonly referred to as a text representation). Disclosed herein is a novel structure for an RNN-T joint network that includes gating and bilinear pooling to improve the fusion of the acoustic and text representations. By combining gating with bilinear pooling, the resulting joint network leverages the respective strengths and complementary features of gating and bilinear pooling while fusing the text representation (i.e., the dense representation) generated by the predictive network with the acoustic representation (i.e., the first high-level feature representation) generated by the audio encoder.

[0033] It has been observed that RNN-T prediction networks can converge faster than RNN-T audio encoders because text priors are often easier to learn than acoustic features. This can result in the RNN-T joint network becoming overly reliant on the text representations generated by the prediction network rather than the acoustic representations generated by the audio encoder when performing ASR on training utterances. For example, an RNN-T joint network:

[0034]

number

[0035] In such situations, the audio encoder may be poorly trained to encode audio samples associated with higher prediction network scores. To reduce these training imbalances, a prediction network regularization routine may be applied, for example, at the beginning of training an RNN-T model. Implementations herein are further directed to using the prediction network regularization routine with joint networks having novel combinatorial structures (e.g., see Equation (11) below) that stack gating and bilinear pooling to fuse the dense representation generated by the prediction network with the high-order feature representation generated by the encoder network, or with joint networks configured with other structures (e.g., see Equation (6), Equation (7), Equation (9), or Equation (10)) that can fuse acoustic and textual representations. The exemplary prediction network regularization routine disclosed herein utilizes a joint network to regularize the dense representation generated by the prediction network with the high-order feature representation generated by the encoder network.

[0036]

number

[0037] To optimally balance the fusion of , the gradients backpropagated to the predictive network are reduced during training. For example, during training, the predictive network regularization routine uses a scaling factor and a gradient stopping function with input tensors that have zero gradient to reduce the dense representation.

[0038]

number

[0039] Recalculate.

[0040] 1 is an example of an audio environment 100. In the audio environment 100, a user 104's way of interacting with a computing device, such as a user device 10, may be through voice input. The user device 10 (also commonly referred to as a device 10) is configured to capture sound (e.g., streaming audio data) from one or more users 104 in the audio environment 100. Here, streaming audio data may refer to utterances 106 spoken by the user 104 that function as audible queries, commands to the device 10, or audible communications captured by the device 10. A voice-enabled system of the device 10 may process the queries or commands by answering the queries and / or causing the commands to be executed / fulfilled by one or more downstream applications.

[0041] The user device 10 may correspond to any computing device associated with a user 104 and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, a mobile device (e.g., a mobile phone, a tablet, a laptop, etc.), a computer, a wearable device (e.g., a smart watch), a smart appliance, a vehicle infotainment system, an Internet of Things (IoT) device, a smart display, a smart speaker, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 in communication with the data processing hardware 12. The memory hardware 14 stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 having audio capture devices (e.g., microphones) 16, 16a for capturing and converting spoken utterances 106 in the audio environment 100 into electrical signals, and audio output devices (e.g., speakers) 16, 16b for communicating audible audio signals (e.g., as output audio data from the device 10). Although the user device 10 implements a single audio capture device 16a in the illustrated example, the user device 10 may implement an array of audio capture devices 16a without departing from the scope of the present disclosure, whereby one or more capture devices 16a in the array may not be physically present on the user device 10 but may be in communication with the audio system 16.

[0042] In the speech environment 100, an ASR system 118 implementing an ASR model, such as the RNN-T model 200, and an optional rescorer 180, resides on the user device 10 of the user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system running within a cloud computing environment) that is in communication with the user device 10 via the network 40. The user device 10 and / or the remote computing device 60 also include an audio subsystem 108 configured to receive utterances 106 spoken by the user 104 and captured by the audio capture device 16a, and to convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the ASR system 118. In the illustrated example, the user speaks each utterance 106, and the audio subsystem 108 converts the utterances 106 into corresponding audio data (e.g., acoustic frames) 110 for input to the ASR system 118. The RNN-T model 200 then receives as input audio data 110 corresponding to the utterance 106 and generates / predicts as output a corresponding transcription 120 (e.g., a recognition result / hypothesis) of the utterance 106. In the illustrated example, the RNN-T model 200 may perform streaming speech recognition to produce an initial speech recognition result 120, 120a, and a rescorer 180 may update (e.g., rescore) the initial speech recognition result 120a to produce a final speech recognition result 120, 120b. The server 60 includes data processing hardware 62 and memory hardware 64 in communication with the data processing hardware 62. The memory hardware 64 stores instructions that, when executed by the data processing hardware 62, cause the data processing hardware 62 to perform one or more operations, such as those disclosed herein.

[0043] The user device 10 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of a transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 may display initial speech recognition results 120a in a streaming manner during time 1, and then display final speech recognition results 120b during time 2. In some configurations, the transcription 120 output from the ASR system 118 is processed, for example, by a natural language understanding / processing (NLU / NLP) module running on the user device 10 or the remote computing device 60, to execute user commands or respond to queries specified by the utterance 106. Additionally or alternatively, a text-to-speech system (TTS) (not shown) (e.g., running on any combination of the user device 10 or the remote computing device 60) may convert the transcription 120 into synthesized speech for audible output by the user device 10 and / or another device.

[0044] In the illustrated example, user 104 interacts with a program or application 50 (e.g., digital assistant application 50) on user device 10 that uses ASR system 118. For example, FIG. 1 shows user 104 communicating with digital assistant application 50, and digital assistant application 50 displaying digital assistant interface 18 on the screen of user device 10 to show the conversation between user 104 and digital assistant application 50. In this example, user 104 asks digital assistant application 50, "What time is the concert tonight?" This question from user 104 is spoken utterance 106 that is captured by audio capture device 16a and processed by audio system 16 of user device 10. In this example, audio system 16 receives spoken utterance 106 and converts it into acoustic frames 110 for input to ASR system 118.

[0045] Continuing with this example, as the user 104 speaks, the RNN-T model 200 receives acoustic frames 110 corresponding to the utterance 106, encodes the acoustic frames 110, and then decodes the encoded acoustic frames 110 into initial speech recognition results 120a. During time 1, the user interface generator 107 presents a representation of the initial speech recognition results 120a of the utterance 106 to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming manner, such that words, word pieces, and / or individual characters appear on the screen of the user device 10 as soon as they are spoken. In some examples, the initial look-ahead audio context is set equal to 0.

[0046] During time 2, user interface generator 107 presents a representation of final speech recognition result 120b of utterance 106, as rescored by rescorer 180, to user 104 of user device 10 via digital assistant interface 18. In some implementations, user interface generator 107 replaces the representation of initial speech recognition result 120a presented at time 1 with the representation of final speech recognition result 120b presented at time 2. Here, time 1 and time 2 may include timestamps corresponding to when user interface generator 107 presents each speech recognition result 120. In this example, the timestamp for time 1 indicates that user interface generator 107 presents initial speech recognition result 120a at an earlier time than final speech recognition result 120b. For example, because the final speech recognition result 120b is estimated to be more accurate than the initial speech recognition result 120a, the final speech recognition result 120b, which is ultimately displayed as the transcription 120, may correct any words that may have been misrecognized in the initial speech recognition result 120a. In this example, the streaming initial speech recognition result 120a output by the RNN-T model 200 and displayed on the screen of the user device 10 at Time 1 is associated with low latency, providing the user 104 with a sense of responsiveness that their query is being processed, while the final speech recognition result 120b output by the rescorer 180 and displayed on the screen at Time 2 utilizes additional speech recognition and / or language models to improve speech recognition quality in terms of accuracy but increase latency. However, because the initial speech recognition result 120a is displayed as the user speaks the utterance 106, the higher latency associated with producing and ultimately displaying the final recognition result 120b is not noticeable to the user 104.

[0047] In the example shown in FIG. 1, digital assistant application 50 may use natural language processing (NLP) to respond to questions posed by user 104. NLP generally refers to the process of interpreting written language (e.g., initial speech recognition results 120a and / or final speech recognition results 120b) and determining whether the written language prompts any response or action. In this example, digital assistant application 50 uses NLP to recognize that a question from user 104 pertains to the user's schedule, and more specifically, to a concert on the user's schedule. By recognizing these details using NLP, the automated assistant returns response 19 to the user's question, where response 19 states, "The venue opens at 6:30 PM, and the concert starts at 8 PM." In some configurations, NLP occurs on a remote server 60 in communication with data processing hardware 12 of user device 10.

[0048] 2 is a schematic diagram of an exemplary RNN-T model 200 that fuses a high-level feature representation (also commonly referred to as an acoustic representation) 224 output by an audio encoder network 220 with a dense representation (also commonly referred to as a text representation) 232 output by a prediction network 230. In particular, the RNN-T model 200 includes a novel joint network 210 that combines gating with bilinear pooling to improve the fusion of the acoustic representation 224 and the text representation 232. By combining gating with bilinear pooling, the joint network 210 leverages the respective strengths and complementary characteristics of gating and bilinear pooling.

[0049] As shown, the RNN-T model 200 includes an encoder network 220, a predictor / decoder network 230, a joint network 210, and a final softmax output layer 240. The encoder network 220 (e.g., an audio encoder) is roughly analogous to an acoustic model (AM) in a conventional ASR system, and generates a feature vector (e.g., the acoustic frame 110 in FIG. 1) x=(x1, x2,..., x t )222 sequence, where

[0050]

number

[0051] , and produces a high-level feature representation (e.g., an acoustic representation) 224 denoted as:

[0052] In the illustrated example, the predictor / decoder network 230, like a language model (LM), calculates the non-blank symbols y output so far by the softmax layer 240. 0, ...,y u-1 242 sequences, dense representation

[0053]

number

[0054] 232, where y0 represents a special sequence start symbol.

[0055] The joint network 210 is connected to the representations produced by the encoder network 220 and the prediction network 230, respectively.

[0056]

number

[0057] In other words, at each output step (e.g., time step), the joint network 210 generates a probability distribution 212 over possible speech recognition hypotheses. Here, a “possible speech recognition hypothesis” corresponds to a set of output labels, each representing a word / word piece / symbol / character in a specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters in the English alphabet, and one label specifying a space. Thus, the joint network 210 may output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels. This set of values ​​may be a vector and may indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels may include word pieces and / or entire words in addition to or instead of graphemes. The output distribution of the joint network 210 may include a posterior probability value for each of the different output labels. Thus, when there are 100 different output labels representing different graphemes or other symbols, the output y of the joint network 210 i may contain 100 different probability values, one for each output label. The probability distributions may then be used to select and assign scores to candidate orthogonal elements (e.g., graphemes, wordpieces, and / or words) in a beam search process (e.g., by the final softmax output layer 240) to determine the transcription 120.

[0058] The joint network 210 generates a high-level feature representation (HFE) of the speech signal received by the joint network 210 at each of a plurality of output steps when performing speech recognition on the utterance 106 (FIG. 1).

[0059]

number

[0060] In the illustrated example, the joint network 210 includes a bilinear pooling layer 250 and a gating layer 260. In some examples, the bilinear pooling layer 250 is stacked on top of the gating layer 260. In these examples, the stacking of the bilinear pooling layer 250 and the gating layer 260 can be mathematically expressed as follows:

[0061]

number

[0062] The softmax layer 240 calculates the distribution

[0063]

number

[0064] Any technique may be employed to select the output label / symbol with the highest probability in 212 as the next output symbol predicted by the RNN-T model 200 in the corresponding output step. In this way, the RNN-T model 200 does not make a conditional independence assumption. Instead, the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model 200 assumes that the output symbol is independent of future acoustic frames 110, thereby allowing the RNN-T model 200 to be employed in a streaming manner. In some examples, the softmax layer 240 is composed of a unified wordpiece or grapheme set generated using all unique wordpieces or graphemes in multiple training datasets.

[0065] In some examples, feature vector x222 includes 80-dimensional log-mel filter bank features formed by stacking three 32-ms acoustic frames with a 10-ms shift to form a 240-dimensional input representation with a 30-millisecond (ms) frame rate, which are then converted to a 512-dimensional representation with added positional embedding using a first linear projection. Continuing with this example, encoder network 220 may include 12 conformer encoder blocks with 8-head self-attention and a convolution kernel size of 15 to further transform the stacked features. Here, encoder network 220 performs a concatenation operation after the third conformer block to achieve a time reduction factor of 2. A fourth conformer block transforms the resulting 1024-dimensional vectors, and then encoder network 220 projects them back to 512 dimensions using a second linear transformation. The remaining eight conformer blocks are used to generate higher-dimensional feature representations.

[0066]

number

[0067] Dimension D for 224 enc = 512, followed by a second linear transformation, followed by a final linear normalization layer. While the illustrated encoder network 220 has a stack of multi-head attention layers / blocks including conformer layers / blocks (e.g., 12 conformer blocks), the disclosure is not so limited. For example, the encoder network 220 may include a stack of transformer layers / blocks or other types of multi-head attention layers / blocks. The encoder network 220 may include a series of multi-head self-attention layers, depth-wise convolutional layers, and feedforward layers. Alternatively, the encoder network 220 may include multiple long short-term memory (LSTM) layers instead of multi-head attention layers / blocks.

[0068] Continuing with this example, the prediction network 230 may be

[0069]

number

[0070] D for 232 pred This is an LSTM-based network that contains two layers of 2,048-dimensional LSTMs along with a 640-dimensional linear projection to produce a fused representation.

[0071]

number

[0072] Dimension D of 212 joint is also set to 640. In some examples, the joint network 210 includes hidden units. Additionally or alternatively, the joint network 210 does not include a fully connected (FC) layer.

[0073] Alternatively, the encoder network 220 includes a stack of self-attention layers / blocks, where the stack of self-attention blocks may include a stack of transformer blocks or a different stack of conformer blocks.

[0074] Alternatively, the prediction network 230 may include a stack of transformer or conformer blocks (or other types of multi-head attention blocks). The prediction network 230 may also be replaced with an embedding lookup table (e.g., a V2 embedding lookup table) to improve latency by outputting a looked-up sparse embedding instead of generating a dense representation. In some implementations, the prediction network 230 is a stateless prediction network.

[0075] As explained above, the prediction network 230 may converge faster than the encoder network 220 during training, thereby allowing the joint network 210 to obtain the high-level feature representations generated by the encoder network 220 when performing ASR on training utterances.

[0076]

number

[0077] To reduce such training imbalance, a predictive network regularization routine may be applied, for example, at the beginning of training of the RNN-T model 200. More specifically, training of the RNN-T model may include using the predictive network regularization routine with a joint network having a novel combinatorial structure (e.g., see Equation (11)) that stacks gating and bilinear pooling to balance the fusion of the dense representation 232 generated by the predictive network 230 and the high-level feature representation 224 generated by the encoder network 220, or with a joint network configured with other structures (e.g., see Equation (6), Equation (7), Equation (9), or Equation (10)) that can fuse acoustic and textual representations. In some examples, the predictive network regularization routine may be applied to the joint network 210 to balance the fusion of the dense representation 232 generated by the predictive network 230 and the high-level feature representation 224 generated by the encoder network 220.

[0078]

number

[0079] The gradients backpropagated to the prediction network 230 during training are reduced to optimally balance the fusion of . For example, during training, applying a prediction network regularization routine may be used to reduce the dense representation using a scaling factor and a gradient stopping function with input tensors that have zero gradient.

[0080]

number

[0081] where m is the index in the current training step and α m is a scaling factor, and sg() is the gradient stopping function such that its input tensor has zero gradient. In this example, 0 ≤ α m When ≦1,

[0082]

number

[0083] The value of α is not changed, but the corresponding gradient that is back-propagated to the prediction network 230 is m This slows down the convergence of the prediction network 230 and reduces the likelihood of the joint network 210 being affected during training.

[0084]

number

[0085] In some examples, the predictive network regularization routine uses a piece-wise linear schedule to regularize the α m Select a value for

[0086]

number

[0087] where m1 and m2 are two predefined parameters. In particular, applying a prediction network regularization routine differs from initializing the RNN-T model 200 with a pre-trained connectionist temporal classification (CTC) model, even when m=0, because the prediction network 230 provides random but fixed-valued projections through which the RNN-T model 200 still predicts y u-1 Compared to other conventional training techniques, training the joint network 210 with a predictive network regularization routine improves the integration of the internal LM during both training and test time by initially discounting the internal LM during training. In particular, the joint network 210 and / or the predictive network regularization routine are applicable to stateless RNN-T models in which the LM history embedded in the predictive network 230 is limited and / or reset for each utterance.

[0088] FIG. 3 is a schematic diagram of an exemplary conformer block 300 that may be used to implement one of the conformer blocks in the stack of conformer blocks of the encoder network 220 of FIG. 2. The conformer block 300 includes a first-half feedforward layer 310, a second-half feedforward layer 340, a multi-head self-attention block 320 and a convolutional layer 330 disposed between the first-half feedforward layer 310 and the second-half feedforward layer 340, and a concatenation operator 305. The first-half feedforward layer 310 processes input audio data 102, including an input mel spectrogram sequence. The multi-head self-attention block 320 then receives the input audio data 102 concatenated with the output of the first-half feedforward layer 310. Intuitively, the role of the multi-head self-attention block 320 is to summarize the noise context separately for each input frame to be enhanced. The convolutional layer 330 subsamples the output of the multi-head self-attention block 320 concatenated with the output of the first-half feedforward layer 310. A late feedforward layer 340 then receives the concatenation of the convolutional layer 330 output and the multi-head self-attention block 320. A layernorm module 350 processes the output from the late feedforward layer 340. The conformer block 300 uses the modulation feature m to transform the input feature x to produce the output feature y 360, which can be expressed mathematically, for example, as follows:

[0089]

number

[0090] 4 is a flowchart of an exemplary configuration of operations for a computer-implemented method 400 for improving the fusion of acoustic and textual representations in an RNN-T model, such as RNN-T model 200. Data processing hardware 510 (e.g., data processing hardware 12 of device 10 of FIG. 1 and / or data processing hardware 62 of computing system 60) may perform the operations for method 400 by executing instructions stored on memory hardware 520 (e.g., memory hardware 14, 64).

[0091] At operation 402, the method 400 selects acoustic frames x=(x1, x2,..., x) that characterize the input utterance 106. t ) 222. The method 400 performs operations 404, 406, and 408 in each of a plurality of output steps. In operation 404, the method 400 generates, by the encoder network 220 of the RNN-T model 200, a high-dimensional feature representation for a corresponding acoustic frame 222 in the sequence of acoustic frames 222.

[0092]

number

[0093] 224.

[0094] At operation 406, the method 400 calculates, by the prediction network 230 of the RNN-T model 200, the non-blank symbols (y 0 ,...,y u-1 )242 dense representations for the corresponding sequences

[0095]

number

[0096] 232, where y0 may represent a special sequence start symbol.

[0097] In operation 408, the method 400 generates a high-dimensional feature representation

[0098]

number

[0099] For example, the joint network 210 may use a bilinear pooling layer 250 stacked on a gating layer 260 to generate a probability distribution, as described above with respect to the joint network 210 of FIG.

[0100]

number

[0101] 212. For example, in operation 408, the method 400 may generate the probability distribution σ as an output from the joint network 210 using equation (11).

[0102]

number

[0103] 212 can be calculated.

[0104] 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems and methods described herein. Computing device 500 is representative of various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computing devices. The components shown, their connections and relationships, and their functionality are merely exemplary and do not limit the implementation of the invention described and / or claimed herein.

[0105] Computing device 500 includes a processor 510 (i.e., data processing hardware) that may be used to implement data processing hardware 12 and / or 62, a memory 520 (i.e., memory hardware) that may be used to implement memory hardware 14 and / or 64, a storage device 530 (i.e., memory hardware) that may be used to implement memory hardware 14 and / or 64, a high-speed interface / controller 540 that connects to memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 that connects to a low-speed bus 570 and storage device 530. Each of components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be mounted on a common motherboard or in other manners as appropriate. The processor 510 can process instructions for execution within the computing device 500, including instructions stored in memory 520 or on storage device 530, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 coupled to a high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memory, as appropriate. Multiple computing devices 500 may also be connected, each providing a portion of the required operations (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system).

[0106] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disk or tape.

[0107] The storage device 530 is capable of providing mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 520, the storage device 530, or memory on the processor 510.

[0108] The high-speed controller 540 manages bandwidth-intensive operations for the computing device 500, while the low-speed controller 560 manages lower-bandwidth-intensive operations. Such allocation of duties is merely exemplary. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 550, which may accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and a low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be coupled to one or more input / output devices such as a keyboard, pointing device, scanner, etc., or to a networking device such as a switch or router, for example, through a network adapter.

[0109] The computing device 500, as shown, may be implemented in several different forms. For example, the computing device 500 may be implemented as a standard server 500a, or multiple times within a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0110] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, and which may be coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0111] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0112] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0113] The processes and logic flows described herein may be performed by one or more programmable processors, also referred to as data processing hardware, that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or be operatively coupled to receive data from or transfer data to, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0114] To provide for user interaction, one or more aspects of the present disclosure may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices may similarly be used to provide for user interaction; for example, feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, voice input, or tactile input. Furthermore, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to the web browser in response to a request received from the web browser on the user's client device.

[0115] Unless otherwise specified, "or" refers to an inclusive or, not an exclusive or. For example, "A, B, or C" refers to any combination or subset of A, B, or C, such as (1) A only, (2) B only, (3) C only, (4) A with B, (5) A with C, (6) B with C, and (7) A with B and C. Similarly, the phrase "at least one of A or B" refers to any combination or subset of A and B, such as (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Moreover, the phrase "at least one of A and B" refers to any combination or subset of A and B, such as (1) at least one A, (2) at least one B, and (3) at least one A and at least one B.

[0116] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0117] 10 User Devices, Devices 12, 62 Data Processing Hardware 14, 64 memory hardware 16 Audio systems, audio capture devices, and audio output devices 16a Audio Capture Device, Audio Capture Device, Capture Device 16b Audio Output Device 18 Digital Assistant Interface 19 responses 40 Network 50 programs or applications, digital assistant applications 60 Remote Computing Device, Server, Remote Server, Computing System 100 Audio Environment 102 input audio data 104 users 106 Spoken utterances, speech, input utterances 107 User Interface Generator 108 Audio Subsystem 110 Input Acoustic Frame, Audio Data, Acoustic Frame 118 ASR System 120 Corresponding transcription, initial speech recognition result, final speech recognition result, transcription, speech recognition result 120a Initial speech recognition results, streaming initial speech recognition results 120b Final speech recognition result, final recognition result 180 Re-scorer 200 RNN-T model, automated speech recognition (ASR) model, ASR model, speech recognition model 210 Joint Network 212 Probability distributions over possible speech recognition hypotheses, distributions, fused representations, probability distributions 220 Audio Encoder Network, Encoder Network 222 Feature vector x=(x1,x2,...,x t ), feature vector x, acoustic frame x=(x1,x2,...,x t ), acoustic frame 224 High-level feature representation, acoustic representation, 230 Prediction Network, Prediction / Decoder Network 232 Dense Representation, Text Representation 240 Final softmax output layer, softmax layer, final softmax layer 242 non-blank symbol y 0, ...,y u-1 , a non-blank symbol 250 Bilinear Pooling Layer, Bilinear Pooling 260 Gating Layer, Gating 300 Conformer Blocks 305 Concatenation Operator 310 First half feedforward layer 320 Multi-head Self-Attention Block 330 convolutional layers 340 Late Feedforward Layer 350 layernorm modules 360 Output Features 400 Computer-implemented methods, methods 500 computing devices 500a Standard Server, Server 500b laptop computer 500c Rack Server System 510 Data Processing Hardware, Processors, and Components 520 Memory Hardware, Memory, Components, Non-Temporary Memory 530 Storage devices, components 540 High-Speed ​​Interface / Controller, Components, High-Speed ​​Interface, High-Speed ​​Controller 550 High-Speed ​​Expansion Port, Components 560 Low-Speed ​​Interface / Controller, Components, Low-Speed ​​Controller 570 Slow Bus 580 Display 590 Low-Speed ​​Expansion Port

Claims

1. An automated speech recognition (ASR) model (200), An encoder network (220), receiving as input a sequence of acoustic frames (222) characterizing an input utterance; generating, in each of a plurality of output steps, a high-level feature representation (224) for a corresponding acoustic frame in the sequence of acoustic frames (222); an encoder network (220) configured to: A prediction network (230), receiving as input the sequence of non-blank symbols (242) output by the final softmax layer (240); generating a dense representation (232) in each of said plurality of output steps; a prediction network (230) configured to: A joint network (210), receiving as input the dense representation (232) produced by the prediction network (230) in each of the plurality of output steps and the high-level feature representation (224) produced by the encoder network (220) in each of the plurality of output steps; generating a probability distribution (212) over the possible speech recognition hypotheses at each of the plurality of output steps; a joint network (210) configured to perform Equipped with the joint network (210) comprises a combinatorial structure that stacks gating (260) and bilinear pooling (250) to fuse the dense representation (232) generated by the prediction network (230) and the high-level feature representation (224) generated by the encoder network (220); The automated speech recognition (ASR) model (200) is configured such that the final softmax layer (240) selects the output symbol with the highest probability in the probability distribution (212) output from the joint network (210) to output the sequence of non-blank symbols (242).

2. 2. The ASR model of claim 1, wherein a regularization method is applied to the predictive network during training by recalculating the dense representation using a scaling factor and a gradient stopping function with input tensors that have zero gradient.

3. 2. The ASR model of claim 1, wherein the joint network does not include a fully connected layer.

4. 2. The ASR model of claim 1, wherein the encoder network comprises a stack of self-attention blocks.

5. 5. The ASR model (200) of claim 4, wherein the stack of self-attention blocks comprises a stack of conformer blocks.

6. 6. The ASR model (200) of claim 5, wherein the stack of conformer blocks comprises a stack of 12 encoder blocks with 8-head self-attention.

7. 5. The ASR model (200) of claim 4, wherein the stack of self-attention blocks comprises a stack of transformer blocks.

8. 8. The ASR model (200) of claim 1, wherein the prediction network (230) comprises a long short-term memory (LSTM)-based prediction network.

9. 8. The ASR model (200) of claim 1, wherein the prediction network (230) comprises a V2 embedded lookup table.

10. 8. The ASR model (200) of claim 1, wherein the prediction network (230) comprises a stateless prediction network.

11. A computer-implemented method (400) that, when executed on data processing hardware (510), causes the data processing hardware (510) to perform operations, the operations comprising: receiving a sequence of acoustic frames (222) characterizing an input utterance; In each of the plurality of output steps, generating, by an encoder network (220) of the speech recognition model (200), a high-level feature representation (224) for a corresponding acoustic frame in the sequence of acoustic frames (222); generating, by a prediction network (230) of the speech recognition model (200), a dense representation (232) for the corresponding sequence of non-blank symbols (242) output by a final softmax layer (240) of the speech recognition model (200); generating a probability distribution (212) over possible speech recognition hypotheses by a joint network (210) of the speech recognition model, which receives the high-level feature representation (224) generated by the encoder network (220) and the dense representation (232) generated by the prediction network (230); Including, the joint network (210) comprises a combinatorial structure that stacks gating (260) and bilinear pooling (250) to fuse the dense representation (232) generated by the prediction network (230) and the high-level feature representation (224) generated by the encoder network (220); The computer-implemented method (400) is configured to select the output symbol with the highest probability in the probability distribution (212) output from the joint network (210) to output the sequence of non-blank symbols (242).

12. 12. The computer-implemented method of claim 11, wherein a regularization method is applied to the predictive network during training by recalculating the dense representation using a scaling factor and a gradient stopping function with input tensors that have zero gradient.

13. The computer-implemented method of claim 11 , wherein the joint network (210) does not include a fully connected layer.

14. 12. The computer-implemented method of claim 11, wherein the encoder network (220) comprises a stack of self-attention blocks.

15. 15. The computer-implemented method of claim 14, wherein the stack of self-attention blocks comprises a stack of conformer blocks.

16. 16. The computer-implemented method of claim 15, wherein the stack of conformer blocks comprises a stack of 12 encoder blocks with 8-head self-attention.

17. 15. The computer-implemented method of claim 14, wherein the stack of self-attention blocks comprises a stack of transformer blocks.

18. 18. The computer-implemented method of claim 11, wherein the prediction network (230) comprises a long short-term memory (LSTM)-based prediction network.

19. 18. The computer-implemented method of claim 11, wherein the predictive network (230) comprises a V2 embedded lookup table.

20. 18. The computer-implemented method of claim 11, wherein the prediction network (230) comprises a stateless prediction network.

Images