Expert hybrid Conformer for streaming multi-language ASR

Abstract

A method (500) of a multi-language ASR model (200) includes receiving a sequence (110) of acoustic frames characterizing an utterance (106) of speech. At the plurality of output steps, the method further comprises: generating, by a first encoder (210) comprising a first plurality of multi-head attention layers (300a), a first high-order feature representation (212) for the acoustic frame; generating, by a second encoder (220) comprising a second plurality of multi-head attention layers (300b), a second high-order feature representation (222) for the corresponding first high-order feature representation; and generating, by the first decoder (240a), a first probability distribution (120a) with respect to the possible speech recognition hypotheses. The gating layer (410) of each respective MoE layer (400) is configured to dynamically route an output (402) from a previous multi-head attention layer to a respective pair of feed-forward expert networks (420) at each of the plurality of output steps.

Description

Technical Field

[0001] The present disclosure relates to an expert hybrid conformer for streaming multilingual ASR. Background Art

[0002] Automatic speech recognition (ASR), the process of taking audio input and transcribing it into text, has largely become an important technology used in mobile and other devices. Generally speaking, automatic speech recognition attempts to provide an accurate transcription of what people say by taking audio input (e.g., a spoken utterance) and transcribing that audio input into text. Modern ASR models are based on the continuous development of deep neural networks, with continuous improvements in both accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay between the client speaking and the transcription). However, one challenge in developing ASR models based on deep learning is that deep neural networks benefit from over-parameterization, resulting in ASR models including well over 100 million parameters and requiring hundreds of thousands of training steps to converge. Therefore, training these over-parameterized ASR models is a resource-intensive process that may not be suitable for devices with limited computing resources and memory. Summary of the Invention

[0003] One aspect of the present disclosure provides a multilingual automatic speech recognition (ASR) model, comprising a first encoder configured to receive as input a sequence of acoustic frames representing speech utterances and, at each of a plurality of output steps, generate a first high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The first encoder comprises a first plurality of multi-headed attention layers. The multilingual ASR model further comprises a second encoder configured to receive as input the first high-order feature representation generated by the first encoder at each of the plurality of output steps and, at each of the plurality of output steps, generate a second high-order feature representation for the corresponding first high-order feature frame. The second encoder is cascaded to the first encoder and comprises a second plurality of multi-headed attention layers. The multilingual ASR model further comprises a first decoder configured to receive as input the second high-order feature representation generated by the second encoder at each of the plurality of output steps and a sequence of N previous non-whitespace symbols output by a final softmax layer at each of the plurality of output steps, and generate a first probability distribution over possible speech recognition hypotheses at each of the plurality of output steps. Each of the first and second multi-head attention layers includes an initial feedforward network, a multi-head self-attention layer, a convolutional layer, and a final feedforward network. At least one of the initial feedforward network or the final feedforward network of at least one corresponding multi-head attention layer in the second plurality of multi-head attention layers includes a corresponding mixture of experts (MoE) layer, wherein each corresponding MoE layer includes a gating layer and multiple feedforward expert networks. The gating layer of each corresponding MoE layer is configured to dynamically route the output from the previous multi-head attention layer to a corresponding pair of feedforward expert networks including the highest weight among the multiple feedforward expert networks at the corresponding multi-head attention layer in the second plurality of multi-head attention layers at each output step in the plurality of output steps, without routing the output to other feedforward expert networks among the multiple feedforward expert networks.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, each initial feedforward network in the second plurality of multi-head attention layers includes a corresponding MoE layer. Each final feedforward network in the second plurality of multi-head attention layers may include a corresponding MoE layer. Each feedforward expert network among the multiple feedforward expert networks of the corresponding MoE layer may include a single feedforward network layer. In some implementations, when the output from the previous multi-head attention layer is dynamically routed to the corresponding pair of feedforward expert networks at each of the multiple output steps, the gating layer of the corresponding MoE layer does not rely on any language information associated with the sequence of audio frames representing the speech.

[0005] In some examples, the gating layer of the corresponding MoE layer is configured to be based on To determine the corresponding weights of each corresponding feedforward expert network among the multiple feedforward expert networks. Here, x includes the output of the previous layer at the corresponding output step, W l Includes the weight matrix of the gated layer at the corresponding l-th multi-head attention layer, and g l Including corresponding weights of corresponding feedforward expert networks. Each feedforward expert network in the pair of feedforward expert networks including the highest weight can be configured to determine a corresponding output based on an output routed from a previous multi-head attention layer by the gating layer, and the corresponding MoE layer is configured to determine the MoE output based on the sum of the corresponding outputs determined by the pair of feedforward expert networks including the highest weight.

[0006] In some implementations, the multilingual ASR model further includes a second decoder configured to receive as input a first high-order feature representation generated by the first encoder at each of a plurality of output steps, and to generate a second probability distribution over possible speech recognition hypotheses at each of the plurality of output steps. Here, the second decoder may be further configured to generate a partial speech recognition result based on the second probability distribution over possible speech recognition hypotheses. In these implementations: the first decoder and the second decoder may each include a corresponding prediction network followed by a corresponding joint network; the corresponding prediction networks of the first decoder and the second decoder have the same structure including one of a long short-term memory (LSTM)-based prediction network or a V2 embedding lookup table; and the corresponding joint networks of the first decoder and the second decoder include the same structure.

[0007] In some examples, the second encoder generates a second high-order feature representation without receiving any of the acoustic frames as input. The first encoder may include a causal encoder. The second encoder may include a non-causal encoder. The first encoder and the second encoder may be jointly trained on a multilingual training utterance set using a negative log-likelihood. In some implementations, the respective MoE layers are trained on an auxiliary loss to encourage load balancing across the multiple feedforward expert networks, and the auxiliary loss is based on an average gate over all frames of each corresponding feedforward expert network and a portion of the output from the previous multi-head attention layer that the gating layer routes to each corresponding feedforward expert network.

[0008] Another aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for performing speech recognition using a mixture of experts conformer. The operations include receiving a sequence of acoustic frames representing speech utterances. At each output step in a plurality of output steps, the operations further include: generating, by a first encoder of a multilingual automatic speech recognition (ASR) model comprising a first plurality of multi-headed attention layers, a first high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames; generating, by a second encoder of the multilingual ASR model comprising a second plurality of multi-headed attention layers, a second high-order feature representation for the corresponding first high-order feature representation; and generating, by a first decoder of the multilingual ASR model, a first probability distribution over possible speech recognition hypotheses based on the second high-order feature representation generated by the second encoder at the corresponding output step and a sequence of N previous non-whitespace symbols output by a final softmax layer at the corresponding output step. Each of the first and second plurality of multi-headed attention layers comprises an initial feedforward network, a multi-headed self-attention layer, a convolutional layer, and a final feedforward network. At least one of the initial feedforward network or the final feedforward network of at least one corresponding multi-head attention layer in the second plurality of multi-head attention layers includes a corresponding mixture of experts (MoE) layer, each corresponding MoE layer including a gating layer and a plurality of feedforward expert networks. The gating layer of each corresponding MoE layer is configured to dynamically route the output from the previous multi-head attention layer at each output step in the plurality of output steps to a corresponding pair of feedforward expert networks among the plurality of feedforward expert networks, including the highest weight among the plurality of feedforward expert networks at the corresponding multi-head attention layer in the second plurality of multi-head attention layers, without routing the output to other feedforward expert networks among the plurality of feedforward expert networks.

[0009] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, each initial feedforward network in the second plurality of multi-head attention layers includes a corresponding MoE layer. Each final feedforward network in the second plurality of multi-head attention layers may include a corresponding MoE layer. Each feedforward expert network among the multiple feedforward expert networks of the corresponding MoE layer may include a single feedforward network layer. In some examples, when the output from the previous multi-head attention layer is dynamically routed to the corresponding pair of feedforward expert networks at each of the multiple output steps, the gating layer of the corresponding MoE layer does not rely on any language information associated with the sequence of audio frames representing the speech.

[0010] In some implementations, the operations further include: performing, by the gating layer of the corresponding MoE layer, a gate at each of the plurality of output steps based on To determine the corresponding weights of each corresponding feedforward expert network among the multiple feedforward expert networks. Here, x includes the output of the previous layer at the corresponding output step, W l Includes the weight matrix of the gated layer at the corresponding l-th multi-head attention layer, and g l Including corresponding weights of corresponding feed-forward expert networks. In some examples, the operations further include: determining, by each feed-forward expert network in the pair of feed-forward expert networks including the highest weight, a corresponding output based on an output routed by the gating layer from a previous multi-head attention layer, and determining, by the corresponding MoE layer, an MoE output based on a sum of corresponding outputs determined by the pair of feed-forward expert networks including the highest weight.

[0011] In some implementations, the operations further include: receiving a first high-order feature representation generated by the first encoder at each of a plurality of output steps as input to a second decoder of the multilingual ASR model; and generating, by the second decoder, a second probability distribution over possible speech recognition hypotheses at each of the plurality of output steps. In these implementations, the operations may further include: generating, by the second decoder, a partial speech recognition result based on the second probability distribution over possible speech recognition hypotheses. In some examples, the first decoder and the second decoder each include a corresponding prediction network followed by a corresponding joint network; the corresponding prediction networks of the first decoder and the second decoder have the same structure including one of a long short-term memory (LSTM)-based prediction network or a V2 embedding lookup table; and the corresponding joint networks of the first decoder and the second decoder include the same structure.

[0012] In some examples, generating the second high-order feature representation includes generating the second high-order feature representation without receiving any of the acoustic frames as input. The first encoder may include a causal encoder. The second encoder may include a non-causal encoder. In some implementations, the operations further include: jointly training the first encoder and the second encoder on a set of multilingual training utterances using negative log-likelihood. In some examples, the operations further include: training the corresponding MoE layer on an auxiliary loss to encourage load balancing across multiple feedforward expert networks. The auxiliary loss is based on an average gate over all frames of each corresponding feedforward expert network and a portion of the output from the previous multi-head attention layer that the gating layer routes to each corresponding feedforward expert network.

[0013] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a schematic diagram of an example speech recognition system.

[0015] Figure 2 is a diagram of an example speech recognition model.

[0016] Figure 3 Schematic diagram of an example multi-head self-attention layer from multiple multi-head self-attention layers.

[0017] Figure 4 is a diagram of an example mixture-of-experts layer.

[0018] Figure 5 is a flow diagram of an example operational arrangement of a computer-implemented method using an expert mixture conformer for streaming multilingual speech recognition.

[0019] Figure 6 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0020] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0021] End-to-end (E2E) automatic speech recognition (ASR) models are traditionally structured to operate in either streaming mode or non-streaming mode. Conventionally, E2E ASR models include an encoder and a decoder as the main components. Applications involving end-user interaction, such as voice search or on-device dictation, may require the model to perform recognition in a streaming manner. Here, performing recognition in a streaming manner means that the ASR model outputs each word in an utterance as it is spoken with as little delay as possible. Other applications, such as offline video subtitle generation, do not require the model to be streaming and can take advantage of future scenarios to improve performance.

[0022] In some scenarios, the E2E ASR model is configured to recognize speech from multiple languages ​​(e.g., a multilingual ASR model). Here, the multilingual ASR model uses a separate language recognition model to perform speech recognition on speech from multiple different languages. Although multilingual ASR models and language recognition models are often used together in downstream tasks (e.g., transcoding and speech translation), they are constructed and executed separately. Separately executing the multilingual ASR model and language recognition model increases the computational and storage costs associated with performing speech recognition.

[0023] Therefore, implementations herein relate to a multilingual ASR model and a method for operating a multilingual ASR model. The method includes generating, by a first encoder of the multilingual ASR model comprising a first plurality of multi-headed attention layers, a first high-order feature representation for a corresponding acoustic frame in a sequence of acoustic frames. The method also includes generating, by a second encoder of the multilingual ASR model comprising a second plurality of multi-headed attention layers, a second high-order feature representation for the corresponding first high-order feature representation. The method also includes generating, by a first decoder of the multilingual ASR model, a first probability distribution over possible speech recognition hypotheses based on the second high-order feature representation generated by the second encoder at a corresponding output step and a sequence of N previous non-whitespace symbols output by a final softmax layer at the corresponding output step. Each of the first and second plurality of multi-headed attention layers includes an initial feedforward network, a multi-headed self-attention layer, a convolutional layer, and a final feedforward network. Furthermore, at least one of the initial feedforward network or the final feedforward network of at least one corresponding multi-headed attention layer in the second plurality of multi-headed attention layers includes a corresponding mixture of experts (MoE) layer, each corresponding MoE layer including a gating layer and multiple feedforward expert networks. The gating layer of each corresponding MoE layer is configured to dynamically route the output from the previous multi-head attention layer at each output step in the multiple output steps to a corresponding pair of feed-forward expert networks including the highest weight among the multiple feed-forward expert networks at the corresponding multi-head attention layer in the second multiple multi-head attention layers, without routing the output to other feed-forward expert networks among the multiple feed-forward expert networks.

[0024] Figure 1 An example system is depicted whereby a user 104 interacts with a computing device, such as a user device 10, by way of voice input. The user device 10 (also generally referred to as device 10) is configured to capture sound (e.g., streamed audio data) from one or more users 104 within the system 100. Here, the streamed audio data may refer to spoken utterances 106 of the user 104, which serve as audible queries, commands to the user device 10, or audible communications captured by the device 10. The voice-enabled system of the user device 10 may process the query or command by answering the query and / or causing the command to be executed / fulfilled by one or more downstream applications.

[0025] The user device 10 may correspond to any computing device associated with the user 104 and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, mobile devices (e.g., smart watches), smart home appliances, Internet of Things (IoT) devices, in-vehicle infotainment systems, smart displays, smart speakers, and the like. The user device 10 includes data processing hardware 12 and memory hardware 14 that communicates with the data processing hardware 12 and stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 having an audio capture device (e.g., a microphone) 16, 16a for capturing spoken utterances 106 within the system 100 and converting the spoken utterances into electrical signals, and a voice output device (e.g., a speaker) 16, 16b for transmitting audible audio signals (e.g., as output data from the user device 10). While the user device 10 may implement an array of audio capture devices 16 a without departing from the scope of this disclosure, one or more of the capture devices 16 a in the array may not physically reside on the user device 10 but rather communicate with the audio system 16 .

[0026] In system 100, an automatic speech recognition (ASR) system 118 implements an ASR model 200 and resides on a user device 10 of user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system executing in a cloud computing environment) that communicates with user device 10 via network 40. In some examples, ASR model 200 can be a recurrent neural network transducer (RNN-T) model. User device 10 and / or remote computing device 60 also includes an audio subsystem 108 that is configured to receive an utterance 106 spoken by user 104 and captured by audio capture device 16a and convert the utterance 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by the ASR system 118. In the example shown, the user speaks a corresponding utterance 106, and the audio subsystem 108 converts the utterance 106 into corresponding audio data (e.g., a sequence of acoustic frames) 110 for input to the ASR system 118. Thereafter, the ASR model 200 receives as input a sequence of acoustic frames 110 corresponding to the utterance 106 and generates / predicts a corresponding transcription 120 (e.g., a speech recognition result / hypothesis) of the utterance 106 at each output step as the ASR model receives (e.g., processes) each acoustic frame 110 in the sequence of acoustic frames 110.

[0027] In the example shown, the ASR model 200 can perform streaming speech recognition to produce initial speech recognition results 120, 120a, and generate final speech recognition results 120, 120b by improving the initial speech recognition results 120a. The speech recognition results 120 can correspond to partial speech recognition results or the entire speech recognition results. In other words, the speech recognition results 120 can correspond to a portion of the utterance 106 or the entire utterance 106. For example, the partial speech recognition results can correspond to a portion of a spoken utterance or even a portion of a spoken word. However, as will be apparent, the ASR model 200 performs additional processing on the final speech recognition results 120b, whereby the final speech recognition results 120b can be delayed relative to the initial speech recognition results 120a.

[0028] The user device 10 and / or the remote computing device 60 also executes a user interface generator 107 that is configured to present a representation of a transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 can display the initial speech recognition result 120a in a streaming manner during time 1 and then display the final speech recognition result 120b in a streaming manner during time 2. Notably, the ASR model 200 outputs the final speech recognition result 120b in a streaming manner, even if the final speech recognition result 120b improves upon the initial speech recognition result 120a. In some configurations, the transcription 120 output from the ASR system 118 is processed, for example, by a natural language understanding (NLU) module executing on the user device 10 or the remote computing device 60, to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of user device 10 or remote computing device 60) may convert transcription 120 into synthesized speech for audible output by user device 10 and / or another device.

[0029] In the example shown, the user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) of the user device 10 using the ASR system 118. For example, Figure 1A user 104 is depicted communicating with a digital assistant application 50, and the digital assistant application 50 displays a digital assistant interface 18 on a screen of a user device 10 depicting the conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50: "What time is the concert tonight?" The question from the user 104 is a spoken utterance 106 that is captured by an audio capture device 16a and processed by the audio system 16 of the user device 10. In this example, the audio system 16 receives the spoken utterance 106 and converts the spoken utterance into a sequence of acoustic frames 110 for input to an ASR system 118.

[0030] Continuing with this example, upon receiving the acoustic frame sequence 110 corresponding to the utterance 106 spoken by the user 104, the ASR model 200 encodes the acoustic frame sequence 110 and then decodes the encoded acoustic frame sequence 110 into an initial speech recognition result 120a. During time 1, the user interface generator 107 presents a representation of the initial speech recognition result 120a of the utterance 106 to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming manner, such that words, word fragments, and / or individual characters appear on the screen as soon as they are spoken. In some examples, the first look-ahead audio context is equal to zero.

[0031] During time 2, the user interface generator 107 presents a representation of the final speech recognition result 120b of the utterance 106 to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming manner, such that the words, word fragments, and / or individual characters appear on the screen immediately as they are generated by the ASR model 200. In some implementations, the user interface generator 107 replaces the representation of the initial speech recognition result 120a presented at time 1 with the representation of the final speech recognition result 120b presented at time 2. Here, time 1 and time 2 may include timestamps corresponding to when the user interface generator 107 presented the respective speech recognition results 120. In this example, the timestamp at time 1 indicates that the user interface generator 107 presented the initial speech recognition result 120a at an earlier time than the final speech recognition result 120b. For example, because final speech recognition result 120b is deemed more accurate than initial speech recognition result 120a, the final speech recognition result 120b ultimately displayed as the transcription 120 can repair any terms that may have been incorrectly recognized in the initial speech recognition result 120a. In this example, the streaming initial speech recognition result 120a output by the ASR model 200 and displayed on the screen of the user device 10 at time 1 is associated with low latency and provides the user 104 with a response that his / her query is being processed, while the final speech recognition result 120b output by the ASR model 200 and displayed on the screen at time 2 utilizes additional speech recognition models and / or language models to improve the speech recognition quality in terms of accuracy, but increases latency. However, since the initial speech recognition result 120a is displayed as the user speaks the utterance 106, the higher latency associated with generating and ultimately displaying the final speech recognition result 120b is not noticeable to the user 104.

[0032] exist Figure 1In the illustrated example, the digital assistant application 50 can use natural language processing to respond to the question posed by the user 104. Natural language processing generally refers to the process of interpreting written language (e.g., the initial speech recognition result 120a and / or the final speech recognition result 120b) and determining whether the written language prompts any action. In this example, the digital assistant application 50 uses natural language processing to identify that the question from the user 104 is about the user’s schedule and, more specifically, about a concert on the user’s schedule. By utilizing natural language processing to identify these details, the automated assistant returns a response 19 to the user’s query, where the response 19 states, “Venue doors open at 6:30 PM and concert starts at 8pm.” In some configurations, the natural language processing occurs on a remote server 60 in communication with the data processing hardware 12 of the user device 10.

[0033] Referring now to Figure 2 In some implementations, the ASR model 200 includes a cascaded encoder 204 and decoders 240. Optionally, the ASR model 200 can include a language ID predictor 230. However, in some scenarios, the ASR model 200 operates without the language ID predictor 230. The first decoder 240, 240a can operate in a streaming fashion such that the first decoder 240a is configured to generate a partial speech recognition result corresponding to the initial speech recognition result 120a. On the other hand, the second decoder 240, 240b is configured to improve the initial speech recognition result 120a output by the first decoder 240a. The second decoder 240b improves the partial speech recognition result by receiving additional right-side context and generating a final speech recognition result 120b. The first decoder 240a and the second decoder 240b each include a corresponding prediction network 260 followed by a corresponding joint network 250. Here, the first prediction network 260, 260a and the first joint network 250, 250a correspond to the first decoder 240a, and the second prediction network 260, 260b and the second joint network 250, 250b correspond to the second decoder 240b. The prediction networks 260a, 260b have the same structure including one of a long short-term memory (LSTM) based prediction network or a V2 embedding lookup table. Further, the corresponding joint networks 250a, 250b have the same structure. While the component structures are the same for the first decoder 240a and the second decoder 240b, the respective components of each decoder 240 are unique and can be trained independently of the components of the other decoder 240.

[0034] The cascaded encoder 204 involves a model structure in which the encoding path includes two encoders 210, 220 that are cascaded such that the output of the first encoder 210 is fed to the input of the second encoder 220 before decoding. The first encoder 210 and the second encoder 220 can be jointly trained on a set of multilingual training utterances using a negative log-likelihood loss. Here, the first encoder 210 and the second encoder 220 can be cascaded regardless of the underlying architecture of each encoder. The encoders 210, 220 can each include a stack of multi-head self-attention layers (i.e., multiple multi-head attention layers) 300. In particular, the first encoder 210 includes a first plurality of multi-head self-attention layers 300, 300a, and the second encoder 220 includes a second plurality of multi-head self-attention layers 300, 300b. In some examples, the first encoder 210 includes a causal encoder, whereby the stack of multi-head attention layers includes one or more unidirectional (LSTM) layers, multiple conformer layers, or multiple transformer layers. For example, the stack of multi-head self-attention layers 300a of the first encoder 210 can include twelve (12) conformer layers, each conformer layer having a multi-head (e.g., eight (8) heads) self-attention mechanism and a convolutional kernel size of fifteen (15). Further, the first encoder 210 can perform a concatenation operation after the third conformer layer to achieve a temporal reduction rate of 2, whereby the resulting 1024-dimensional vector is transformed by the fourth conformer layer and then projected back to a 512-dimensional vector using another linear transformation. Thereafter, an additional eight (8) conformer layers follow with a final normalization layer. Thus, the first encoder 210 can include 110 million parameters. Each layer of the first encoder 210 receives zero right-side context (e.g., receives zero future acoustic frames).

[0035] The second encoder 220 includes an acausal encoder, whereby the stack of multi-head self-attention layers 300b includes one of the following: one or more bidirectional LSTM layers, multiple conformer layers, or multiple transformer layers. For example, the second encoder 220 can include a 512-dimensional linear projection to transform the input features, followed by five (5) 512-dimensional conformer layers and a final linear normalization layer, resulting in 50 million parameters. Here, the second encoder 220 can receive additional right-side context, e.g., a total of fifteen (15) frames of right-side context, whereby each conformer layer receives three (3) frames of right-side context.

[0036] With continued reference to Figure 2 , the first encoder 210 receives a sequence of d-dimensional feature vectors (e.g., a sequence of acoustic frames 110) x = (x1, x2,..., x T ), where , and generates a first high-order feature representation 212 for the corresponding acoustic frame 110 in the acoustic frame sequence 110 at each output step. Similarly, the second encoder 220 is connected to the first encoder 210 in a cascade manner and receives the first high-order feature representation 212 as input, and generates a second high-order feature representation 222 for the corresponding first high-order feature representation 212 at each output step. In some cases, the second encoder 220 generates the second high-order feature representation 222 without receiving any acoustic frame in the acoustic frames 110 as input. In these cases, the second encoder 220 generates the second high-order feature representation 222 using only the first high-order feature representation 212 as input. Therefore, the first high-order feature representation 212 output from the first encoder 210 is fed to the language ID predictor 230 and the first decoder 240a, while the second high-order feature representation 222 output from the second encoder 220 is fed to the second decoder 240b and the language ID predictor 230. However, in a configuration where the ASR model 200 does not include the language ID predictor 230 , the first high-order feature representation 212 and the second high-order feature representation 222 are fed to the first decoder 240 a and the second decoder 204 b , respectively, and not to the language ID predictor 230 .

[0037] Continue to refer Figure 2 , the first decoder 240a includes a first joint network 250a and a first prediction network 260a. The first joint network 250a is configured to receive the dense representation 265 generated by the first prediction network 260a and the first high-order feature representation 212 generated by the first encoder 210 as input, and generate an initial speech recognition result 120a for the corresponding acoustic frame 110 at each output step. Here, the first joint network 250a uses the first high-order feature representation 212 and the dense representation 265 to generate the initial speech recognition result 120a. The first decoder 240a operates in a streaming manner, so that the first decoder 240a allows the initial speech recognition result 120a to correspond to the partial speech recognition result.

[0038] In some implementations, the initial speech recognition result 120a includes a first probability distribution about possible speech recognition hypotheses. Therefore, the initial speech recognition result 120a can be used interchangeably herein with the first probability distribution 120a about possible speech recognition hypotheses. Therefore, the first joint network 250a can generate a first probability distribution 120a about possible speech recognition hypotheses at each output step (e.g., time step). Here, a "possible speech recognition hypothesis" corresponds to a set of output labels / symbols (also referred to as "speech units"), each output label / symbol representing a grapheme (symbol / character) or word fragment in a specified natural language. For example, when the natural language is English, the set of output labels can include twenty-seven (27) symbols, for example, one label for each of the 26 letters in the English alphabet, and one label specifying a space. Therefore, the first joint network 250a can output a set of values ​​indicating the likelihood of each output label in a predetermined set of output labels occurring. The set of values ​​can be a vector (e.g., a one-hot vector) and can indicate a second probability distribution about the set of output labels. In some scenarios, the output labels are graphemes (e.g., individual characters, and potentially punctuation marks and other symbols), but the set of output labels is not limited thereto. For example, in addition to or in lieu of graphemes, the set of output labels may include word fragments and / or entire words. These output labels may also be other types of speech units, such as phonemes or subphones. The first probability distribution 120a of the first joint network 250a may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output of the joint network 250 may include 100 different probability values, one for each output label. The first probability distribution 120a may then be used (e.g., by a final Softmax layer (not shown) of the first joint network 250a) to select candidate orthographic elements (e.g., graphemes, word fragments, and / or words) in a beam search process and assign scores to them for use in determining the initial speech recognition result 120a. For example, the first joint network 250a may select the top N possible speech recognition hypotheses with the highest probabilities as output for the initial speech recognition result 120a.

[0039] In some implementations, the first prediction network 260a receives as input the non-whitespace symbol sequence output by the final softmax layer of the first joint network 250a and generates a dense representation 265 at each output step. That is, the first joint network 250a receives the dense representation 265 for the previous initial speech recognition result 120a and uses the dense representation 265 to generate a subsequent initial speech recognition result 120a.

[0040] In some configurations, the language ID predictor 230 of the ASR model 200 is configured to receive as input the first high-order feature representation 212 generated by the first encoder 210 at each of the plurality of output steps and the second high-order feature representation 222 generated by the second encoder 220 at each of the plurality of output steps. Furthermore, the language ID predictor 230 may generate a concatenation 231 of the first high-order feature representation 212 and the second high-order feature representation 222. Thereafter, the language ID predictor 230 is further configured to generate a language prediction representation 232 based on the concatenation 231 of the first high-order feature representation 212 and the second high-order feature representation 222 at each of the plurality of output steps. Advantageously, by generating the concatenation 231, the language ID predictor 230 uses diverse inputs to generate the language prediction representation 232.

[0041] The language prediction representation 232 indicates the corresponding language of the spoken utterance. For example, because the ASR model 200 is a multilingual ASR model, the spoken utterance can be in any number of languages. Therefore, using the cascade 231, the language ID predictor 230 predicts the corresponding language of the spoken utterance. The language prediction representation 232 can be used for downstream tasks (e.g., transcoding or speech translation) and / or to improve speech recognition results. That is, the second decoder 240b can use the language prediction representation 232 to improve the initial speech recognition results 120a generated by the first decoder 240a. In some examples, the language ID predictor 230 generates the language prediction representation 232 on a frame-by-frame basis. In these examples, the spoken utterance can include multiple utterances, and the language ID predictor 230 generates the language prediction representation 232 for each acoustic frame 110 in the acoustic frame sequence 110. For example, for the first portion of the acoustic frame sequence, the language prediction representation 232 can indicate that the first language was spoken, while for the second portion of the acoustic frame sequence, the language prediction representation 232 indicates that the second language was spoken.

[0042] Continue to refer Figure 2, the second decoder 240b includes a second joint network 250b and a second prediction network 260b. In some configurations, the second joint network 250b is configured to receive as input the dense representation 265 generated by the second prediction network 260b, the second high-order feature representation 222 generated by the second encoder 220, and the language prediction representation 232 generated by the language ID predictor 230, and generate a final speech recognition result 120b for the corresponding acoustic frame 110 at each output step. Here, the second joint network 250b uses the second high-order feature representation 222, the language prediction representation 232, and the second dense representation 265 to generate the final speech recognition result 120b. In some configurations, the second joint network 250b generates the final speech recognition result 120b without using the language prediction representation 232. In some examples, the second decoder 240b generates a cascade of the second high-order feature representation 222 and the language prediction representation 232, and uses the cascade to generate the final speech recognition result 120b.

[0043] In some implementations, the final speech recognition result 120b includes a second probability distribution about possible speech recognition hypotheses. Therefore, the final speech recognition result 120b can be used interchangeably with the second probability distribution 120b about possible speech recognition hypotheses in this article. Therefore, the second joint network 250b can generate a second probability distribution 120b about possible speech recognition hypotheses at each output step (e.g., time step). Here, a "possible speech recognition hypothesis" corresponds to a set of output labels / symbols (also referred to as "speech units"), each output label / symbol representing a grapheme (symbol / character) or word fragment in a specified natural language. For example, when the natural language is English, the set of output labels can include twenty-seven (27) symbols, for example, one label for each of the 26 letters in the English alphabet, and one label specifies a space. Therefore, the second joint network 250b can output a set of values ​​indicating the likelihood of each output label in a predetermined set of output labels occurring. The set of values ​​can be a vector (e.g., a one-hot vector) and can indicate a first probability distribution about the set of output labels. In some scenarios, the output labels are graphemes (e.g., individual characters, and potentially punctuation marks and other symbols), but the set of output labels is not limited thereto. For example, in addition to or in lieu of graphemes, the set of output labels may include word fragments and / or entire words. These output labels may also be other types of speech units, such as phonemes or subphones. The second probability distribution 120b of the second joint network 250b may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output of the second joint network 250b may include 100 different probability values, one for each output label. The second probability distribution 120b may then be used (e.g., by a final Softmax layer (not shown) of the second joint network 250b) to select candidate orthographic elements (e.g., graphemes, word fragments, and / or words) during a beam search process and assign scores to them for use in determining the final speech recognition result 120b. For example, the second joint network 250b may select the top N possible speech recognition hypotheses with the highest probabilities as output for the final speech recognition result 120b.

[0044] In some implementations, the second prediction network receives as input the non-whitespace symbol sequence output by the final softmax layer of the second joint network 250b and generates a dense representation 265 at each output step. That is, the second joint network 250b receives the dense representation 265 for the previous final speech recognition result 120b and uses the dense representation 265 to generate a subsequent final speech recognition result 120b.

[0045] In some implementations, the language ID predictor 230 uses more acoustic information (e.g., longer audio features) to generate a more accurate language prediction representation 232. Therefore, in order to utilize all past acoustic frames 110 but still generate the language prediction representation 232 on a frame-by-frame basis, the language ID predictor 230 uses non-parametric statistical pooling. That is, the language ID predictor 230 converts the first high-order feature representation 212 into an average of the first high-order feature representation 212. and standard deviation It is worth noting that the language ID predictor 230 determines the mean and standard deviation in a streaming manner, which are represented by the following equations: In equations 1 and 2, h1 represents the first high-order feature representation 212. The first high-order feature representation 212 is converted into a concatenated vector using statistical pooling. The language ID predictor 230 then uses two fully connected layers followed by a softmax output layer to convert the concatenated vectors into a language prediction representation 232. Therefore, the frame-synchronized language ID predictor 230 is efficient to operate in a streaming manner and requires only a small amount of computational cost during execution.

[0046] In some implementations, the ASR model 200 jointly trains the first encoder 210, the second encoder 220, and the language ID predictor 230 on a set of multilingual training utterances. Here, a language ID target token is added as the first token of the corresponding ground-truth transcription of each multilingual training utterance in the set of multilingual training utterances. The language ID target token identifies the language of the corresponding multilingual training utterance. That is, the set of multilingual training utterances can include training utterances in any number of different languages, and the language ID target token identifies the actual language (e.g., ground-truth label) of the multilingual training utterance for training purposes.

[0047] During training, the training process generates a first loss for the first encoder 210 and a second loss for the second encoder 220, which are represented by the following equations: In equations 3 and 4, Denotes the loss of the decoder 240 (e.g., a recurrent neural network Transducer loss), x denotes the acoustic frame sequence 110, and y denotes the transcription 120. The ASR model 200 uses two separate decoders 240, and therefore, the training loss of the ASR model 200 is expressed by the following equation: In Equation 5, represents the loss of the first decoder 240a, represents the loss of the second decoder 240b, represents a weighting factor for the loss of the first decoder 240a, and represents the weight factor of the loss of the second decoder 240b. In addition, the training process generates a third loss for the language ID predictor 230, which is represented by the following equation: In Equation 6, represents the third loss for the language ID predictor 230, and l t The one-hot language prediction of t represents the label. Therefore, the training process trains the ASR model 200 using the final training loss according to the following equation: In Equation 7, is a scalar weight for the loss of the language ID predictor 230. Therefore, the training process trains the ASR model 200 by minimizing the weighted sum of the first loss, the second loss, and the third loss.

[0048] Figure 3 shows the output from the cascaded encoder 204 ( Figure 2 ). The example multi-head self-attention layer 300 is a stack 300 of multi-head self-attention layers of the first encoder 210 (e.g., the first plurality of multi-head self-attention layers 300a) and / or the second encoder 220 (e.g., the second plurality of multi-head self-attention layers 300b). The example multi-head self-attention layer 300 (also referred to as "layer 300") includes an initial feed-forward network 310 and a final feed-forward network 340, wherein a multi-head self-attention module (i.e., multi-head self-attention layer) 320 and a convolution module (i.e., convolution layer) 330 are disposed between the initial feed-forward network 310 and the final feed-forward network 340. The example multi-head self-attention layer 300 also includes a layernorm module 350 and concatenation operators 305, 305a-d. In some examples, at least one of the initial feedforward network 310 or the final feedforward network 340 includes a respective mixture of experts (MoE) layer 400. In one configuration, the initial feedforward network 310 or the final feedforward network 340 includes a respective MoE layer 400. In another configuration, both the initial feedforward network 310 and the final feedforward network 340 include a respective MoE layer 400.

[0049] The initial feedforward network 310 is configured to receive the acoustic frame sequence 110 as input and process each acoustic frame 110 corresponding to an utterance in the acoustic frame sequence 110 to generate a first feedforward output 312. In some examples, the initial feedforward network 310 is configured to receive the layer output 352 generated by the immediately preceding multi-head self-attention layer 300 in addition to or instead of the acoustic frame sequence 110. Figure 4 Describing in more detail, when the initial feedforward network 310 includes the corresponding MoE layer 400, the initial feedforward network 310 uses the corresponding MoE layer 400 to process each acoustic frame 110 corresponding to the utterance in the acoustic frame sequence 110 and generates a first feedforward output 312. Next, the first cascade operator 305a concatenates the first feedforward output 312 with the corresponding acoustic frame 110 to generate a first concatenated input 314. Subsequently, the multi-head self-attention module 320 receives the first concatenated input 314 and generates a self-attention output 322 based on the first concatenated input 314. The second cascade operator 305b concatenates the self-attention output 322 with the first concatenated input 314 to generate a second concatenated input 324.

[0050] The convolution module 330 is configured to subsample the second concatenated input 324 and generate a convolution output 332 based on the second concatenated input 324. The third concatenation operator 305c concatenates the convolution output 332 with the second concatenated input 324 to generate a third concatenated input 334. The final feedforward network 340 is configured to receive the third concatenated input 334 as input and generate a second feedforward output 342. Figure 4 Described in more detail, when the final feedforward network 340 includes the corresponding MoE layer 400, the final feedforward network 340 uses the corresponding MoE layer 400 to process the third cascaded input 334 and generate a second feedforward output 342. The fourth concatenation operator 305d concatenates the second feedforward output 342 with the third cascaded input 334 to generate a fourth cascaded input 344. Finally, the layernorm module 350 processes the fourth cascaded input 344 from the final feedforward network 340 to generate a corresponding layer output 352 or a corresponding high-order feature representation 212, 222. Mathematically, the example multi-head self-attention layer 300 uses the modulation feature m to transform the input feature x (e.g., the acoustic frame sequence 110) to produce the output feature y as follows:

[0051] The example multi-headed self-attention layer 300 generates a layer output 352 at each of the plurality of output steps, which is passed to the next multi-headed self-attention layer 300 in the plurality of multi-headed self-attention layers 300. The final multi-headed self-attention layer 300 in the plurality of multi-headed self-attention layers 300 uses the layer output 352 from the immediately preceding multi-headed self-attention layer 300 to generate high-order feature representations 212, 222 for the corresponding acoustic frame 110 in the acoustic frame sequence 110. Thus, each multi-headed self-attention layer 300 before the final multi-headed self-attention layer 300 generates a layer output 352, which is passed to the next multi-headed self-attention layer 300, and the final multi-headed self-attention layer 300 generates high-order feature representations 212, 222. For example, the final multi-headed self-attention layer 300 of the first encoder 210 generates the first high-order feature representation 212, while the final multi-headed self-attention layer 300 of the second encoder 220 generates the second high-order feature representation 222.

[0052] Figure 4 An example MoE layer 400 is shown. The MoE layer 400 can correspond to a corresponding MoE layer 400 from the first plurality of multi-headed attention layers 300a (e.g., from the first encoder 210) and / or the second plurality of multi-headed attention layers 300b (e.g., from the second encoder 220). Each MoE layer 400 includes a gating layer 410, a plurality of feed-forward expert networks 420, 420a-n, and an output layer 430. The gating layer 410 is configured to dynamically route outputs 402 to a predetermined number of feed-forward expert networks 420 from the plurality of feed-forward expert networks 420. Each output 402 can correspond to a corresponding acoustic frame 110 from the sequence of acoustic frames 110, such that the gating layer 410 routes each output to a corresponding predetermined number of feed-forward expert networks 420 associated with the corresponding acoustic frame 110. Each feed-forward expert network 420 includes a corresponding neural network including corresponding parameters. In some examples, each feedforward expert network 420 in the plurality of feedforward expert networks 420 of a corresponding MoE layer comprises a single feedforward network layer. Thus, during training, each corresponding feedforward expert network 420 updates corresponding parameters of the corresponding feedforward expert network 420 to focus on a specific speech recognition task. For example, training the ASR model 200 to recognize speech from multiple different languages ​​causes a subset of the feedforward expert networks 420 to update corresponding parameters specifically for recognizing a particular one of these languages. Thus, each feedforward expert network 420 is trained to process certain speech inputs better than other feedforward expert networks 420.

[0053] In some examples, the gating layer 410 dynamically routes the output 402 to a corresponding pair (i.e., two (2)) of feed-forward expert networks 420 from among the plurality of feed-forward expert networks 420. However, the gating layer 410 can dynamically route the output 402 to any number of feed-forward expert networks 420 from among the plurality of feed-forward expert networks 420. When the predetermined number of feed-forward expert networks 420 is less than the number of the plurality of feed-forward expert networks 420, the MoE layer 400 is sparsely activated. That is, because the gating layer 410 only routes the output 402 to the predetermined number of feed-forward expert networks 420, only the predetermined number of feed-forward expert networks 420 process the output 402, while the other feed-forward expert networks 420 remain idle.

[0054] The output 402 dynamically routed by the gating layer 410 can be combined with the acoustic frame sequence 110, the layer output 352 generated by the previous multi-head self-attention layer 300, and / or the fourth cascaded input 344 ( Figure 3 ). More specifically, when the MoE layer 400 is integrated with the initial feed-forward network 310, the output 402 dynamically routed by the gating layer 410 may correspond to the acoustic frame sequence 110 or the layer output 352 generated by the previous multi-head self-attention layer 300 in the stack 300 of multi-head self-attention layers. Here, the output 402 corresponds to the acoustic frame sequence 110 of the initial multi-head attention layer 300 from the stack 300 of multi-head self-attention layers, and the output 402 corresponds to the layer output 352 of each other multi-head attention layer 300. Alternatively, when the MoE layer 400 is integrated with the final feed-forward network 340, the output dynamically routed by the gating layer 410 may correspond to the fourth cascaded input 344.

[0055] It is noteworthy that the gating layer 410 simply routes the output 402 to a predetermined number of feed-forward expert networks 420. That is, the gating layer 410 does not generate any output based on the received output 402, but rather simply forwards (i.e., routes) the received output 402. More specifically, to dynamically route the output 402, the gating layer 410 uses a softmax activation function to model the probability distribution of weights 412 for each corresponding feed-forward expert network 420 from the plurality of feed-forward expert networks 420 based on the output 402. The probability distribution of the weights 412 indicates to what extent each feed-forward expert network 420 is capable of processing the received output 402. In other words, the probability distribution of the weights 412 indicates which feed-forward expert network(s) 420 are best suited to generate a code for the received output 402. Each corresponding weight 412 from the probability distribution of weights 412 corresponds to one of the feed-forward expert networks 420. In some examples, gating layer 410 determines each respective weight 412 from the probability distribution of weights 412 according to the following equation: In Equation 9, x represents the output 402, W l represents the weight matrix of the gating layer 410 at the corresponding l-th multi-head self-attention layer 300, and g l Represents the corresponding weights 412 of the corresponding feed-forward expert network in the feed-forward expert network 420 .

[0056] In the example shown, the gating layer 410 receives the output 402 and determines a corresponding weight 412 for each respective feed-forward expert network 420, the corresponding weight indicating how well the respective feed-forward expert network 420 processes the output 402. Thereafter, the gating layer 410 is configured to select a predetermined number (i.e., a pair) of feed-forward expert networks 420 that include the highest weights 412 from the probability distribution of the weights 412. Continuing with the example shown, the gating layer 410 selects (e.g., represented by solid lines) the first feed-forward expert network 420, 420a and the third feed-forward expert network 420, 420c as the predetermined number of feed-forward expert networks 420 having the highest weights 412. Thus, the gating layer 410 dynamically routes the output 402 to the first feedforward expert network 420a and the third feedforward expert network 420c, but does not route the output 402 to other feedforward expert networks 420 (e.g., the second feedforward expert network 420, 420b and the nth feedforward expert network 420, 420n (indicated by the dashed lines)) among the plurality of feedforward expert networks 420. Consequently, only the first feedforward expert network 420a and the third feedforward expert network 420c process the output 402 because the gating layer 410 only routes the output 402 to the first feedforward expert network 420a and the third feedforward expert network 420c. Each corresponding feedforward expert network 420 that receives the output 402 dynamically routed by the gating layer 410 processes the output 402 to determine a corresponding output 422. Continuing with the example shown, the first feed-forward expert network 420a generates a first corresponding output 422, 422a and the third feed-forward expert network 420c generates a third corresponding output 422, 422c, while the other feed-forward expert networks 420 do not generate any output.

[0057] Finally, the output layer 430 receives as input the corresponding output 422 generated by each feedforward expert network 420 and generates a corresponding feedforward output 312, 342 at each of the multiple output steps. When the MoE layer 400 is integrated with the initial feedforward network 310, the output layer 430 generates the corresponding first feedforward output 312. On the other hand, when the MoE layer 400 is integrated with the final feedforward network 340, the output layer 430 generates the corresponding second feedforward output 342. The output layer 430 generates the corresponding feedforward output 312, 342 by weighting and summing the corresponding outputs 422. The output layer 430 sums the corresponding outputs 422 to generate the MoE outputs (i.e., the first feedforward output 312 or the second feedforward output 342) 312, 342. In some examples, the output layer 430 sums and weights the corresponding outputs 422 according to the corresponding weights 412 of the feedforward expert networks 420 that generated the corresponding outputs 422. For example, the output layer 430 may generate corresponding feed-forward outputs 312, 342 according to the following equations: In Equation 10, represents the weight 413 of the i-th feedforward expert network 420 at the l-th layer, is the corresponding output 422 of the i-th feedforward expert network 420 at the l-th layer.

[0058] It is noteworthy that in some implementations (e.g., when the ASR model 200 does not include the language ID predictor 230), the gating layer 410 does not rely on any language information associated with the acoustic frame sequence 110 representing the utterance 106 when dynamically routing the output 402. Instead, the gating layer 410 processes the output 402 without receiving or processing any indication of which language is associated with the acoustic frame sequence 110 to determine the probability distribution of weights 412, thereby determining which feedforward expert networks 420 to route the output 402 to. For example, the MoE layer 400 may include 100 feedforward expert networks 420, whereby two corresponding feedforward expert networks 420 are best suited to process Spanish speech input. In this example, the gating layer 410 may process the output 402 corresponding to the Spanish speech input and, based on determining the weights 412 for each feedforward expert network 420, determine to route the output to the two corresponding feedforward expert networks 420 that are best suited to process the Spanish speech input. In short, the gating layer 410 determines that the two corresponding feedforward expert networks 420 that are most suitable for processing Spanish speech input should process the output 402 based on the acoustic information from the acoustic frame sequence 110 without receiving any indication that the output 402 corresponds to Spanish speech. Advantageously, the multiple feedforward expert networks 420 can each be focused on (i.e., specifically trained for) a specific speech recognition task. The specific speech recognition task can be recognizing speech from a certain domain, language, etc. In addition, by routing the output 402 only to a predetermined number of feedforward expert networks 420, the MoE layer minimizes the computational and storage costs for processing a specific speech input.

[0059] In some implementations, each MoE layer 400 is trained on an auxiliary loss to encourage load balancing across the multiple feedforward expert networks 420. Here, load balancing refers to preventing a subset of the multiple feedforward expert networks 420 from having the largest weights 412 for most speech inputs. In other words, load balancing ensures that not a subset of the multiple feedforward expert networks 420 has the largest weights 412 for all of the speech recognition tasks. The auxiliary loss can be represented by the following equation: In Equation 11, represents the average number of times the i-th feedforward expert network 420 is selected over all acoustic frames 110 (i.e., the average gate), and is the expert decision count of the i-th expert derived from a predetermined number of feed-forward expert networks 420. Therefore, the auxiliary loss is based on the average gate over all frames of each corresponding feed-forward expert network and a portion of the output from the previous multi-head attention layer that the gating layer 410 routes to each corresponding feed-forward expert network 420. Figure 5is a flow chart of an example operational arrangement of a computer-implemented method 500 for executing a streaming end-to-end multilingual ASR model with a mixture-of-experts layer. The method 500 may be implemented using a memory device stored in memory hardware 620 ( Figure 6 ) on the data processing hardware 610 ( Figure 6 ) is executed on the . Data processing hardware 610 and memory hardware 620 may reside on Figure 1 Each of the computing devices 600 ( Figure 6 ) on the corresponding user device 10 and / or remote computing device 60.

[0060] At operation 502, method 500 includes receiving an acoustic frame sequence 110 representing an utterance 106 of speech. At each output step of a plurality of output steps, method 500 performs operations 504 to 508. At operation 504, method 500 includes generating, by a first encoder 210 of a multilingual ASR model 200, a first high-order feature representation 212 for a corresponding acoustic frame 110 in the acoustic frame sequence 110. Here, the first encoder 210 includes a first plurality of multi-headed attention layers 300a. At operation 506, method 500 includes generating, by a second encoder 220 of the multilingual ASR model 200, a second high-order feature representation 222 for the corresponding first high-order feature representation 212. Here, the second encoder 220 includes a second plurality of multi-headed attention layers 300b. At operation 508, method 500 includes generating, by a first decoder 240a of the multilingual ASR model 200, a first probability distribution 120a regarding possible speech recognition hypotheses. Here, generating a first probability distribution 120a about possible speech recognition hypotheses is based on the first high-order feature representation 212 generated by the first encoder 210 at the corresponding output step and the sequence of N previous non-blank symbols output by the final softmax layer at the corresponding output step. Each multi-headed attention layer 300 in the first plurality of multi-headed attention layers 300a and the second plurality of multi-headed attention layers 300b includes an initial feedforward network 310, a multi-headed self-attention layer 320, a convolutional layer 330, and a final feedforward network 340. At least one of the initial feedforward network 310 or the final feedforward network 340 of at least one corresponding multi-headed attention layer 300 in the second plurality of multi-headed attention layers 300b includes a corresponding MoE layer 400. In addition, each corresponding MoE layer 400 includes a gating layer 410 and a plurality of feedforward expert networks 420. The gating layer 410 of each corresponding MoE layer 400 is configured to dynamically route the output 402 from the previous multi-head attention layer 300 to a corresponding pair of feedforward expert networks 420 including the highest weight 412 among the multiple feedforward expert networks 420 at the corresponding multi-head attention layer 300 in the second multiple multi-head attention layers 300b at each output step in the multiple output steps, without routing the output 402 to other feedforward expert networks 420 among the multiple feedforward expert networks 420.

[0061] Figure 6 6 is a schematic diagram of an example computing device 600 that can be used to implement the systems and methods described in this document. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0062] Computing device 600 includes a processor 610, memory 620, storage device 630, a high-speed interface / controller 640 connected to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 connected to a low-speed bus 670 and storage device 630. Each of components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and can be mounted on a common motherboard or otherwise as appropriate. Processor 610 can process instructions for execution within computing device 600, including instructions stored in memory 620 or on storage device 630, to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 680 coupled to high-speed interface 640. In other implementations, multiple processors and / or multiple buses, as well as multiple memories and multiple types of memory, can be used as appropriate. In addition, multiple computing devices 600 can be connected, with each device providing a portion of the necessary operations (e.g., as a server bank, a blade server group, or a multi-processor system).

[0063] Memory 620 stores information non-temporarily within computing device 600. Memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporary memory 620 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0064] The storage device 630 can provide mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various implementations, the storage device 630 can be a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0065] The high-speed controller 640 manages bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages less bandwidth-intensive operations. This division of responsibilities is exemplary only. In some implementations, the high-speed controller 640 is coupled to the memory 620, the display 680 (e.g., via a graphics processor or accelerator), and the high-speed expansion ports 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and the low-speed expansion ports 690. The low-speed expansion ports 690, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, or networking devices, such as a switch or router, for example, via a network adapter.

[0066] Computing device 600 can be implemented in many different forms, as shown. For example, it can be implemented as a standard server 600a or multiple times as a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0067] Various implementations of the systems and techniques described herein can be realized using digital electronic and / or optical circuitry, integrated circuitry, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor, which can be either special purpose or general purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0068] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., a magnetic disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0069] The processes and logic flows described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware) that execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows can also be performed by dedicated logic circuit systems, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits). For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from a read-only memory or random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operably coupled to receive data from the one or more mass storage devices or transfer data to the one or more mass storage devices or both. However, a computer does not necessarily have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0070] To provide for interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touch screen) for displaying information to the user, and possibly a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0071] A number of implementations have been described. However, it will be appreciated that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims.

Claims

1. A multilingual automatic speech recognition (ASR) model (200), comprising: A first encoder (210), comprising a first plurality of multi-head attention layers (300a), wherein the first encoder (210) is configured to: receiving as input a sequence of acoustic frames (110) representing a spoken utterance (106); and At each of a plurality of output steps, generating a first high-order feature representation (212) for a corresponding acoustic frame (110) in the sequence of acoustic frames (110); A second encoder (220), cascaded to the first encoder (210) and comprising a second plurality of multi-head attention layers (300b), the second encoder (220) being configured to: receiving as input the first high-order feature representation (212) generated by the first encoder (210) at each of the plurality of output steps; and At each output step in the plurality of output steps, generating a second high-order feature representation (222) for the corresponding first high-order feature representation (212); and A first decoder (240a), the first decoder being configured to: receiving as input the first high-order feature representation (212) generated by the first encoder (210) at a corresponding output step and a sequence of N previous non-whitespace symbols output by a final softmax layer at each corresponding output in the corresponding output step; and At each of the plurality of output steps, generating a first probability distribution (120a) over possible speech recognition hypotheses, in: Each multi-head attention layer (300) in the first plurality of multi-head attention layers (300a) and the second plurality of multi-head attention layers (300b) comprises an initial feedforward network (310), a multi-head self-attention layer (320), a convolutional layer (330), and a final feedforward network (340); At least one of the initial feedforward network (310) or the final feedforward network (340) of at least one of the second plurality of multi-head attention layers (300b) corresponding to the multi-head attention layer (300) includes a corresponding mixture of experts (MoE) layer (400), each corresponding MoE layer (400) including a gating layer (410) and a plurality of feedforward expert networks (420); and The gating layer (410) of each corresponding MoE layer (400) is configured to dynamically route the output (402) from the previous multi-head attention layer (400) at each output step in the plurality of output steps to a corresponding pair of feedforward expert networks (420) among the plurality of feedforward expert networks (420) including the highest weight (412) among the plurality of feedforward expert networks (420) at the corresponding multi-head attention layer (300) in the second plurality of multi-head attention layers (300b), without routing the output (402) to other feedforward expert networks (420) among the plurality of feedforward expert networks (420).

2. The multilingual ASR model (200) of claim 1, wherein: Each initial feed-forward network (310) in the second plurality of multi-head attention layers (300b) includes the corresponding MoE layer (400).

3. The multilingual ASR model (200) according to claim 1 or 2, wherein: Each final feed-forward network (340) in the second plurality of multi-head attention layers (300b) comprises the corresponding MoE layer (400).

4. The multilingual ASR model (200) according to any one of claims 1 to 3, wherein: Each feed-forward expert network (420) among the plurality of feed-forward expert networks (420) of the corresponding MoE layer (400) includes a single feed-forward network layer.

5. The multilingual ASR model (200) according to any one of claims 1 to 4, wherein: When dynamically routing the output (402) from the previous multi-head attention layer (300) to the corresponding pair of feed-forward expert networks (420) at each of the plurality of output steps, the gating layer (410) of the corresponding MoE layer (400) does not rely on any linguistic information associated with the sequence of audio frames (110) representing the utterance (106).

6. The multilingual ASR model (200) according to any one of claims 1 to 5, wherein: The gating layer (410) of the respective MoE layer (400) is configured to determine, at each output step of the plurality of output steps, a respective weight (412) of each corresponding feed-forward expert network (420) among the plurality of feed-forward expert networks (420) based on: Where x comprises the output (402) of the previous layer at the corresponding output step, W l The weight matrix of the gating layer (410) at the corresponding l-th multi-head attention layer (300) is included, and g l The respective weights (412) of the corresponding feed-forward expert network (420) are included.

7. The multilingual ASR model (200) according to any one of claims 1 to 6, wherein: each feed-forward expert network (420) of the pair of feed-forward expert networks (420) including the highest weight (412) is configured to determine a corresponding output (422) based on the output (402) from the previous multi-head attention layer (300) routed by the gating layer (410); and The respective MoE layer (400) is configured to determine a MoE output (312, 342) based on a sum of the respective outputs (422) determined by the pair of feedforward expert networks (420) including the highest weight (412).

8. The multilingual ASR model (200) according to any one of claims 1 to 7, further comprising: A second decoder (240b), the second decoder being configured to: receiving as input the second high-order feature representation (222) generated by the second encoder (220) at each of the plurality of output steps; and At each of the plurality of output steps, a second probability distribution over possible speech recognition hypotheses is generated (120b).

9. The multilingual ASR model (200) of claim 8, wherein: The second decoder (240b) is further configured to generate a partial speech recognition result based on the second probability distribution (120b) over possible speech recognition hypotheses.

10. The multilingual ASR model (200) according to claim 8 or 9, wherein: The first decoder (240a) and the second decoder (240b) each include a corresponding prediction network (260) followed by a corresponding joint network (250); The corresponding prediction networks (260) of the first decoder (240a) and the second decoder (240b) have the same structure including one of the following: Prediction network based on long short-term memory (LSTM); or V2 embeds a lookup table; and The corresponding joint networks (250) of the first decoder (240a) and the second decoder (240b) comprise the same structure.

11. The multilingual ASR model (200) according to any one of claims 1 to 10, wherein: The second encoder (220) generates the second high-order feature representation (222) without receiving any of the acoustic frames (110) as input.

12. The multilingual ASR model (200) according to any one of claims 1 to 11, wherein: The first encoder (210) comprises a causal encoder.

13. The multilingual ASR model (200) according to any one of claims 1 to 12, wherein: The second encoder (220) comprises a non-causal encoder.

14. The multilingual ASR model (200) according to any one of claims 1 to 13, wherein: The first encoder (210) and the second encoder (220) are jointly trained on a set of multilingual training utterances using negative log-likelihood.

15. The multilingual ASR model (200) according to any one of claims 1 to 14, wherein: The respective MoE layers (400) are trained on an auxiliary loss to encourage load balancing across the plurality of feed-forward expert networks (420); and The auxiliary loss is based on the average gate over all frames of each corresponding feed-forward expert network (420) and a portion of the output from the previous multi-head attention layer (300) that the gating layer (410) routes to each corresponding feed-forward expert network (420).

16. A computer-implemented method (500) which, when executed on data processing hardware (610), causes the data processing hardware (610) to perform operations comprising: receiving a sequence of acoustic frames (110) representing a speech utterance (106); At each of multiple output steps: generating, by a first encoder (210) of a multilingual automatic speech recognition (ASR) model (200), a first high-order feature representation (212) for a corresponding acoustic frame (110) in the acoustic frame sequence (110), wherein the first encoder (210) includes a first plurality of multi-head attention layers (300a); generating a second high-order feature representation (222) for the corresponding first high-order feature representation (212) by a second encoder (220) of the multilingual ASR model (200), wherein the second encoder (220) includes a second plurality of multi-head attention layers (300b); and generating, by a first decoder (240ab) of the multilingual ASR model (200), a first probability distribution (120a) over possible speech recognition hypotheses based on the first high-order feature representation (212) generated by the first encoder (210) at a corresponding output step and a sequence of N previous non-whitespace symbols output by a final softmax layer at the corresponding output step, in: Each multi-head attention layer (300) in the first plurality of multi-head attention layers (300a) and the second plurality of multi-head attention layers (300b) comprises an initial feedforward network (310), a multi-head self-attention layer (320), a convolutional layer (330), and a final feedforward network (340); At least one of the initial feedforward network (310) or the final feedforward network (340) of at least one of the second plurality of multi-head attention layers (300b) corresponding to the multi-head attention layer (300) includes a corresponding mixture of experts (MoE) layer (400), each corresponding MoE layer (400) including a gating layer (410) and a plurality of feedforward expert networks (420); and The gating layer (410) of each corresponding MoE layer (400) is configured to dynamically route the output (402) from the previous multi-head attention layer (300) at each output step in the plurality of output steps to a corresponding pair of feedforward expert networks (420) including the highest weight (412) among the plurality of feedforward expert networks (412) at the corresponding multi-head attention layer (300) in the second plurality of multi-head attention layers (300b) among the plurality of feedforward expert networks (420), without routing the output (402) to other feedforward expert networks (420) among the plurality of feedforward expert networks (420).

17. The computer-implemented method (500) of claim 16, wherein: Each initial feed-forward network (310) in the second plurality of multi-head attention layers (300b) includes the corresponding MoE layer (400).

18. The computer-implemented method (500) of claim 16 or 17, wherein: Each final feed-forward network (340) in the second plurality of multi-head attention layers (300b) comprises the corresponding MoE layer (400).

19. The computer-implemented method (500) of any one of claims 16 to 18, wherein: Each feed-forward expert network (420) among the plurality of feed-forward expert networks (420) of the corresponding MoE layer (400) includes a single feed-forward network layer.

20. The computer-implemented method (500) of any one of claims 16 to 19, wherein: When dynamically routing the output (402) from the previous multi-head attention layer (300) to the corresponding pair of feed-forward expert networks (420) at each of the plurality of output steps, the gating layer (410) of the corresponding MoE layer (400) does not rely on any linguistic information associated with the sequence of audio frames (110) representing the utterance (106).

21. The computer-implemented method (500) of any one of claims 16 to 20, wherein the operations further comprise: Determining, by the gating layer (410) of the corresponding MoE layer (400), at each of the plurality of output steps, a corresponding weight (412) of each corresponding feedforward expert network (420) among the plurality of feedforward expert networks (420) based on: Where x comprises the output (402) of the previous layer at the corresponding output step, W l The weight matrix of the gating layer (410) at the corresponding l-th multi-head attention layer (300) is included, and g l The respective weights (412) of the corresponding feed-forward expert network (420) are included.

22. The computer-implemented method (500) of any one of claims 16 to 21, wherein: The operations further include: determining, by each of the pair of feed-forward expert networks (420) including the highest weight (412), a corresponding output (422) based on the output (402) routed from the previous multi-head attention layer (300) by the gating layer (410); and An MoE output (312, 342) is determined by the respective MoE layer (400) based on a sum of the respective outputs (422) determined by the pair of feedforward expert networks (422) including the highest weight (412).

23. The computer-implemented method (500) of any one of claims 16 to 22, wherein: The operations further include: receiving the second high-order feature representation (222) generated by the second encoder (220) at each of the plurality of output steps as input to a second decoder (240b) of the multilingual ASR model (200); and A second probability distribution (120b) over possible speech recognition hypotheses is generated by the second decoder (240b) at each of the plurality of output steps.

24. The computer-implemented method (500) of claim 23, wherein: The operations further include generating, by the second decoder (240b), a partial speech recognition result based on the second probability distribution (120b) regarding possible speech recognition hypotheses.

25. The computer-implemented method (500) of claim 23 or 24, wherein: The first decoder (240a) and the second decoder (240b) each include a corresponding prediction network (260) followed by a corresponding joint network (250); The corresponding prediction networks (260) of the first decoder (240a) and the second decoder (240b) have the same structure including one of the following: Prediction network based on long short-term memory (LSTM); or V2 embeds a lookup table; and The corresponding joint networks (250) of the first decoder (240a) and the second decoder (240b) comprise the same structure.

26. The computer-implemented method (500) of any one of claims 16 to 25, wherein: Generating the second high-order feature representation (222) includes generating the second high-order feature representation (222) without receiving any of the acoustic frames (110) as input.

27. The computer-implemented method (500) of any one of claims 16 to 26, wherein: The first encoder (210) comprises a causal encoder.

28. The computer-implemented method (500) of any one of claims 16 to 27, wherein: The second encoder (220) comprises a non-causal encoder.

29. The computer-implemented method (500) of any one of claims 16 to 28, wherein: The operations further include jointly training the first encoder (210) and the second encoder (220) on a set of multilingual training utterances using negative log-likelihood.

30. The computer-implemented method (500) of any one of claims 16 to 29, wherein: The operations further include training the corresponding MoE layer (400) on an auxiliary loss based on an average gate over all frames of each corresponding feed-forward expert network (420) and a portion of the output from the previous multi-head attention layer (300) that the gating layer (410) routes to each corresponding feed-forward expert network (420) to encourage load balancing across the plurality of feed-forward expert networks (420).

Images