Streaming end-to-end multilingual speech recognition using joint language identification

Abstract

The method (400) includes receiving a sequence of acoustic frames (110) as an input to an automatic speech recognition (ASR) model (200). The method also includes generating, by a first encoder (210), a first high-level feature representation (212) for the corresponding acoustic frame. The method also includes generating, by a second encoder (220), a second high-level feature representation (222) for the corresponding first high-level feature representation. The method also includes generating, by a language identification (ID) predictor (230), a language prediction representation (232) based on a combination (231) of the first high-level feature representation and the second high-level feature representation. The method also includes generating, by a first decoder (240a), a first probability distribution (120a) over possible speech recognition hypotheses based on a combination of the second high-level feature representation and the language prediction representation.

Description

[Technical Field]

[0001] The present disclosure relates to streaming end-to-end multilingual speech recognition with collaborative language identification. [Background technology]

[0002] Automatic speech recognition (ASR), the process of taking audio input and transcribing it into text, has become a critical technology used in mobile and other devices. Generally, automatic speech recognition attempts to provide an accurate transcription of what a person said by taking audio input (e.g., a spoken utterance) and transcribing that audio input into text. Modern ASR models, based on the ongoing development of deep neural networks, continually improve both accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay between the client's speech and the transcription). Despite the vast number of people being bilingual, most ASR models are monolingual. Therefore, an ASR model that is compatible with several different languages ​​while maintaining the accuracy and latency performance metrics of modern ASR models is desirable for a vast number of bilingual speakers. Summary of the Invention

[0003] One aspect of the present disclosure provides a multilingual automatic speech recognition (ASR) model. The ASR model includes a first encoder configured to receive as input a sequence of acoustic frames and generate, in each of a plurality of output steps, a first high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The ASR model also includes a second encoder configured to receive as input the first high-order feature representation generated by the first encoder in each of the plurality of output steps and generate, in each of the plurality of output steps, a second high-order feature representation for the corresponding first high-order feature representation. The ASR model also includes a language identification (ID) predictor configured to receive as input a combination of the first high-order feature representation generated by the first encoder in each of the plurality of output steps and the second high-order feature representation generated by the second encoder in each of the plurality of output steps and generate, in each of the plurality of output steps, a language predicted representation. The ASR model also includes a first decoder configured to receive as input a combination of the second high-level feature representation generated by the second encoder in each of the plurality of output steps and the language prediction representation generated by the language ID predictor in each of the plurality of output steps, and to generate a first probability distribution over the possible speech recognition hypotheses in each of the plurality of output steps.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the ASR model also includes a second decoder configured to receive as input the first high-order feature representation generated by the first encoder in each of the plurality of output steps and to generate a second probability distribution over the possible speech recognition hypotheses in each of the plurality of output steps. Here, the second decoder may be further configured to generate a partial speech recognition result based on the second probability distribution over the possible speech recognition hypotheses. In these implementations, the first decoder and the second decoder may each include a corresponding prediction network followed by a corresponding collaborative network, wherein the corresponding prediction networks of the first decoder and the second decoder have the same structure, including one of a long short-term memory (LSTM)-based prediction network or a V2 embedded lookup table, and the corresponding collaborative networks of the first decoder and the second decoder include the same structure.

[0005] In some examples, the second encoder generates the second high-level feature representation without receiving any acoustic frames as input. The first encoder includes a causal encoder having one of multiple unidirectional long short-term memory (LSTM) layers, multiple conformer layers, or multiple transformer layers. In some implementations, the second encoder includes a non-causal encoder having one or more bidirectional long short-term memory (LSTM) layers, multiple conformer layers, or multiple transformer layers.

[0006] In some implementations, the first encoder, the second encoder, and the language ID predictor are jointly trained on the set of multilingual training utterances by generating a first loss for the first encoder, a second loss for the second encoder, a third loss for the language ID predictor, and minimizing a weighted sum of the first loss, the second loss, and the third loss. In these implementations, a language ID target token may be added as a first token to a corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances. The language ID target token identifies the language of the corresponding multilingual training utterance. Alternatively, a language ID target token may be added at each position where a code switch occurs in the corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances.

[0007] Another aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for streaming end-to-end multilingual speech recognition with collaborative language recognition. The operations include receiving a sequence of acoustic frames as input to an automatic speech recognition (ASR) model. The operations also include, in each of a plurality of output steps, generating a first high-level feature representation by a first encoder of the ASR model for a corresponding acoustic frame in the sequence of acoustic frames. The operations also include, in each of the plurality of output steps, generating a second high-level feature representation by a second encoder of the ASR model for the corresponding first high-level feature representation. The operations also include, in each of the plurality of output steps, generating a language predicted representation by a language identification (ID) predictor of the ASR model. Here, the language predicted representation is based on a combination of the first high-level feature representation generated by the first encoder in each of the plurality of output steps and the second high-level feature representation generated by the second encoder in each of the plurality of output steps. The operations also include generating, at each of the plurality of output steps, a first probability distribution over the possible speech recognition hypotheses by a first decoder of the ASR model, where the first probability distribution over the possible speech recognition hypotheses is based on a combination of the second higher-order feature representation generated by the second encoder at each of the plurality of output steps and the language prediction representation generated by the language ID predictor at each of the plurality of output steps.

[0008] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operations further include, at each of the plurality of output steps, generating, by a second decoder of the ASR model, a second probability distribution over the possible speech recognition hypotheses. The second probability distribution over the possible speech recognition hypotheses is based on the first high-order feature representation generated by the first encoder at each of the plurality of output steps. In these implementations, the operations may further include generating, by the second decoder, a partial speech recognition result based on the second probability distribution over the possible speech recognition hypotheses. The first and second decoders may each include a corresponding prediction network followed by a corresponding collaborative network, the corresponding prediction networks of the first and second decoders having the same structure and including one of a long-term short-term (LSTM)-based prediction network or a V2 embedded lookup table, and the corresponding collaborative networks of the first and second decoders including the same structure.

[0009] In some examples, the second encoder generates the second high-level feature representation without receiving any acoustic frames as input. In some examples, the first encoder includes a causal encoder including one of multiple one-way long short-term memory (LSTM) layers, multiple conformer layers, or multiple transformer layers. The second encoder includes a non-causal encoder including one of one or more bidirectional long short-term memory (LSTM) layers, multiple conformer layers, or multiple transformer layers.

[0010] In some examples, the operations further include jointly training the first encoder, the second encoder, and the language ID predictor on the set of multilingual training utterances by generating a first loss for the encoder, generating a second loss for the second encoder, generating a third loss for the language ID predictor, and minimizing a weighted sum of the first loss, the second loss, and the third loss. In these examples, the operations may further include adding a language ID token as a first token of a corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances. The language ID target token identifies the language of the corresponding multilingual training utterance. Alternatively, the operations further include adding a language ID target token at each position where a code switch occurs in the corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances.

[0011] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of an exemplary speech recognition system. [Figure 2A] FIG. 1 is a schematic diagram of an exemplary speech recognition model. [Figure 2B] FIG. 1 is a schematic diagram of an exemplary speech recognition model. [Figure 3] FIG. 3 is a schematic diagram of an example prediction network of the example speech recognition model of FIGS. 2A and 2B. [Figure 4] 1 is a flowchart of an exemplary sequence of operations for a computer-implemented method for streaming end-to-end multilingual speech recognition with joint language identification. [Figure 5]1 is a flowchart of an exemplary sequence of operations for a computer-implemented method for streaming end-to-end multilingual speech recognition with joint language identification. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0013] Like reference numbers refer to like elements in the various drawings.

[0014] End-to-end (E2E) automatic speech recognition (ASR) models are traditionally structured to operate in either streaming or non-streaming modes. Traditionally, E2E ASR models include an encoder and a decoder as their main components. Applications involving end-user interaction, such as voice search or on-device listening, may require the model to perform recognition in a streaming manner. Here, performing recognition in a streaming manner refers to the ASR model outputting each word of an utterance as it is spoken with as little latency as possible. Other applications, such as offline video captioning, do not require the model to be streamed and can utilize future context to improve performance.

[0015] In some implementations, the E2E ASR model is configured to recognize speech from multiple languages ​​(e.g., an E2E multilingual ASR model). Here, the multilingual ASR model and the language identification model are often used together in downstream tasks (e.g., code-switching and speech conversion), but the models are constructed and run separately. Thus, using both independent multilingual ASR models and independent language identification models unnecessarily complicates the overall ASR system by increasing computational and storage costs.

[0016] Accordingly, embodiments herein are directed to an E2E multilingual ASR model that jointly generates a transcription and a language prediction representation. The ASR model includes a first encoder configured to generate a first high-level feature representation of a corresponding acoustic frame and a second encoder configured to generate a second high-level feature representation of the corresponding first high-level feature representation. The ASR model also includes a language identification predictor configured to receive a concatenation of the first high-level feature representation and the second high-level feature representation and generate a language prediction representation. The ASR model also includes a first decoder configured to receive the concatenation of the second high-level feature representation and the language prediction representation and generate a first probability distribution over possible speech recognition hypotheses (e.g., transcriptions). Thus, the ASR model (e.g., a single E2E model) outputs both a transcription and a language prediction representation. Notably, the ASR model does not receive or use any oracle language identification information to generate the transcription and the language prediction representation. As will become apparent, the ASR model operates in either streaming or non-streaming mode to generate transcriptions and language prediction representations.

[0017] 1 is an example of an audio environment 100. In the audio environment 100, a user 104's way of interacting with a computing device, such as a user device 10, may be via voice input. The user device 10 (also generally referred to as device 10) is configured to capture sounds (e.g., streaming audio data) from one or more users 104 in the audio environment 100. Here, streaming audio data may refer to spoken utterances 106 by the users 104 that function as audible queries, commands to the user device 10, or audible communications captured by the device 10. A voice-enabled system of the user device 10 may process the queries or commands by responding to the queries and / or causing the commands to be executed / performed by one or more downstream applications.

[0018] The user device 10 may correspond to any computing device associated with a user 104 and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, a mobile device (e.g., a smart watch), a smart appliance, an Internet of Things (IoT) device, an in-vehicle infotainment system, a smart display, a smart speaker, etc. The user device 10 includes data processing hardware 12 and storage hardware 14 in communication with the data processing hardware 12, storing instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 comprising audio capture devices (e.g., microphones) 16, 16a for capturing and converting spoken speech 106 into electrical signals, and audio output devices (e.g., speakers) 16, 16b for communicating audible audio signals (e.g., as output data from the user device 10) within the audio environment 100. The user device 10 may implement an array of audio capture devices 16a without departing from the scope of this disclosure, whereby one or more capture devices 16a in the array may not be physically present at the user device 10 but may communicate with the audio system 16.

[0019] In the voice environment 100, an automatic speech recognition (ASR) system 118 implements an ASR model 200 and resides on the user device 10 of the user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system running in a cloud computing environment) that communicates with the user device 10 via the network 40. In some examples, the ASR model 200 may be a recurrent neural network-transducer (RNN-T) model. The user device 10 and / or the remote computing device 60 also include an audio subsystem 108 configured to receive utterances 106 spoken by the user 104 and captured by the audio capture device 16a and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the ASR system 118. In the illustrated example, the user speaks each utterance 106, and the audio subsystem 108 converts the utterances 106 into corresponding audio data (e.g., a sequence of acoustic frames) 110 for input to the ASR system 118. The ASR model 200 then receives as input a sequence of acoustic frames 110 corresponding to the utterance 106, and in each output step, generates / predicts a corresponding transcription 120 (e.g., a speech recognition result / hypothesis) of the utterance 106 as the ASR model receives (e.g., processes) each acoustic frame 110 in the sequence of acoustic frames 110.

[0020] In the illustrated example, the ASR model 200 may perform streaming speech recognition to generate an initial speech recognition result 120, 120b and then generate a final speech recognition result 120, 120a by improving the initial speech recognition result 120b. The speech recognition result 120 may correspond to either a partial speech recognition result or an entire speech recognition result. In other words, the speech recognition result 120 may correspond to either a portion of the utterance 106 or the entire utterance 106. For example, a partial speech recognition result may correspond to a portion of a spoken utterance or even a portion of a spoken term. However, as will become apparent, the ASR model 200 performs additional processing on the final speech recognition result 120a, which may cause the final speech recognition result 120a to lag behind the initial speech recognition result 120b.

[0021] The user device 10 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of the transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 can display the initial speech recognition result 120b in a streaming format during time 1, and then display the final speech recognition result 120a in a streaming format during time 2. Notably, the ASR model 200 outputs the final speech recognition result 120a in a streaming format even if the final speech recognition result 120a improves on the initial speech recognition result 120b. In some configurations, the transcription 120 output from the ASR system 118 is processed by, for example, a natural language understanding (NLU) module executing on the user device 10 or the remote computing device 60 to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., running on any combination of the user device 10 or the remote computing device 60) can convert the transcription 120 into synthesized speech for audible output by the user device 10 and / or other devices.

[0022] In the illustrated example, a user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) on a user device 10 that uses an ASR system 118. For example, FIG. 1 shows a user 104 communicating with the digital assistant application 50 and the digital assistant application 50 displaying a digital assistant interface 18 on the screen of the user device 10 to show a conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, "What time is the concert today?" This question from the user 104 is a spoken utterance 106 that is captured by an audio capture device 16a and processed by an audio system 16 of the user device 10. In this example, the audio system 16 receives the spoken utterance 106 and converts it into a sequence of acoustic frames 110 for input to the ASR system 118.

[0023] Continuing with this example, as the ASR model 200 receives a sequence of acoustic frames 110 corresponding to an utterance 106 as the user 104 speaks, it encodes the sequence of acoustic frames 110 and then decodes the encoded sequence of acoustic frames 110 into initial speech recognition results 120b. During time 1, the user interface generator 107 presents, via the digital assistant interface 18, a representation of the initial speech recognition results 120b of the utterance 106 to the user 104 of the user device 10 in a streaming format, with words, word fragments, and / or individual characters appearing on the screen as they are spoken. In some examples, the first look-ahead audio context is equal to zero.

[0024] During time 2, user interface generator 107 presents a representation of final speech recognition result 120a of utterance 106 to user 104 of user device 10 via digital assistant interface 18 in a streaming format, such that words, word fragments, and / or individual characters are displayed on the screen as soon as they are generated by ASR model 200. In some implementations, user interface generator 107 replaces the representation of initial speech recognition result 120b presented at time 1 with the representation of final speech recognition result 120a presented at time 2, where time 1 and time 2 may include timestamps corresponding to when user interface generator 107 presents each speech recognition result 120. In this example, the timestamp for time 1 indicates that user interface generator 107 presents initial speech recognition result 120b at an earlier time than final speech recognition result 120a. For example, because the final speech recognition result 120a is estimated to be more accurate than the initial speech recognition result 120b, the final speech recognition result 120a, which is ultimately displayed as the transcription 120, can correct any terms that may have been misrecognized in the initial speech recognition result 120b. In this example, the streaming of the initial speech recognition result 120b output by the ASR model 200 is displayed on the screen of the user device 10 at time 1 and associated with low latency, providing the user 104 with a sense of responsiveness that the user's query is being processed, while the final speech recognition result 120a output by the ASR model 200 and displayed on the screen at time 2 utilizes additional speech recognition and / or language models to improve speech recognition quality in terms of accuracy, albeit with increased latency. However, because the initial speech recognition result 120b is displayed as the user speaks the utterance 106, the higher latency associated with generating and ultimately displaying the final speech recognition result 120a is not noticeable to the user 104.

[0025] In the example shown in FIG. 1 , the digital assistant application 50 can respond to a question posed by the user 104 using natural language processing. Natural language processing generally refers to the process of interpreting written language (e.g., initial speech recognition result 120b and / or final speech recognition result 120a) and determining whether the written language prompts any action. In this example, the digital assistant application 50 uses natural language processing to recognize that the question from the user 104 concerns the user's schedule, more specifically, a concert on the user's schedule. By recognizing these details using natural language processing, the automated assistant returns a response 19 to the user's query, where response 19 states, "The venue doors open at 6:30 PM, and the concert starts at 8:00 PM." In some configurations, natural language processing occurs on a remote server 60 in communication with the data processing hardware 12 of the user device 10.

[0026] 2A and 2B, in some implementations, an ASR model 200 includes a cascade encoder 204, a language identification (ID) predictor 230, and a decoder 240. Here, a first decoder 240, 240a is configured to improve an initial speech recognition result 120b output by a second decoder 240, 240b. The second decoder 240b may operate in a streaming manner, such that the second decoder 240b is configured to generate a partial speech recognition result corresponding to the initial speech recognition result 120b. Meanwhile, the first decoder 240a improves the partial speech recognition result by receiving additional correct context to generate a final speech recognition result 120a. Each of the first decoder 240a and the second decoder 240b includes a corresponding prediction network 300 followed by a corresponding joint network 250. Here, the first prediction network 300, 300a and the first collaborative network 250, 250a correspond to the first decoder 240a, and the second prediction network 300, 300b and the second collaborative network 250, 250b correspond to the second decoder 240b. The prediction networks 300a, 300b have the same structure, including one of a long short-term memory (LSTM)-based prediction network or a V2 embedded lookup table. Furthermore, the corresponding collaborative networks 250a, 250b have the same structure. Although the component structure is the same for the first decoder 240a and the second decoder 240b, each component of each decoder 240 is unique and can be trained independently of the components of the other decoders 240.

[0027] The cascaded encoder 204 refers to a model structure in which an encoding path includes two encoders 210, 220, cascaded together such that the output of the first encoder 210 provides the input of the second encoder 220 before decoding. Here, the first encoder 210 and the second encoder 220 may be cascaded regardless of the underlying architecture of each encoder. The encoders 210, 220 may each include a stack of multi-head attention layers. In some examples, the first encoder 210 includes a causal encoder with multiple unidirectional (LSTM) layers, multiple conformer layers, or multiple transformer layers. For example, the first encoder 210 may include 12 conformer layers, each with a multi-head (e.g., 8-head) self-attention mechanism and a convolution kernel size of 15. Furthermore, the first encoder 210 can perform a combining operation after the third conformer layer achieves a time reduction factor of 2, whereby the resulting 1024-dimensional vector is transformed by the fourth conformer layer and then backprojected to a 512-dimensional vector using another linear transformation. This is followed by eight other conformer layers and a final normalization layer. Thus, the first encoder 210 may include 110 million parameters. Each layer of the first encoder 210 receives zero correct context (e.g., zero future acoustic frames).

[0028] The second encoder 220 may include an acausal encoder having one or more bidirectional LSTM layers, multiple conformer layers, or multiple transformer layers. For example, the second encoder 220 may include a 512-dimensional linear projection to transform input features, followed by five 512-dimensional conformer layers and a final linear normalization layer, resulting in 50 million parameters. Here, the second encoder 220 may receive additional correct contexts, e.g., a total of 15 frames of correct contexts, such that each conformer layer receives three frames of correct contexts.

[0029] 2A and 2B, a first encoder 210 receives a sequence of d-dimensional feature vectors (e.g., a sequence of acoustic frames 110) x=(x1, x2, ..., x T )(in the formula,

number

[0030] 2A and 2B, the second decoder 240b includes a second collaborative network 250b and a second prediction network 300b. The second collaborative network 250b is configured to receive as input the dense representation 350 generated by the second prediction network 300b and the first high-level feature representation 212 generated by the first encoder 210, and to generate, at each output step, an initial speech recognition result 120b for the corresponding acoustic frame 110. Here, the second collaborative network 250b uses the first high-level feature representation 212 and the dense representation 350 to generate the initial speech recognition result 120b. The second decoder 240b operates in a streaming manner, such that the initial speech recognition result 120b may correspond to a partial speech recognition result.

[0031] In some implementations, the initial speech recognition result 120b includes a second probability distribution over the possible speech recognition hypotheses. In this manner, the initial speech recognition result 120b may be used interchangeably herein as the second probability distribution over the possible speech recognition hypotheses 120b. Accordingly, the second collaborative network 250b can generate the second probability distribution over the possible speech recognition hypotheses 120b at each output step (e.g., time step). Here, a "possible speech recognition hypothesis" corresponds to a set of output labels / symbols (also called "phonetic units"), each representing a grapheme (symbol / character) or word fragment of a specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters of the English alphabet and one label representing a space. Accordingly, the second collaborative network 250b may output a set of values ​​indicating the likelihood of occurrence of each of the predetermined set of output labels. This set of values ​​may be a vector (e.g., a one-hot vector) and may indicate the second probability distribution over the set of output labels. In some scenarios, the output labels are graphemes (e.g., individual characters, potential punctuation marks, or other symbols), but the set of output labels is not so limited. For example, the set of output labels can include word fragments and / or entire words in addition to or instead of graphemes. The output labels can also be other types of speech units, such as phonemes or subphonemes. The second probability distribution 120b of the second collaborative network 250b can include posterior probability values ​​for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output of the collaborative network 250 can include 100 different probability values, one for each output label. The second probability distribution 120b can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, word fragments, and / or words) in a beam search process (e.g., by the final softmax layer of the second collaborative network 250b) to determine the initial speech recognition result 120b.For example, the second collaborative network 250b may select the N best possible speech recognition hypotheses with the highest probabilities as outputs of the initial speech recognition results 120b.

[0032] In some implementations, the second prediction network 300b receives as input the sequence of non-empty symbols output by the final softmax layer of the second collaborative network 250b, and at each output step, generates a dense representation 350. That is, the second collaborative network 250b receives the dense representation 350 for a previous initial speech recognition result 120b and uses the dense representation 350 to generate a subsequent initial speech recognition result 120b.

[0033] 2A , in some configurations, the language ID predictor 230 of the exemplary ASR model 200, 200 a is configured to receive as input the first high-level feature representation 212 generated by the first encoder 210 in each of the plurality of output steps and the second high-level feature representation 222 generated by the second encoder 220 in each of the plurality of output steps. Additionally, the language ID predictor 230 may generate a combination 231 of the first high-level feature representation 212 and the second high-level feature representation 222. The language ID predictor 230 is then further configured to generate a language predicted representation 232 based on the combination 231 of the first high-level feature representation 212 and the second high-level feature representation 222 in each of the plurality of output steps. Advantageously, upon receiving the combination 231, the language ID predictor 230 generates the language predicted representation 232 using the diverse inputs.

[0034] The language prediction representation 232 indicates the corresponding language of the spoken utterance. For example, because the ASR model 200 is a multilingual ASR model, the spoken utterance may be in any number of languages. Thus, using the combination 231, the language ID predictor 230 predicts the corresponding language of the spoken utterance. The language prediction representation 232 may be used for downstream tasks (e.g., code-switching or speech conversion) and / or may be used to improve speech recognition results. That is, the first decoder 240a may use the language prediction representation 232 to improve the initial speech recognition result 120b generated by the second decoder 240b. In some examples, the language ID predictor 230 generates the language prediction representation 232 for each frame. In these examples, the spoken utterance may include multiple utterances, and the language ID predictor 230 generates the language prediction representation 232 for each acoustic frame 110 in the sequence of acoustic frames 110. For example, in a first portion of the sequence of acoustic frames, the language prediction representation 232 may indicate that a first language was spoken, while in a second portion of the sequence of acoustic frames, the language prediction representation 232 indicates that a second language was spoken.

[0035] 2A , the first decoder 240a includes a first collaborative network 250a and a first prediction network 300a. In some configurations, the first collaborative network 250a is configured to receive as input the dense representation 350 generated by the first prediction network 300a, the second high-level feature representation 222 generated by the second encoder 220, and the language prediction representation 232 generated by the language ID predictor 230, and to generate a final speech recognition result 120a for the corresponding acoustic frame 110 at each output step. Here, the collaborative network 250 uses the second high-level feature representation 222, the language prediction representation 232, and the dense representation 350 to generate the final speech recognition result 120a. In some examples, the first decoder 240a generates a combination of the second high-level feature representation 222 and the language prediction representation 232 and uses this combination to generate the final speech recognition result 120a.

[0036] In some implementations, the final speech recognition result 120a includes a first probability distribution over possible speech recognition hypotheses. In this manner, the final speech recognition result 120a may be used interchangeably with the term "first probability distribution over possible speech recognition hypotheses" 120a herein. Accordingly, the first collaborative network 250a may generate a first probability distribution over possible speech recognition hypotheses 120a at each output step (e.g., time step). Here, a "possible speech recognition hypothesis" corresponds to a set of output labels / symbols (also called "phonetic units"), each representing a grapheme (symbol / character) or word fragment of a specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters of the English alphabet and one label representing a space. Accordingly, the first collaborative network 250a may output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels. This set of values ​​may be a vector (e.g., a one-hot vector) and may indicate a first probability distribution over the set of output labels. In some scenarios, the output labels are graphemes (e.g., individual characters, potential punctuation marks, or other symbols), but the set of output labels is not so limited. For example, the set of output labels can include word fragments and / or entire words in addition to or instead of graphemes. The output labels can also be other types of speech units, such as phonemes or subphonemes. The first probability distribution 120a of the first collaborative network 250a can include posterior probability values ​​for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output of the first collaborative network 250a can include 100 different probability values, one for each output label. The first probability distribution 120a can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, word fragments, and / or words) in a beam search process (e.g., by a final softmax layer of the first collaborative network 250a (not shown)) to determine the final speech recognition result 120a.For example, the first collaborative network 250a may select the N best possible speech recognition hypotheses with the highest probabilities as outputs for the final speech recognition results 120a.

[0037] In some implementations, the first prediction network 300a receives as input the sequence of non-empty symbols output by the final softmax layer of the first collaborative network 250a, and at each output step, generates a dense representation 350. That is, the first collaborative network 250a receives the dense representation 350 for the previous final speech recognition result 120a and uses the dense representation 350 to generate the subsequent final speech recognition result 120a.

[0038] 2B , in some configurations, the language ID predictor 230 of the exemplary ASR model 200, 200 b is configured to receive as input the first high-level feature representation 212 generated by the first encoder 210 in each of a plurality of output steps and to generate a language predicted representation 232 in each of a plurality of output steps. Here, the language ID predictor 230 generates the language predicted representation 232 based solely on the first high-level feature representation 212 and outputs the language predicted representation 232 to the second encoder 220. Advantageously, in this configuration, both the first decoder 240 a and the second encoder 220 benefit from using the language predicted representation 232.

[0039] Thus, in these configurations, the second encoder 220 is further configured to receive the first high-order feature representation 212 and the language predictive representation 232 as input and to generate a second high-order feature representation 222 for a corresponding first high-order feature representation 212 at each output step. The second encoder 220 may generate a combination 221 of the first high-order feature representation 212 and the language predictive representation 232 and generate the second high-order feature representation based on the combination 221. In some examples, the second encoder 220 generates the second high-order feature representation 222 without receiving any acoustic frames 110 as input. In these examples, the second encoder 220 generates the second high-order feature representation 222 using only the first high-order feature representation 212 and the language predictive representation 232 as input.

[0040] 2B , the first decoder 240a includes a first collaborative network 250a and a first prediction network 300a. In some configurations, the first collaborative network 250a is configured to receive as input the dense representation 350 generated by the first prediction network 300a and the second high-level feature representation 222 generated by the second encoder 220, and to generate a final speech recognition result 120a for the corresponding acoustic frame 110 at each output step. Here, the collaborative network 250 uses the second high-level feature representation 222 and the dense representation 350 to generate the final speech recognition result 120a.

[0041] 2A and 2B, the language ID predictor 230 uses more acoustic information (e.g., longer audio features) to generate a more accurate language predicted representation 232. Therefore, to utilize all past acoustic frames 110 but still generate the language predicted representation 232 for each frame, the language ID predictor 230 uses non-parametric statistical pooling. That is, the language ID predictor 230 generates the first high-level feature representation 212 using the mean (μ t ) and standard deviation (σ t) into a combination with . Notably, the language ID predictor 230 determines the mean and standard deviation in a streaming manner, represented by:

number

[0042] In some implementations, the ASR model 200 jointly trains the first encoder 210, the second encoder 220, and the language ID predictor 230 on a set of multilingual training utterances. Here, a language ID target token is added as the first token of the corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances. The language ID target token identifies the language of the corresponding multilingual training utterance. That is, the set of multilingual training utterances can include training utterances in any number of different languages, and the language ID target token identifies the actual language (e.g., ground truth label) of the multilingual training utterance for training purposes.

[0043] During training, the training process generates a first loss for the first encoder 210 and a second loss for the second encoder 220, which are denoted by: L rnnt =lnP(y|x 1:T ) (3)

number

number

[0044] FIG. 3 illustrates that the prediction network 300 of either the first decoder 240a or the second decoder 240b (FIG. 2) receives as input a sequence of non-blank symbols, limited to the N previous non-blank symbols output by the final softmax layer of the collaborative network 250. That is, the sequence of non-blank symbols 120, 120a-n output by the final softmax layer of the collaborative network 250 may correspond to the non-blank symbols of the speech recognition result 120. Thus, the terms non-blank symbols 120 and sequence of speech recognition result 120 may be used interchangeably herein. In some examples, N is equal to 2. In other examples, N is equal to 5, although the disclosure is non-limiting and N may be equal to any integer. In some implementations, the prediction network 300 includes a multi-head attention mechanism 302, which shares a shared embedding matrix 304 across each head 302A-302H of the multi-head attention mechanism. In one example, the multi-head attention mechanism 302 includes four heads. However, any number of heads can be used with the multi-head attention mechanism 302. In particular, the multi-head attention mechanism significantly improves performance while minimizing the increase in model size. As explained in more detail below, each head 302A-H contains its own row of position vector 308, and rather than combining the outputs 318A-H from all heads, which would increase model size, the outputs 318A-H are instead averaged by a head averaging module 322.

[0045] Referring to the first head 302A of the multi-head attention mechanism 302, the head 302A uses a shared embedding matrix 304 to generate a corresponding embedding 306, 306a-n (e.g.,

number

number

[0046] The corresponding embedding generated by the shared embedding matrix 304 for each non-blank symbol 120 in the sequence of non-blank symbols 120 is the same for all heads 302A-H of the multi-head attention mechanism 302, but each head 302A-H defines a different set / row of position vectors 308. For example, the first head 302A defines the row PV of the position vectors Aa~An 308Aa to 308An, and the second head 302B defines rows PV Ba~Bn 308 Ba~Bn , and the H-th head 302H defines another different row of the position vector PV Ha~Hn 308 Ha~Hn This stipulates:

[0047] For each non-blank symbol in the received sequence of non-blank symbols 120, the first head 302A also weights the corresponding embedding 306, via the weighting layer 310, in proportion to the similarity between the corresponding embedding and the respective position vector PV 308 assigned thereto. In some examples, the similarity may include cosine similarity (e.g., cosine distance). In the example shown, the weighting layer 310 outputs a sequence of weighted embeddings 312, 312Aa-An, each associated with a corresponding embedding 306, weighted in proportion to the respective position vector PV 308 assigned thereto. Stated differently, the weighted embedding 312 output by the weighting layer 310 for each embedding 306 may correspond to the dot product between the embedding 306 and the respective position vector PV 308. The weighted embedding 312 may be interpreted as overriding the embedding in proportion to how similar the embedding is to the positioning associated with the respective position vector PV 308. To increase computational speed, the prediction network 300 includes non-recurrent layers, so the sequence of weighted embeddings 312Aa-An are not combined but instead averaged by a weighted average module 316 to produce, as output from the first head 302A, a weighted average 318A of the weighted embeddings 312Aa-An, expressed as follows:

number

[0048] In Equation 8, h represents the index of the head 302, n represents the position in the context, and e represents the embedding dimension. e contains the size of the corresponding dimension. The position vector PV 308 does not need to be trainable and may contain random values. In particular, even if the weighted embedding 312 is averaged, the position vector PV 308 can potentially preserve position history information, alleviating the need to provide recurrent connections at each layer of the prediction network 300.

[0049] The operations described above with respect to the first head 302A are similarly performed for each of the other heads 302B-H of the multi-head attention mechanism 302. Due to the different sets of positioning vectors PV 308 defined by each head 302, the weighting layer 310 outputs a sequence of weighted embeddings 312Ba-Bn, 312Ha-Hn for each of the other heads 302B-H that differs from the sequence of weighted embeddings 312Aa-Aa for the first head 302A. The weighted average module 316 then generates a weighted average 318B-H of each of the corresponding weighted embeddings 312 for the sequences of non-blank symbols as output from each of the other corresponding heads 302B-H.

[0050] In the illustrated example, the prediction network 300 includes a head averaging module 322 that averages the weighted averages 318A-H output from corresponding heads 302A-H. A projection layer 326 with SWISH can receive as input an output 324 from the head averaging module 322 corresponding to the average of the weighted averages 318A-H and generate as output a projection output 328. A final layer normalization 330 can normalize the projection output 328 to provide a single embedding vector 350 at a corresponding time step from the multiple time steps. The prediction network 300 generates only a single embedding vector 350 at each of the multiple time steps following the initial time step.

[0051] In some configurations, the prediction network 300 does not implement a multi-head attention mechanism 302 and only performs the operations described above with respect to the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-An is simply passed through the projection layer 326 and layer normalization 330 to provide a single embedding vector 350.

[0052] 4 is a flowchart of an exemplary arrangement of operations for a method 400 of streaming end-to-end multilingual speech recognition with collaborative language identification. Method 400 may be executed by data processing hardware 610 (FIG. 6) using instructions stored in memory hardware 620 (FIG. 6). Data processing hardware 610 and memory hardware 620 may reside in user device 10 and / or remote computing device 60 of FIG. 1.

[0053] At operation 402, the method 400 includes receiving a sequence of acoustic frames 110 as input to the ASR model 200. At operation 404, the method 400 includes generating, by a first encoder 210 of the ASR model 200 and in each of a plurality of output steps, a first high-level feature representation 212 for a corresponding acoustic frame 110 in the sequence of acoustic frames 110. At operation 406, the method 400 includes generating, by a second encoder 220 of the ASR model 200 and in each of a plurality of output steps, a second high-level feature representation 222 for the corresponding first high-level feature representation 212. At operation 408, the method 400 includes generating, by the language ID predictor 230 of the ASR model 200 and at each of the multiple output steps, a language predicted representation 232 based on a combination 231 of the first high-level feature representation 212 generated by the first encoder 210 at each of the multiple output steps and the second high-level feature representation 222 generated by the second encoder 220 at each of the multiple output steps. At operation 410, the method 400 includes generating, by the first decoder 240, 240a of the ASR model 200 and at each of the multiple output steps, a first probability distribution 120a over the possible speech recognition hypotheses based on a combination of the second high-level feature representation 222 generated by the second encoder 220 at each of the multiple output steps and the language predicted representation 232 generated by the language ID predictor 230 at each of the multiple output steps.

[0054] 5 is a flowchart of an exemplary arrangement of operations for a method 500 of streaming end-to-end multilingual speech recognition with collaborative language identification. Method 500 may be executed by data processing hardware 610 (FIG. 6) using instructions stored in memory hardware 620 (FIG. 6). Data processing hardware 610 and memory hardware 620 may reside in user device 10 and / or remote computing device 60 of FIG. 1.

[0055] At operation 502, the method 500 includes receiving a sequence of acoustic frames 110 as input to the ASR model 200. At operation 504, the method 500 includes generating, by a first encoder 210 of the ASR model 200 and in each of a plurality of output steps, a first high-level feature representation 212 for a corresponding acoustic frame 110 in the sequence of acoustic frames 110. At operation 506, the method 500 includes generating, by a language ID predictor 230 of the ASR model 200 and in each of a plurality of output steps, a language prediction representation 232 for the corresponding first high-level feature representation 212. At operation 508, the method 500 includes generating, by the second encoder 220 of the ASR model 200 and at each of the plurality of output steps, a second high-level feature representation 222 based on a combination 221 of the first high-level feature representation 212 generated by the first encoder 210 and the language predicted representation 232 generated by the language ID predictor 230 at each of the plurality of output steps. At operation 510, the method 500 includes generating, by the first decoder 240a of the ASR model 200 and at each of the plurality of output steps, a first probability distribution 120a over possible speech recognition hypotheses (e.g., final speech recognition results 120a) for the corresponding second high-level feature representation 222.

[0056] 6 is a schematic diagram of an exemplary computing device 600 that can be used to implement the systems and methods described herein. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the embodiments of the invention described and / or claimed in this document.

[0057] Computing device 600 includes a processor 610, memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be mounted on a common motherboard or mounted in any other suitable manner. Processor 610 processes instructions for execution within computing device 600, including instructions stored in memory 620 or storage device 630, and can display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 680 connected to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used as needed, along with multiple memories and types of memories. Also, multiple computing devices 600 may be connected, each performing some of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0058] The memory 620 stores non-transitory information within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transitory memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0059] Storage device 630 can provide mass storage for computing device 600. In some embodiments, storage device 630 is a computer-readable medium. In various different implementations, storage device 630 can be a device array including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 620, storage device 630, or memory on processor 610.

[0060] High-speed controller 640 manages bandwidth-intensive operations of computing device 600, while low-speed controller 660 manages lower-bandwidth-intensive operations. This allocation of roles is merely exemplary. In some implementations, high-speed controller 640 (e.g., via a graphics processor or accelerator) is coupled to memory 620, display 680, and to high-speed expansion port 650, which can accept various expansion cards (not shown). In some implementations, low-speed controller 660 is coupled to storage device 630 and low-speed expansion port 690. Low-speed expansion port 690 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet, etc.) that can be connected to one or more input / output devices, such as a keyboard, pointing device, scanner, or network device, such as a switch or router, for example, via a network adapter.

[0061] The computing device 600, as shown, may be implemented in many different forms. For example, it may be implemented as a standard server 600a, or as a laptop computer 600b, or as part of a rack server system 600c, or multiple times in a group of such servers 600a.

[0062] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general-purpose, and may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor coupled to receive data and instructions from the storage system and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0063] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0064] The processes and logic flows described herein may be implemented by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operably coupled to one or more mass storage devices for receiving data, transmitting data, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal or removable hard disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0065] To provide for interaction with a user, aspects of the present disclosure may be implemented on a computer having a display device, such as a cathode ray tube (CRT), liquid crystal display (LCD) monitor, or touchscreen for displaying information to a user, and, optionally, a keyboard and pointing device, such as a mouse or trackball, by which a user can provide input to the computer. Other types of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, a computer may interact with a user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0066] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. 1. A multilingual automatic speech recognition (ASR) model, comprising: a first encoder, receiving as input a sequence of multiple acoustic frames; a first encoder configured to generate, at each of a plurality of output steps corresponding to each of the plurality of acoustic frames, a first high-level feature representation by processing a feature vector of a corresponding acoustic frame in the sequence of acoustic frames; a second encoder, In each of the plurality of output steps, the first high-level feature representation generated by the first encoder is received as an input; a second encoder configured to generate a second high-level feature representation by processing a corresponding first high-level feature representation in each of the plurality of output steps; 1. A language identification (ID) predictor, comprising: receiving as input a combination of the first high-level feature representation generated by the first encoder in each of the plurality of output steps and the second high-level feature representation generated by the second encoder in each of the plurality of output steps; a language identification (ID) predictor configured to generate a language predicted representation in each of the plurality of output steps; a first decoder, receiving as input a combination of the second high-level feature representation produced by the second encoder in each of the plurality of output steps and the language predicted representation produced by the language identification (ID) predictor in each of the plurality of output steps; a first decoder configured to generate a first probability distribution over possible speech recognition hypotheses at each of the plurality of output steps; ASR models, including:

2. In each of the plurality of output steps, the first high-level feature representation generated by the first encoder is received as an input; 2. The ASR model of claim 1, further comprising a second decoder configured to generate a second probability distribution over possible speech recognition hypotheses at each of the plurality of output steps.

3. 3. The ASR model of claim 2, wherein the second decoder is further configured to generate a partial speech recognition result based on the second probability distribution over possible speech recognition hypotheses.

4. each of the first decoder and the second decoder includes a corresponding prediction network followed by a corresponding collaborative network; the corresponding prediction networks of the first decoder and the second decoder are Long short-term memory (LSTM)-based predictive networks, or V2 Embedded Lookup Table and The ASR model of claim 2 , wherein the corresponding joint networks of the first decoder and the second decoder include the same structure.

5. The ASR model of claim 1 , wherein the second encoder generates the second higher-level feature representation without receiving any of the acoustic frames as input.

6. The first encoder comprises: Multiple one-way long short-term memory (LSTM) layers, Multiple conformer layers, or Multiple Transformer Layers The ASR model of claim 1 , comprising a causal encoder comprising one of:

7. The second encoder comprises: one or more bidirectional long short-term memory (LSTM) layers; Multiple conformer layers, or Multiple Transformer Layers The ASR model of claim 1 , comprising a non-causal encoder comprising one of:

8. the first encoder, the second encoder, and the language identification (ID) predictor; generating a first loss for the first encoder; generating a second loss for the second encoder; generating a third loss for the language identification (ID) predictor; minimizing a weighted sum of the first loss, the second loss, and the third loss; 10. The ASR model of claim 1, wherein the ASR model is jointly trained on a set of multilingual training utterances by

9. 9. The ASR model of claim 8, wherein a language ID target token is added as a first token of a corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances, the language ID target token identifying a language of the corresponding multilingual training utterance.

10. 9. The ASR model of claim 8, wherein a language ID target token is added to each position where a code-switch occurs in a corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances.

11. 1. A computer-implemented method comprising: receiving a sequence of a plurality of acoustic frames as input to an automatic speech recognition (ASR) model; generating, in each of a plurality of output steps corresponding to each of the plurality of acoustic frames, a first high-level feature representation by processing, with a first encoder of the ASR model, a feature vector of a corresponding acoustic frame in the sequence of acoustic frames; In each of a plurality of output steps, generating a second high-level feature representation by processing the corresponding first high-level feature representation with a second encoder of the ASR model; generating, by a language identification (ID) predictor of the ASR model, a language predicted representation based on a combination of the first high-level feature representation generated by the first encoder in each of the plurality of output steps and the second high-level feature representation generated by the second encoder in each of the plurality of output steps; generating, by a first decoder of the ASR model, a first probability distribution over possible speech recognition hypotheses based on a combination of the second high-level feature representation generated by the second encoder in each of the plurality of output steps and the language prediction representation generated by the language identification (ID) predictor in each of the plurality of output steps; A method comprising:

12. 12. The method of claim 11, further comprising generating, by a second decoder of the ASR model, at each of the plurality of output steps, a second probability distribution over possible speech recognition hypotheses based on the first high-dimensional feature representation produced by the first encoder at each of the plurality of output steps.

13. 13. The method of claim 12, further comprising generating, by the second decoder, a partial speech recognition result based on the second probability distribution over possible speech recognition hypotheses.

14. each of the first decoder and the second decoder includes a corresponding prediction network followed by a corresponding collaborative network; the corresponding prediction networks of the first decoder and the second decoder are Long short-term memory (LSTM)-based predictive networks, or V2 Embedded Lookup Table and The method of claim 12 , wherein the corresponding collaboration networks of the first decoder and the second decoder include the same structure.

15. The method of claim 11 , wherein the second encoder generates the second higher-level feature representation without receiving any of the acoustic frames as input.

16. The first encoder comprises: Multiple one-way long short-term memory (LSTM) layers, Multiple conformer layers, or Multiple Transformer Layers The method of claim 11 , comprising a causal encoder comprising one of:

17. The second encoder comprises: one or more bidirectional long short-term memory (LSTM) layers; Multiple conformer layers, or Multiple Transformer Layers The method of claim 11 , comprising a non-causal encoder comprising one of:

18. generating a first loss for the first encoder; generating a second loss for the second encoder; generating a third loss for the language identification (ID) predictor; and minimizing a weighted sum of the first loss, the second loss, and the third loss, thereby further comprising jointly training the first encoder, the second encoder, and the language identification (ID) predictor on a set of multilingual training utterances. The method of claim 11.

19. adding a language ID target token as a first token of a corresponding ground truth transcription of each training utterance in the set of multilingual training utterances; 20. The method of claim 18, wherein the language ID target token identifies a language of the corresponding training utterance.

20. 20. The method of claim 18, further comprising adding a language ID target token at each position where a code switch occurs in a corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances.

Images