Streaming End-to-End Multilingual Speech Recognition with Joint Language Identification

Abstract

The method (400) includes receiving a sequence of acoustic frames (110) as an input to an automatic speech recognition (ASR) model (200). The method also includes generating, by a first encoder (210), a first high-level feature representation (212) for the corresponding acoustic frame. The method also includes generating, by a second encoder (220), a second high-level feature representation (222) for the corresponding first high-level feature representation. The method also includes generating, by a language identification (ID) predictor (230), a language prediction representation (232) based on a combination (231) of the first high-level feature representation and the second high-level feature representation. The method also includes generating, by a first decoder (240a), a first probability distribution (120a) over possible speech recognition hypotheses based on a combination of the second high-level feature representation and the language prediction representation.

Description

[Technical field]

[0001] The present disclosure relates to streaming end-to-end multilingual speech recognition with collaborative language identification. [Background technology]

[0002] Automatic speech recognition (ASR), the process of taking audio input and transcribing it into text, has become a very important technology used in mobile and other devices. In general, automatic speech recognition attempts to provide an accurate transcription of what a person has spoken by taking an audio input (e.g., a spoken utterance) and transcribing the audio input into text. Based on the ongoing development of deep neural networks, modern ASR models are continuously improving in both accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay between the client's speech and the transcription). Despite the vast number of people being bilingual, the majority of ASR models are only monolingual. Thus, an ASR model that is compatible with several different languages ​​while maintaining the accuracy and latency performance metrics of modern ASR models becomes desirable for a vast number of bilingual speakers. Summary of the Invention

[0003] One aspect of the present disclosure provides a multilingual automatic speech recognition (ASR) model. The ASR model includes a first encoder configured to receive as an input a sequence of acoustic frames and generate a first high-order feature representation of a corresponding acoustic frame in the sequence of acoustic frames in each of a plurality of output steps. The ASR model also includes a second encoder configured to receive as an input the first high-order feature representation generated by the first encoder in each of the plurality of output steps and generate a second high-order feature representation for the corresponding first high-order feature representation in each of the plurality of output steps. The ASR model also includes a language identification (ID) predictor configured to receive as an input a combination of the first high-order feature representation generated by the first encoder in each of the plurality of output steps and the second high-order feature representation generated by the second encoder in each of the plurality of output steps and generate a language prediction representation in each of the plurality of output steps. The ASR model also includes a first decoder configured to receive as input a combination of the second higher-level feature representation generated by the second encoder in each of the multiple output steps and the language prediction representation generated by the language ID predictor in each of the multiple output steps, and to generate a first probability distribution over possible speech recognition hypotheses in each of the multiple output steps.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the ASR model also includes a second decoder configured to receive as input the first high-level feature representation generated by the first encoder in each of the multiple output steps and generate a second probability distribution over the possible speech recognition hypotheses in each of the multiple output steps. Here, the second decoder may be further configured to generate a partial speech recognition result based on the second probability distribution over the possible speech recognition hypotheses. In these implementations, the first decoder and the second decoder may each include a corresponding prediction network followed by a corresponding collaborative network, where the corresponding prediction networks of the first decoder and the second decoder have the same structure, including one of a long short-term memory (LSTM)-based prediction network or a V2 embedded lookup table, and the corresponding collaborative networks of the first decoder and the second decoder include the same structure.

[0005] In some examples, the second encoder generates the second high-level feature representation without receiving any acoustic frames as input. The first encoder includes a causal encoder having one of a plurality of unidirectional long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers. In some implementations, the second encoder includes a non-causal encoder having one or more bidirectional long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers.

[0006] In some implementations, the first encoder, the second encoder, and the language ID predictor are jointly trained on the set of multilingual training utterances by generating a first loss for the first encoder, a second loss for the second encoder, a third loss for the language ID predictor, and minimizing a weighted sum of the first loss, the second loss, and the third loss. In these implementations, the language ID target token may be added as a first token of a corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances. The language ID target token identifies the language of the corresponding multilingual training utterance. Alternatively, the language ID target token may be added to each position where a code switch occurs in the corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances.

[0007] Another aspect of the disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for streaming end-to-end multilingual speech recognition with collaborative language recognition. The operations include receiving a sequence of acoustic frames as an input to an automatic speech recognition (ASR) model. The operations also include generating, in each of a plurality of output steps, a first high-level feature representation by a first encoder of the ASR model for a corresponding acoustic frame in the sequence of acoustic frames. The operations also include generating, in each of the plurality of output steps, a second high-level feature representation by a second encoder of the ASR model for the corresponding first high-level feature representation. The operations also include generating, in each of the plurality of output steps, a language prediction representation by a language identification (ID) predictor of the ASR model. Here, the language prediction representation is based on a combination of the first high-level feature representation generated by the first encoder in each of the plurality of output steps and the second high-level feature representation generated by the second encoder in each of the plurality of output steps. The operations also include generating, at each of the multiple output steps, a first probability distribution over the possible speech recognition hypotheses by a first decoder of the ASR model, where the first probability distribution over the possible speech recognition hypotheses is based on a combination of the second higher-level feature representation generated by the second encoder at each of the multiple output steps and the language prediction representation generated by the language ID predictor at each of the multiple output steps.

[0008] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operations further include generating, at each of the plurality of output steps, a second probability distribution over the possible speech recognition hypotheses by a second decoder of the ASR model. The second probability distribution over the possible speech recognition hypotheses is based on the first high-order feature representation generated by the first encoder at each of the plurality of output steps. In these implementations, the operations may further include generating, by the second decoder, a partial speech recognition result based on the second probability distribution over the possible speech recognition hypotheses. The first and second decoders may each include a corresponding prediction network followed by a corresponding collaborative network, the corresponding prediction networks of the first and second decoders having the same structure and including one of a long-term-term (LSTM) based prediction network or a V2 embedded lookup table, and the corresponding collaborative networks of the first and second decoders including the same structure.

[0009] In some examples, the second encoder generates the second high-level feature representation without receiving any acoustic frames as input. In some examples, the first encoder includes a causal encoder including one of a plurality of one-way long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers. The second encoder includes a non-causal encoder including one or more bidirectional long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers.

[0010] In some examples, the operations further include jointly training the first encoder, the second encoder, and the language ID predictor on the set of multilingual training utterances by generating a first loss for the encoder, generating a second loss for the second encoder, generating a third loss for the language ID predictor, and minimizing a weighted sum of the first loss, the second loss, and the third loss. In these examples, the operations may further include adding a language ID token as a first token of a corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances. The language ID target token identifies a language of the corresponding multilingual training utterance. Alternatively, the operations further include adding a language ID target token to each position where a code switch occurs in the corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances.

[0011] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0012] [Figure 1] 1 is a schematic diagram of an exemplary speech recognition system. [Figure 2A] FIG. 2 is a schematic diagram of an exemplary speech recognition model. [Figure 2B] FIG. 2 is a schematic diagram of an exemplary speech recognition model. [Diagram 3] FIG. 3 is a schematic diagram of an example prediction network of the example speech recognition model of FIGS. 2A and 2B. [Figure 4] 1 is a flowchart of an exemplary sequence of operations for a computer-implemented method for streaming end-to-end multilingual speech recognition with joint language identification. [Diagram 5]1 is a flowchart of an exemplary sequence of operations for a computer-implemented method for streaming end-to-end multilingual speech recognition with joint language identification. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Like reference numbers indicate like elements in the various drawings.

[0014] End-to-end (E2E) automatic speech recognition (ASR) models are traditionally structured to operate in either streaming or non-streaming modes. Traditionally, E2E ASR models include an encoder and a decoder as major components. Applications involving end-user interaction, such as voice search or on-device listening, may require the model to perform recognition in a streaming manner. Here, performing recognition in a streaming manner refers to the ASR model outputting each word of an utterance as it is spoken with as little latency as possible. Other applications, such as offline video captioning, do not require the model to be streamed and can take advantage of future context to improve performance.

[0015] In some implementations, the E2E ASR model is configured to recognize speech from multiple languages ​​(e.g., an E2E multilingual ASR model), where the multilingual ASR model and the language identification model are often used together in downstream tasks (e.g., code switching and speech conversion), but the models are constructed and run separately. Thus, using both an independent multilingual ASR model and an independent language identification model unnecessarily complicates the overall ASR system by increasing computational and storage costs.

[0016] Accordingly, embodiments herein are directed to an E2E multilingual ASR model that jointly generates a transcription and a language prediction representation. The ASR model includes a first encoder configured to generate a first high-level feature representation of a corresponding acoustic frame, and a second encoder configured to generate a second high-level feature representation of the corresponding first high-level feature representation. The ASR model also includes a language identification predictor configured to receive a concatenation of the first high-level feature representation and the second high-level feature representation and generate a language prediction representation. The ASR model also includes a first decoder configured to receive a concatenation of the second high-level feature representation and the language prediction representation and generate a first probability distribution over possible speech recognition hypotheses (e.g., transcriptions). Thus, the ASR model (e.g., a single E2E model) jointly outputs both the transcription and the language prediction representation. Notably, the ASR model does not receive or use any oracle language identification information to generate the transcription and the language prediction representation. As will become apparent, the ASR model operates in either streaming or non-streaming modes to generate the transcription and language prediction representations.

[0017] 1 is an example of an audio environment 100. In the audio environment 100, a way in which a user 104 interacts with a computing device, such as a user device 10, may be via voice input. The user device 10 (also generally referred to as device 10) is configured to capture sounds (e.g., streaming audio data) from one or more users 104 in the audio environment 100. Here, the streaming audio data may refer to spoken utterances 106 by the user 104 that function as audible queries, commands to the user device 10, or audible communications captured by the device 10. A voice-enabled system of the user device 10 may process the queries or commands by responding to the queries and / or causing the commands to be executed / fulfilled by one or more downstream applications.

[0018] The user device 10 may correspond to any computing device associated with a user 104 and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, a mobile device (e.g., a smart watch), a smart appliance, an Internet of Things (IoT) device, an in-vehicle infotainment system, a smart display, a smart speaker, and the like. The user device 10 includes data processing hardware 12 and storage hardware 14 in communication with the data processing hardware 12, storing instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 with audio capture devices (e.g., microphones) 16, 16a for capturing and converting spoken speech 106 into electrical signals, and audio output devices (e.g., speakers) 16, 16b for communicating audible audio signals (e.g., as output data from the user device 10) within the audio environment 100. The user device 10 may implement an array of audio capture devices 16a without departing from the scope of this disclosure, whereby one or more capture devices 16a in the array may not be physically present at the user device 10 but may communicate with the audio system 16.

[0019] In the speech environment 100, an automatic speech recognition (ASR) system 118 implements an ASR model 200 and resides on the user device 10 of the user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system running in a cloud computing environment) that communicates with the user device 10 via a network 40. In some examples, the ASR model 200 may be a recurrent neural network-transducer (RNN-T) model. The user device 10 and / or the remote computing device 60 also include an audio subsystem 108 configured to receive utterances 106 spoken by the user 104 and captured by the audio capture device 16a, and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the ASR system 118. In the example shown, the user speaks each utterance 106, and the audio subsystem 108 converts the utterances 106 into corresponding audio data (e.g., a sequence of acoustic frames) 110 for input to the ASR system 118. The ASR model 200 then receives as input a sequence of acoustic frames 110 corresponding to the utterance 106, and at each output step, generates / predicts a corresponding transcription 120 (e.g., a speech recognition result / hypothesis) of the utterance 106 as the ASR model receives (e.g., processes) each acoustic frame 110 in the sequence of acoustic frames 110.

[0020] In the illustrated example, the ASR model 200 may perform streaming speech recognition to generate an initial speech recognition result 120, 120b and generate a final speech recognition result 120, 120a by improving the initial speech recognition result 120b. The speech recognition result 120 may correspond to either a partial speech recognition result or an entire speech recognition result. In other words, the speech recognition result 120 may correspond to either a portion of the utterance 106 or the entire utterance 106. For example, the partial speech recognition result may correspond to a portion of a spoken utterance or even a portion of a spoken term. However, as will become apparent, the ASR model 200 performs additional processing on the final speech recognition result 120a, such that the final speech recognition result 120a may lag behind the initial speech recognition result 120b.

[0021] The user device 10 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of the transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 can display the initial speech recognition result 120b in a streaming format during time 1, and then display the final speech recognition result 120a in a streaming format during time 2. Notably, the ASR model 200 outputs the final speech recognition result 120a in a streaming format even if the final speech recognition result 120a improves the initial speech recognition result 120b. In some configurations, the transcription 120 output from the ASR system 118 is processed by, for example, a natural language understanding (NLU) module executing on the user device 10 or the remote computing device 60 to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., running on any combination of the user device 10 or the remote computing device 60) can convert the transcription 120 into synthetic speech for audible output by the user device 10 and / or other devices.

[0022] In the illustrated example, a user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) of the user device 10 using an ASR system 118. For example, FIG. 1 shows a user 104 communicating with the digital assistant application 50 and the digital assistant application 50 displaying a digital assistant interface 18 on the screen of the user device 10 to show a conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, "What time is the concert today?" This question from the user 104 is a spoken utterance 106 that is captured by the audio capture device 16a and processed by the audio system 16 of the user device 10. In this example, the audio system 16 receives the spoken utterance 106 and converts it into a sequence of acoustic frames 110 for input to the ASR system 118.

[0023] Continuing with this example, as the ASR model 200 receives the sequence of acoustic frames 110 corresponding to the utterance 106 as the user 104 speaks, it encodes the sequence of acoustic frames 110 and then decodes the encoded sequence of acoustic frames 110 into an initial speech recognition result 120b. During time 1, the user interface generator 107 presents a representation of the initial speech recognition result 120b of the utterance 106 to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming format such that words, word fragments, and / or individual characters appear on the screen as they are spoken. In some examples, the first look-ahead audio context is equal to zero.

[0024] During time 2, the user interface generator 107 presents a representation of the final speech recognition result 120a of the utterance 106 to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming format such that words, word fragments, and / or individual characters are displayed on the screen as soon as they are generated by the ASR model 200. In some implementations, the user interface generator 107 replaces the representation of the initial speech recognition result 120b presented at time 1 with the representation of the final speech recognition result 120a presented at time 2. Here, time 1 and time 2 may include timestamps corresponding to when the user interface generator 107 presents the respective speech recognition results 120. In this example, the timestamp for time 1 indicates that the user interface generator 107 presents the initial speech recognition result 120b at an earlier time than the final speech recognition result 120a. For example, since the final speech recognition result 120a is estimated to be more accurate than the initial speech recognition result 120b, the final speech recognition result 120a, which is ultimately displayed as the transcription 120, can correct any terms that may have been misrecognized in the initial speech recognition result 120b. In this example, the streaming of the initial speech recognition result 120b output by the ASR model 200 is displayed on the screen of the user device 10 at time 1 and is associated with low latency, providing the user 104 with a sense of responsiveness that the user's query is being processed, while the final speech recognition result 120a output by the ASR model 200 and displayed on the screen at time 2 utilizes additional speech recognition models and / or language models to improve the speech recognition quality in terms of accuracy, albeit at increased latency. However, since the initial speech recognition result 120b is displayed as the user speaks the utterance 106, the higher latency associated with generating and ultimately displaying the final speech recognition result 120a is not noticeable to the user 104.

[0025] In the example shown in FIG. 1, the digital assistant application 50 may use natural language processing to respond to a question posed by the user 104. Natural language processing generally refers to the process of interpreting written language (e.g., initial speech recognition results 120b and / or final speech recognition results 120a) and determining whether the written language prompts any action. In this example, the digital assistant application 50 uses natural language processing to recognize that the question from the user 104 concerns the user's schedule, more specifically, a concert on the user's schedule. By recognizing these details using natural language processing, the automated assistant returns a response 19 to the user's query, where the response 19 states, "The venue's doors will open at 6:30 p.m. and the concert will begin at 8 p.m." In some configurations, the natural language processing occurs on a remote server 60 that communicates with the data processing hardware 12 of the user device 10.

[0026] 2A and 2B, in some implementations, the ASR model 200 includes a cascade encoder 204, a language identification (ID) predictor 230, and a decoder 240. Here, the first decoder 240, 240a is configured to improve the initial speech recognition result 120b output by the second decoder 240, 240b. The second decoder 240b may operate in a streaming manner, such that the second decoder 240b is configured to generate a partial speech recognition result corresponding to the initial speech recognition result 120b. Meanwhile, the first decoder 240a improves the partial speech recognition result by receiving additional correct context to generate the final speech recognition result 120a. Each of the first decoder 240a and the second decoder 240b includes a corresponding prediction network 300 followed by a corresponding joint network 250. Here, the first prediction network 300, 300a and the first collaborative network 250, 250a correspond to the first decoder 240a, and the second prediction network 300, 300b and the second collaborative network 250, 250b correspond to the second decoder 240b. The prediction networks 300a, 300b have the same structure, including one of a long short-term memory (LSTM)-based prediction network or a V2 embedding lookup table. Furthermore, the corresponding collaborative networks 250a, 250b have the same structure. Although the component structure is the same for the first decoder 240a and the second decoder 240b, each component of each decoder 240 is unique and can be trained independently of the components of the other decoder 240.

[0027] The cascaded encoder 204 refers to a model structure in which the encoding path includes two encoders 210, 220 that are cascaded such that the output of the first encoder 210 feeds the input of the second encoder 220 before decoding. Here, the first encoder 210 and the second encoder 220 may be cascaded regardless of the underlying architecture of each encoder. The encoders 210, 220 may each include a stack of multi-head attention layers. In some examples, the first encoder 210 includes a causal encoder with any one of multiple one-way (LSTM) layers, multiple conformer layers, or multiple transformer layers. For example, the first encoder 210 may include 12 conformer layers, each with a multi-head (e.g., 8 heads) self-attention mechanism and a convolution kernel size of 15. Furthermore, the first encoder 210 may perform a combining operation after the third conformer layer achieves a time reduction factor of 2, whereby the resulting 1024-dimensional vector is transformed by the fourth conformer layer and then backprojected to a 512-dimensional vector using another linear transformation. Then, another eight conformer layers are followed by a final normalization layer. Thus, the first encoder 210 may include 110 million parameters. Each layer of the first encoder 210 receives zero correct context (e.g., receives zero future acoustic frames).

[0028] The second encoder 220 includes an acausal encoder having one of one or more bidirectional LSTM layers, multiple conformer layers, or multiple transformer layers. For example, the second encoder 220 may include a 512-dimensional linear projection to transform the input features, followed by five 512-dimensional conformer layers, and a final linear normalization layer, resulting in 50 million parameters. Here, the second encoder 220 may receive additional correct contexts, for example, a total of 15 frames of correct contexts, whereby each conformer layer receives three frames of correct contexts.

[0029] 2A and 2B, a first encoder 210 receives a sequence of d-dimensional feature vectors (e.g., a sequence of acoustic frames 110) x=(x1, x2, ..., x T )(in the formula,

number

[0030] 2A and 2B, the second decoder 240b includes a second collaborative network 250b and a second prediction network 300b. The second collaborative network 250b is configured to receive as input the dense representation 350 generated by the second prediction network 300b and the first high-level feature representation 212 generated by the first encoder 210, and generate an initial speech recognition result 120b for the corresponding acoustic frame 110 at each output step. Here, the second collaborative network 250b uses the first high-level feature representation 212 and the dense representation 350 to generate the initial speech recognition result 120b. The second decoder 240b operates in a streaming manner, such that the second decoder 240b operates such that the initial speech recognition result 120b may correspond to a partial speech recognition result.

[0031] In some implementations, the initial speech recognition results 120b include a second probability distribution over the possible speech recognition hypotheses. In this manner, the initial speech recognition results 120b may be used interchangeably herein as the second probability distribution over the possible speech recognition hypotheses 120b. Thus, the second collaborative network 250b may generate, at each output step (e.g., time step), a second probability distribution over the possible speech recognition hypotheses 120b. Here, a "probable speech recognition hypothesis" corresponds to a set of output labels / symbols (also called "phonetic units") each representing a grapheme (symbol / character) or word fragment of a specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters of the English alphabet and one label representing a space. Thus, the second collaborative network 250b may output a set of values ​​indicating the likelihood of occurrence of each of the given set of output labels. This set of values ​​may be a vector (e.g., one-hot vector) and may indicate a second probability distribution over the set of output labels. In some scenarios, the output labels are graphemes (e.g., individual characters, potential punctuation or other symbols, etc.), but the set of output labels is not so limited. For example, the set of output labels can include word fragments and / or entire words in addition to or instead of graphemes. The output labels can also be other types of speech units, such as phonemes or subphonemes. The second probability distribution 120b of the second collaborative network 250b can include posterior probability values ​​for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output of the collaborative network 250 can include 100 different probability values, one for each output label. The second probability distribution 120b can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, word fragments, and / or words) in a beam search process (e.g., by a final softmax layer of the second collaborative network 250b) to determine the initial speech recognition result 120b.For example, the second collaborative network 250b may select the N best possible speech recognition hypotheses having the highest probabilities as output of the initial speech recognition results 120b.

[0032] In some implementations, the second prediction network 300b receives as input the sequence of non-empty symbols output by the final softmax layer of the second collaborative network 250b and, at each output step, generates a dense representation 350. That is, the second collaborative network 250b receives the dense representation 350 for a previous initial speech recognition result 120b and uses the dense representation 350 to generate a subsequent initial speech recognition result 120b.

[0033] 2A specifically, in some configurations, the language ID predictor 230 of the exemplary ASR model 200, 200a is configured to receive as input the first high-level feature representation 212 generated by the first encoder 210 in each of the multiple output steps and the second high-level feature representation 222 generated by the second encoder 220 in each of the multiple output steps. Additionally, the language ID predictor 230 may generate a combination 231 of the first high-level feature representation 212 and the second high-level feature representation 222. Thereafter, the language ID predictor 230 is further configured to generate a language predicted representation 232 based on the combination 231 of the first high-level feature representation 212 and the second high-level feature representation 222 in each of the multiple output steps. Advantageously, upon receiving the combination 231, the language ID predictor 230 generates the language predicted representation 232 using the diverse inputs.

[0034] The language prediction representation 232 indicates the corresponding language of the spoken utterance. For example, since the ASR model 200 is a multilingual ASR model, the spoken utterance may be in any number of languages. Thus, using the combination 231, the language ID predictor 230 predicts the corresponding language of the spoken utterance. The language prediction representation 232 may be used for downstream tasks (e.g., code switching or speech conversion) and / or may be used to improve the speech recognition results. That is, the first decoder 240a may use the language prediction representation 232 to improve the initial speech recognition results 120b generated by the second decoder 240b. In some examples, the language ID predictor 230 generates the language prediction representation 232 for each frame. In these examples, the spoken utterance may include multiple utterances, and the language ID predictor 230 generates the language prediction representation 232 for each acoustic frame 110 in the sequence of acoustic frames 110. For example, in a first portion of the sequence of acoustic frames, the language prediction representation 232 may indicate that a first language was spoken, while in a second portion of the sequence of acoustic frames, the language prediction representation 232 indicates that a second language was spoken.

[0035] 2A, the first decoder 240a includes a first collaborative network 250a and a first prediction network 300a. In some configurations, the first collaborative network 250a is configured to receive as input the dense representation 350 generated by the first prediction network 300a, the second high-level feature representation 222 generated by the second encoder 220, and the language prediction representation 232 generated by the language ID predictor 230, and generate a final speech recognition 120a for the corresponding acoustic frame 110 at each output step. Here, the collaborative network 250 uses the second high-level feature representation 222, the language prediction representation 232, and the dense representation 350 to generate the final speech recognition result 120a. In some examples, the first decoder 240a generates a combination of the second high-level feature representation 222 and the language prediction representation 232, and uses the combination to generate the final speech recognition result 120a.

[0036] In some implementations, the final speech recognition result 120a includes a first probability distribution over the possible speech recognition hypotheses. In this manner, the final speech recognition result 120a may be used interchangeably with the first probability distribution over the possible speech recognition hypotheses 120a herein. Thus, the first collaborative network 250a may generate a first probability distribution over the possible speech recognition hypotheses 120a at each output step (e.g., time step), where a "probable speech recognition hypothesis" corresponds to a set of output labels / symbols (also called "phonetic units") each representing a grapheme (symbol / character) or word fragment of a specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters of the English alphabet and one label representing a space. Thus, the first collaborative network 250a may output a set of values ​​indicating the likelihood of occurrence of each of the predetermined set of output labels. This set of values ​​may be a vector (e.g., one-hot vector) and may indicate a first probability distribution over the set of output labels. In some scenarios, the output labels are graphemes (e.g., individual characters, potential punctuation or other symbols, etc.), but the set of output labels is not so limited. For example, the set of output labels can include word fragments and / or entire words in addition to or instead of graphemes. The output labels can also be other types of speech units, such as phonemes or subphonemes. The first probability distribution 120a of the first collaborative network 250a can include posterior probability values ​​for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output of the first collaborative network 250a can include 100 different probability values, one for each output label. The first probability distribution 120a can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, word fragments, and / or words) in a beam search process (e.g., by a final softmax layer of the first collaborative network 250a (not shown)) to determine the final speech recognition result 120a.For example, the first collaborative network 250a may select the N best possible speech recognition hypotheses having the highest probabilities as output of the final speech recognition result 120a.

[0037] In some implementations, the first prediction network 300a receives as input the sequence of non-empty symbols output by the final softmax layer of the first collaborative network 250a and, at each output step, generates a dense representation 350. That is, the first collaborative network 250a receives the dense representation 350 for a previous final speech recognition result 120a and uses the dense representation 350 to generate a subsequent final speech recognition result 120a.

[0038] 2B, in some configurations, the language ID predictor 230 of the example ASR model 200, 200b is configured to receive as input the first high-level feature representation 212 generated by the first encoder 210 in each of a plurality of output steps, and generate a language prediction representation 232 in each of a plurality of output steps. Here, the language ID predictor 230 generates the language prediction representation 232 based solely on the first high-level feature representation 212, and outputs the language prediction representation 232 to the second encoder 220. Advantageously, in this configuration, both the first decoder 240a and the second encoder 220 benefit from using the language prediction representation 232.

[0039] Thus, in these configurations, the second encoder 220 is further configured to receive the first high-level feature representation 212 and the language prediction representation 232 as inputs and generate a second high-level feature representation 222 for a corresponding first high-level feature representation 212 at each output step. The second encoder 220 may generate a combination 221 of the first high-level feature representation 212 and the language prediction representation 232 and generate the second high-level feature representation based on the combination 221. In some examples, the second encoder 220 generates the second high-level feature representation 222 without receiving any acoustic frames 110 as inputs. In these examples, the second encoder 220 generates the second high-level feature representation 222 using only the first high-level feature representation 212 and the language prediction representation 232 as inputs.

[0040] 2B, the first decoder 240a includes a first collaborative network 250a and a first prediction network 300a. In some configurations, the first collaborative network 250a is configured to receive as input the dense representation 350 generated by the first prediction network 300a and the second high-level feature representation 222 generated by the second encoder 220, and generate a final speech recognition 120a for the corresponding acoustic frame 110 at each output step. Here, the collaborative network 250 uses the second high-level feature representation 222 and the dense representation 350 to generate the final speech recognition result 120a.

[0041] 2A and 2B, the language ID predictor 230 uses more acoustic information (e.g., longer audio features) to generate a more accurate language predicted representation 232. Thus, to utilize all past acoustic frames 110, but still generate the language predicted representation 232 for each frame, the language ID predictor 230 uses non-parametric statistical pooling. That is, the language ID predictor 230 generates the first high-level feature representation 212 using the average (μ t ) and standard deviation (σ t) Noteworthy, the language ID predictor 230 determines the mean and standard deviation in a streaming manner, represented by:

number

[0042] In some implementations, the ASR model 200 jointly trains the first encoder 210, the second encoder 220, and the language ID predictor 230 on a set of multilingual training utterances, where a language ID target token is added as the first token of the corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances. The language ID target token identifies the language of the corresponding multilingual training utterance. That is, the set of multilingual training utterances can include training utterances in any number of different languages, and the language ID target token identifies the actual language (e.g., ground truth label) of the multilingual training utterance for training purposes.

[0043] During training, the training process generates a first loss for the first encoder 210 and a second loss for the second encoder 220, which are denoted by: L rnnt = lnP(y|x 1:T ) (3)

number

number

[0044] FIG. 3 illustrates that the prediction network 300 of either the first decoder 240a or the second decoder 240b (FIG. 2) receives as input a sequence of non-whitespace symbols, limited to the N previous non-whitespace symbols output by the final softmax layer of the collaborative network 250. That is, the sequence of non-whitespace symbols 120, 120a-n output by the final softmax layer of the collaborative network 250 may correspond to the non-whitespace symbols of the speech recognition result 120. Thus, the terms non-whitespace symbols 120 and sequence of speech recognition result 120 may be used interchangeably herein. In some examples, N is equal to 2. In other examples, N is equal to 5, although the disclosure is non-limiting and N may be equal to any integer. In some implementations, the prediction network 300 includes a multi-head attention mechanism 302, which shares a shared embedding matrix 304 across each head 302A-302H of the multi-head attention mechanism. In one example, the multi-head attention mechanism 302 includes four heads. However, any number of heads can be used with the multi-head attention mechanism 302. In particular, the multi-head attention mechanism significantly improves performance while minimizing the increase in model size. As explained in more detail below, each head 302A-H contains its own row of position vector 308, and rather than combining the outputs 318A-H from all heads, which would increase the model size, the outputs 318A-H are instead averaged by a head averaging module 322.

[0045] Referring to the first head 302A of the multi-head attention mechanism 302, the head 302A uses a shared embedding matrix 304 to derive a corresponding embedding 306, 306a-n (e.g.,

number

number

[0046] The corresponding embeddings generated by the shared embedding matrix 304 for each non-whitespace symbol 120 in the sequence of non-whitespace symbols 120 are the same for all heads 302A-H of the multi-head attention mechanism 302, but each head 302A-H defines a different set / rows of position vectors 308. For example, the first head 302A defines row PV Aa~An 308Aa to 308An, and the second head 302B defines a row PV Ba~Bn 308 Ba~Bn , and the H-th head 302H is a different row of the position vector PV Ha~Hn 308 Ha~Hn This section stipulates:

[0047] For each non-blank symbol in the received sequence of non-blank symbols 120, the first head 302A also weights the corresponding embedding 306, via the weighting layer 310, in proportion to the similarity between the corresponding embedding and the respective position vector PV 308 assigned thereto. In some examples, the similarity may include cosine similarity (e.g., cosine distance). In the example shown, the weighting layer 310 outputs a sequence of weighted embeddings 312, 312Aa-An, each associated with a corresponding embedding 306 being weighted in proportion to the respective position vector PV 308 assigned thereto. Stated otherwise, the weighted embedding 312 output by the weighting layer 310 for each embedding 306 may correspond to a dot product between the embedding 306 and the respective position vector PV 308. The weighted embedding 312 may be interpreted as overriding the embedding in proportion to how similar the embedding is to the positioning associated with the respective position vector PV 308. To increase computational speed, the prediction network 300 includes a non-recurrent layer, so that the sequence of weighted embeddings 312Aa-An are not combined, but instead averaged by a weighted average module 316 to produce a weighted average 318A of the weighted embeddings 312Aa-An as output from the first head 302A, expressed as follows:

number

[0048] In Equation 8, h represents the index of the head 302, n represents the position in the context, and e represents the embedding dimension. e contains the size of the corresponding dimension. The position vector PV 308 does not need to be trainable and may contain random values. In particular, even if the weighted embedding 312 is averaged, the position vector PV 308 can potentially preserve position history information, alleviating the need to provide recurrent connections at each layer of the prediction network 300.

[0049] The operations described above with respect to the first head 302A are similarly performed for each of the other heads 302B-H of the multi-head attention mechanism 302. Due to the different sets of positioned vectors PV 308 defined by each head 302, the weighting layer 310 outputs a sequence of weighted embeddings 312Ba-Bn, 312Ha-Hn for each of the other heads 302B-H that differs from the sequence of weighted embeddings 312Aa-Aa for the first head 302A. The weighted average module 316 then generates a weighted average 318B-H of each of the corresponding weighted embeddings 312 of the sequence of non-blank symbols as output from each of the other corresponding heads 302B-H.

[0050] In the illustrated example, the prediction network 300 includes a head averaging module 322 that averages the weighted averages 318A-H output from corresponding heads 302A-H. A projection layer 326 with SWISH can receive as an input an output 324 from the head averaging module 322 corresponding to the average of the weighted averages 318A-H and generate as an output a projection output 328. A final layer normalization 330 can normalize the projection output 328 to provide a single embedding vector 350 at a corresponding time step from the multiple time steps. The prediction network 300 generates only a single embedding vector 350 at each of the multiple time steps following the initial time step.

[0051] In some configurations, the prediction network 300 does not implement a multi-head attention mechanism 302 and only performs the operations described above with respect to the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-An is simply passed through the projection layer 326 and layer normalization 330 to provide a single embedding vector 350.

[0052] 4 is a flowchart of an exemplary arrangement of operations for a method 400 of streaming end-to-end multilingual speech recognition with collaborative language identification. The method 400 may be executed in data processing hardware 610 (FIG. 6) using instructions stored in memory hardware 620 (FIG. 6). The data processing hardware 610 and memory hardware 620 may reside in the user device 10 and / or the remote computing device 60 of FIG. 1.

[0053] At operation 402, the method 400 includes receiving a sequence of acoustic frames 110 as input to the ASR model 200. At operation 404, the method 400 includes generating, by a first encoder 210 of the ASR model 200 and in each of a plurality of output steps, a first high-level feature representation 212 for a corresponding acoustic frame 110 in the sequence of acoustic frames 110. At operation 406, the method 400 includes generating, by a second encoder 220 of the ASR model 200 and in each of a plurality of output steps, a second high-level feature representation 222 for the corresponding first high-level feature representation 212. At operation 408, the method 400 includes generating, by the language ID predictor 230 of the ASR model 200 and at each of the multiple output steps, a language predicted representation 232 based on a combination 231 of the first high-level feature representation 212 generated by the first encoder 210 at each of the multiple output steps and the second high-level feature representation 222 generated by the second encoder 220 at each of the multiple output steps. At operation 410, the method 400 includes generating, by the first decoder 240, 240a of the ASR model 200 and at each of the multiple output steps, a first probability distribution 120a over the possible speech recognition hypotheses based on a combination of the second high-level feature representation 222 generated by the second encoder 220 at each of the multiple output steps and the language predicted representation 232 generated by the language ID predictor 230 at each of the multiple output steps.

[0054] 5 is a flowchart of an exemplary arrangement of operations for a method 500 of streaming end-to-end multilingual speech recognition with collaborative language identification. The method 500 may be executed in data processing hardware 610 (FIG. 6) using instructions stored in memory hardware 620 (FIG. 6). The data processing hardware 610 and memory hardware 620 may reside in the user device 10 and / or the remote computing device 60 of FIG. 1.

[0055] At operation 502, the method 500 includes receiving a sequence of acoustic frames 110 as an input to the ASR model 200. At operation 504, the method 500 includes generating, by a first encoder 210 of the ASR model 200 and in each of a plurality of output steps, a first high-level feature representation 212 for a corresponding acoustic frame 110 in the sequence of acoustic frames 110. At operation 506, the method 500 includes generating, by a language ID predictor 230 of the ASR model 200 and in each of a plurality of output steps, a language prediction representation 232 for the corresponding first high-level feature representation 212. At operation 508, the method 500 includes generating, by the second encoder 220 of the ASR model 200 and at each of the multiple output steps, a second high-level feature representation 222 based on a combination 221 of the first high-level feature representation 212 generated by the first encoder 210 at each of the multiple output steps and the language predicted representation 232 generated by the language ID predictor 230 at each of the multiple output steps. At operation 510, the method 500 includes generating, by the first decoder 240a of the ASR model 200 and at each of the multiple output steps, a first probability distribution 120a over possible speech recognition hypotheses (e.g., final speech recognition results 120a) for the corresponding second high-level feature representation 222.

[0056] 6 is a schematic diagram of an exemplary computing device 600 that can be used to implement the systems and methods described in this document. The computing device 600 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0057] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be mounted on a common motherboard or may be mounted in other suitable manners. Processor 610 can process instructions for execution within computing device 600, including instructions stored in memory 620 or storage device 630, and display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 680 connected to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used as needed, along with multiple memories and multiple types of memories. Also, multiple computing devices 600 may be connected, with each device performing some of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0058] The memory 620 stores non-transient information within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0059] The storage device 630 can provide mass storage for the computing device 600. In some embodiments, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 can be a device array, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional embodiments, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0060] The high-speed controller 640 manages bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages lower bandwidth-intensive operations. Such an allocation of roles is merely exemplary. In some implementations, the high-speed controller 640 (e.g., via a graphics processor or accelerator) is coupled to memory 620, a display 680, and to a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to a storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690 can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet, etc.) that can be connected, for example, via a network adapter, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0061] The computing device 600 may be implemented in many different forms, as shown in the figure: for example, it may be implemented as a standard server 600a, or as a laptop computer 600b, or as part of a rack server system 600c, or multiple times in a group of such servers 600a.

[0062] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general purpose and may include implementations in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor coupled to receive data and instructions from the storage system and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0063] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0064] The processes and logic flows described herein may be executed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be executed by special purpose logic circuitry, such as special purpose logic circuitry, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, and any one or more processors of any kind of digital computer. In general, a processor will receive instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. In general, a computer also includes one or more mass storage devices, for example, magnetic disks, magnetic optical disks, or optical disks, for storing data, or is operatively coupled to one or more mass storage devices for receiving data, transmitting data, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0065] To provide for interaction with a user, aspects of the present disclosure may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, touch screen, for displaying information to a user, and, optionally, a keyboard and pointing device, e.g., a mouse or trackball, by which a user can provide input to the computer. Other types of devices may also be used to provide for interaction with a user, e.g., feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, a computer may interact with a user by sending and receiving documents to a device used by the user, e.g., by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0066] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A multilingual automatic speech recognition (ASR) model (200), comprising: A first encoder (210), receiving as input a sequence of acoustic frames (110); a first encoder (210) configured to generate, at each of a plurality of output steps, a first high-level feature representation (212) for a corresponding acoustic frame (110) in the sequence of acoustic frames (110); A second encoder (220), In each of the plurality of output steps, the first high-level feature representation (212) generated by the first encoder (210) is received as an input; a second encoder (220) configured to generate a second high-level feature representation (222) for a corresponding first high-level feature representation (212) in each of the plurality of output steps; A language identification (ID) predictor (230), comprising: receiving as input a combination (231) of the first high-level feature representation (212) generated by the first encoder (210) in each of the plurality of output steps and the second high-level feature representation (222) generated by the second encoder (220) in each of the plurality of output steps; a language identification (ID) predictor (230) configured to generate a language predicted representation (232) in each of the plurality of output steps; A first decoder, receiving as input a combination of the second high-level feature representation (222) generated by the second encoder (220) in each of the plurality of output steps and the language predicted representation (232) generated by the language ID predictor (230) in each of the plurality of output steps; a first decoder (240a) configured to generate, at each of the plurality of output steps, a first probability distribution (120a) over possible speech recognition hypotheses; A multilingual automatic speech recognition (ASR) model (200), comprising:

2. In each of the plurality of output steps, the first high-level feature representation (212) generated by the first encoder (210) is received as an input; 2. The ASR model of claim 1, further comprising a second decoder configured to generate a second probability distribution over possible speech recognition hypotheses at each of the plurality of output steps.

3. 3. The ASR model of claim 1, wherein the second decoder is further configured to generate a partial speech recognition result based on the second probability distribution over possible speech recognition hypotheses.

4. each of the first decoder (240a) and the second decoder (240b) includes a corresponding prediction network (300) followed by a corresponding collaboration network (250); the corresponding prediction networks (300) of the first decoder (240a) and the second decoder (240b) Long short-term memory (LSTM)-based prediction networks, or V2 Embedded Lookup Table and 4. The ASR model (200) of claim 2 or 3, wherein the corresponding joint networks (250) of the first decoder (240a) and the second decoder (240b) include the same structure.

5. The ASR model (200) of any one of claims 1 to 4, wherein the second encoder (220) generates the second high-level feature representation (222) without receiving any of the acoustic frames (110) as input.

6. The first encoder (210) Multiple one-way long short-term memory (LSTM) layers, Multiple conformer layers, or Multiple Transformer Layers The ASR model (200) of any one of claims 1 to 5, comprising a causal encoder comprising one of:

7. The second encoder (220) one or more bidirectional long short-term memory (LSTM) layers; Multiple conformer layers, or Multiple Transformer Layers The ASR model (200) of any one of claims 1 to 6, comprising a non-causal encoder comprising one of:

8. The first encoder (210), the second encoder (220), and the language ID predictor (230), generating a first loss for the first encoder (210); generating a second loss for the second encoder (220); generating a third loss for the language ID predictor (230); minimizing a weighted sum of the first loss, the second loss, and the third loss; The ASR model (200) of any one of claims 1 to 7, wherein the ASR model (200) is jointly trained on a set of multilingual training utterances by

9. 9. The ASR model of claim 8, wherein a language ID target token is added as a first token of a corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances, the language ID target token identifying a language of the corresponding multilingual training utterance.

10. 10. The ASR model (200) of claim 8 or 9, wherein a language ID target token is added to each location where a code-switch occurs in a corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances.

11. A computer-implemented method (400) that, when executed on data processing hardware (610), causes the data processing hardware (620) to perform operations, the operations comprising: receiving a sequence of acoustic frames (110) as input to an automatic speech recognition (ASR) model (200); generating, in each of a plurality of output steps, a first encoder (210) of the ASR model (200) for a corresponding acoustic frame (110) in the sequence of acoustic frames (110); generating, in each of a plurality of output steps, a second high-level feature representation (222) for a corresponding first high-level feature representation (212) by a second encoder (220) of the ASR model (200); generating, by a language identification (ID) predictor (230) of the ASR model (200), a language predicted representation (232) based on a combination (231) of the first high-level feature representation (212) generated by the first encoder (210) in each of the plurality of output steps and the second high-level feature representation (222) generated by the second encoder (220) in each of the plurality of output steps; generating, by a first decoder (240a) of the ASR model (200), a first probability distribution (120a) over possible speech recognition hypotheses based on a combination of the second high-level feature representation (222) generated by the second encoder (220) in each of the plurality of output steps and the language prediction representation (232) generated by the language ID predictor (230) in each of the plurality of output steps; A computer-implemented method (400) comprising:

12. 12. The computer-implemented method (400) of claim 11, wherein the operations further include generating, by a second decoder (240b) of the ASR model (200), in each of the plurality of output steps, a second probability distribution (120b) over possible speech recognition hypotheses based on the first high-level feature representation (212) generated by the first encoder (210) in each of the plurality of output steps.

13. 13. The computer-implemented method (400) of claim 12, wherein the operations further comprise generating, by the second decoder (240b), a partial speech recognition result based on the second probability distribution (120b) over possible speech recognition hypotheses.

14. each of the first decoder (240a) and the second decoder (240b) includes a corresponding prediction network (300) followed by a corresponding collaboration network (250); the corresponding prediction networks (300) of the first decoder (240a) and the second decoder (240b) Long short-term memory (LSTM)-based prediction networks, or V2 Embedded Lookup Table and 14. The computer-implemented method (400) of claim 12 or 13, wherein the corresponding collaborative networks (250) of the first decoder (240a) and the second decoder (240b) comprise the same structure.

15. The computer-implemented method (400) of any one of claims 11 to 14, wherein the second encoder (220) generates the second high-level feature representation (220) without receiving any of the acoustic frames (110) as input.

16. The first encoder (210) Multiple one-way long short-term memory (LSTM) layers, Multiple conformer layers, or Multiple Transformer Layers The computer-implemented method (400) of any one of claims 11 to 15, further comprising a causal encoder comprising one of:

17. The second encoder (220) one or more bidirectional long short-term memory (LSTM) layers; Multiple conformer layers, or Multiple Transformer Layers The computer-implemented method (400) of any one of claims 11 to 16, further comprising a non-causal encoder comprising one of:

18. The operation, generating a first loss for the first encoder (210); generating a second loss for the second encoder (220); generating a third loss for the language ID predictor (230); and minimizing a weighted sum of the first loss, the second loss, and the third loss, thereby and further comprising jointly training the first encoder (210), the second encoder (220), and the language ID predictor (230) on a set of multilingual training utterances. A computer implemented method (400) according to any one of claims 11 to 17.

19. the operations further include adding a language ID target token as a first token of a corresponding ground truth transcription of each training utterance in the set of multilingual training utterances; 20. The computer-implemented method of claim 18, wherein the language ID target token identifies a language of the corresponding training utterance.

20. 20. The computer-implemented method (400) of claim 18 or 19, wherein the operations further comprise adding a language ID target token to each position where a code-switch occurs in a corresponding ground truth transcription of each multilingual training utterance in the set of multilingual training utterances.

Images