Streaming end-to-end multi-language speech recognition with federated language identification
The multilingual ASR model addresses the limitation of single-language ASR systems by integrating encoders and predictors to provide simultaneous transcription and language prediction, enhancing accuracy and reducing costs through a unified model architecture.
Patent Information
- Application Number
- CN202380036081.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-25
- Filing Date
- 2023-03-23
- Publication Date
- 2024-12-06
AI Technical Summary
Existing automatic speech recognition (ASR) models are typically limited to a single language, failing to meet the needs of bilingual speakers who require accurate and low-latency transcription across multiple languages.
A multilingual ASR model is developed that integrates a first encoder, a second encoder, a language identification predictor, and decoders to generate both transcription and language prediction representations simultaneously, without relying on oracle language identification, allowing for streaming and non-streaming modes of operation.
The model achieves accurate and low-latency multilingual speech recognition by improving initial transcription results with additional context, reducing computational and storage costs by integrating language identification within a single model.
Smart Images

Figure CN119096292A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to streaming end-to-end multi-language speech recognition with joint language identification. Background Art
[0002] Automatic speech recognition (ASR), the process of obtaining audio input and transcribing it into text, has become an important technology used in mobile devices and other devices to a large extent. In general, automatic speech recognition attempts to provide accurate transcription of what a person says by obtaining audio input (e.g., voice utterances) and transcribing the audio input into text. Based on the continuous development of deep neural networks, new ASR models are constantly improving in terms of accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay between the client speaking and transcription). Although a large number of people are bilingual, most ASR models are only compatible with a single language. Therefore, for a large number of bilingual speakers, an ASR model that is compatible with several different languages while still maintaining the accuracy and latency performance indicators of the new ASR model will be desirable. Summary of the invention
[0003] One aspect of the present disclosure provides a multilingual automated speech recognition (ASR) model. The ASR model includes a first encoder, which is configured to receive an acoustic frame sequence as input, and generate a first high-order feature representation for a corresponding acoustic frame in the acoustic frame sequence at each of a plurality of output steps. The ASR model also includes a second encoder, which is configured to receive a first high-order feature representation generated by the first encoder at each of a plurality of output steps as input, and generate a second high-order feature representation for the corresponding first high-order feature representation at each of a plurality of output steps. The ASR model also includes a language identification (ID) predictor, which is configured to receive a concatenation of a first high-order feature representation generated by the first encoder at each of a plurality of output steps and a second high-order feature representation generated by the second encoder at each of a plurality of output steps as input, and generate a language prediction representation at each of a plurality of output steps. The ASR model also includes a first decoder configured to receive as input a concatenation of a second high-order feature representation generated by the second encoder at each of a plurality of output steps and a language prediction representation generated by the language ID predictor at each of a plurality of output steps, and to generate a first probability distribution over possible speech recognition prediction hypotheses at each of the plurality of output steps.
[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the ASR model includes a second decoder, which is configured to: receive a first high-order feature representation generated by the first encoder at each of a plurality of output steps as input, and generate a second probability distribution over possible speech recognition prediction texts at each of a plurality of output steps. Here, the second decoder may also be configured to generate a partial speech recognition result based on a second probability distribution over possible speech recognition prediction texts. In these implementations, the first decoder and the second decoder may each include a corresponding prediction network, followed by a corresponding joint network, the corresponding prediction networks of the first decoder and the second decoder having the same structure including a prediction network based on a long short-term memory (LSTM) or one of a V2 embedding lookup table, and the corresponding joint networks of the first decoder and the second decoder include the same structure.
[0005] In some examples, the second encoder generates the second high-order feature representation without receiving any of the acoustic frames as input. The first encoder includes a causal encoder having one of a plurality of unidirectional long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers. In some implementations, the second encoder includes a non-causal encoder having one of one or more bidirectional long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers.
[0006] In some implementations, the first encoder, the second encoder, and the language ID predictor are jointly trained on the multilingual training utterance set by: generating a first loss for the first encoder; generating a second loss for the second encoder; generating a third loss for the language ID predictor; and minimizing a weighted sum of the first loss, the second loss, and the third loss. In these implementations, a language ID target word-gram may be added as a first word-gram of a corresponding ground-truth transcription of each multilingual training utterance in the multilingual training utterance set. The language ID target word-gram identifies the language of the corresponding multilingual training utterance. Alternatively, the language ID target word-gram may be added to each position where a code-switch occurs in the corresponding ground-truth transcription of each multilingual training utterance in the multilingual training utterance set.
[0007] Another aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for streaming end-to-end multilingual speech recognition using joint language identification. The operations include receiving a sequence of acoustic frames as input to an automatic speech recognition (ASR) model. The operations also include generating, by a first encoder of the ASR model, a first high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames at each of a plurality of output steps. The operations also include generating, by a second encoder of the ASR model, a second high-order feature representation for a corresponding first high-order feature representation at each of a plurality of output steps. The operations also include generating, by a language identification (ID) predictor of the ASR model, a language prediction representation at each of a plurality of output steps. Here, the language prediction representation is based on a concatenation of a first high-order feature representation generated by the first encoder at each of a plurality of output steps and a second high-order feature representation generated by the second encoder at each of a plurality of output steps. The operations also include generating, by a first decoder of the ASR model, a first probability distribution over possible speech recognition prediction text at each of a plurality of output steps. Here, the first probability distribution over possible speech recognition prediction text is based on a concatenation of a second high-order feature representation generated by the second encoder at each of the plurality of output steps and a language prediction representation generated by the language ID predictor at each of the plurality of output steps.
[0008] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operation also includes generating a second probability distribution on possible speech recognition prediction texts by a second decoder of the ASR model at each of a plurality of output steps. The second probability distribution on possible speech recognition prediction texts is based on a first high-order feature representation generated by a first encoder at each of a plurality of output steps. In these implementations, the operation may also include generating a partial speech recognition result by a second decoder based on a second probability distribution on possible speech recognition prediction texts. The first decoder and the second decoder may each include a corresponding prediction network, followed by a corresponding joint network, the corresponding prediction networks of the first decoder and the second decoder having the same structure including a prediction network based on a long short-term (LSTM) or one of a V2 embedding lookup table, and the corresponding joint networks of the first decoder and the second decoder include the same structure.
[0009] In some examples, the second encoder generates the second high-order feature representation without receiving any of the acoustic frames as input. In some implementations, the first encoder includes a causal encoder comprising one of a plurality of unidirectional long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers. The second encoder includes a non-causal encoder comprising one of one or more bidirectional long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers.
[0010] In some examples, the operations also include jointly training the first encoder, the second encoder, and the language ID predictor on the set of multilingual training utterances by: generating a first loss for the encoder; generating a second loss for the second encoder; generating a third loss for the language ID predictor; and minimizing a weighted sum of the first loss, the second loss, and the third loss. In these examples, the operations may also include adding a language ID target word as a first word of the corresponding ground-truth transcription of each multilingual training utterance in the set of multilingual training utterances. The language ID target word identifies the language of the corresponding multilingual training utterance. Alternatively, the operations also include adding the language ID target word to each position where a code switch occurs in the corresponding ground-truth transcription of each multilingual training utterance in the set of multilingual training utterances.
[0011] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 is a schematic diagram of an example speech recognition system.
[0013] Figure 2A and Figure 2B is a diagram of an example speech recognition model.
[0014] Figure 3 yes Figure 2A and Figure 2B Diagram of the example prediction network for the example speech recognition model.
[0015] Figure 4 and Figure 5 is a flow diagram of an example arrangement of operations of a computer-implemented method of streaming end-to-end multilingual speech recognition with joint language identification.
[0016] Figure 6 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.
[0017] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION
[0018] End-to-end (E2E) automatic speech recognition (ASR) models are traditionally structured to operate in either streaming mode or non-streaming mode. Conventionally, E2E ASR models include an encoder and a decoder as the main components. Applications involving end-user interaction, such as voice search or dictation on a device, may require the model to perform recognition in a streaming manner. Here, performing recognition in a streaming manner means that the ASR model outputs each word of the utterance with as little delay as possible as the word is spoken. Other applications, such as offline video captioning, do not require the model to be streaming and can use subsequent context to improve performance.
[0019] In some implementations, the E2E ASR model is configured to recognize speech from multiple languages (e.g., an E2E multilingual ASR model). Here, although the multilingual ASR model and the language identification model are often used together in downstream tasks (e.g., code switching and speech translation), the models are built and executed separately. Thus, using both an independent multilingual ASR model and an independent language identification model unnecessarily complicates the overall ASR system by increasing computational and storage costs.
[0020] Therefore, the implementation of this article relates to an E2E multilingual ASR model that generates a transcription and a language prediction representation together. The ASR model includes: a first encoder, the first encoder is configured to generate a first high-order feature representation for a corresponding acoustic frame, and a second encoder, the second encoder is configured to generate a second high-order feature representation for the corresponding first high-order feature representation. The ASR model also includes a language identification predictor, the language identification predictor is configured to receive the concatenation of the first high-order feature representation and the second high-order feature representation and generate a language prediction representation. The ASR model also includes a first decoder, the first decoder is configured to receive the concatenation of the second high-order feature representation and the language prediction representation and generate a first probability distribution over possible speech recognition prediction text (e.g., transcription). Therefore, the ASR model (e.g., a single E2E model) outputs both the transcription and the language prediction representation together. It is worth noting that the ASR model never receives or uses any oracle language identification information to generate the transcription and the language prediction representation. As will become apparent, the ASR model operates in a streaming or non-streaming mode to produce the transcription and the language prediction representation.
[0021] Figure 1is an example of a speech environment 100. In the speech environment 100, the way in which a user 104 interacts with a computing device (such as a user device 10) may be through voice input. The user device 10 (also generally referred to as the device 10) is configured to capture sound (e.g., streaming audio data) from one or more users 104 within the speech environment 100. Here, the streaming audio data may refer to the spoken utterance 106 of the user 104, which serves as an audible query, a command to the user device 10, or an audible communication captured by the device 10. The voice-enabled system of the user device 10 can respond to the query or command by answering the query and / or causing the command to be executed / fulfilled by one or more downstream applications.
[0022] The user device 10 may correspond to any computing device associated with the user 104 and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, mobile devices (e.g., smart watches), smart home appliances, Internet of Things (IoT) devices, in-vehicle infotainment systems, smart displays, smart speakers, and the like. The user device 10 includes data processing hardware 12 and memory hardware 14 that communicates with the data processing hardware 12 and stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 also includes an audio system 16 having an audio capture device (e.g., a microphone) 16, 16a for capturing spoken speech 106 in the speech environment 100 and converting the spoken speech into electrical signals, and a speech output device (e.g., a speaker) 16, 16b for communicating using audible audio signals (e.g., as output data from the user device 10). While the user device 10 may implement an array of audio capture devices 16 a without departing from the scope of the present disclosure, one or more of the capture devices 16 a in the array may not physically reside on the user device 10 but may be in communication with the audio system 16 .
[0023] In the speech environment 100, an automated speech recognition (ASR) system 118 implements an ASR model 200 and resides on a user device 10 of a user 104 and / or resides on a remote computing device 60 (e.g., one or more remote servers of a distributed system executed in a cloud computing environment) that communicates with the user device 10 via a network 40. In some examples, the ASR model 200 may be a recurrent neural network transducer (RNN-T) model. The user device 10 and / or the remote computing device 60 also includes an audio subsystem 108 that is configured to receive an utterance 106 spoken by the user 104 and captured by the audio capture device 16a, and convert the utterance 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by the ASR system 118. In the example shown, the user speaks a corresponding utterance 106, and the audio subsystem 108 converts the utterance 106 into corresponding audio data (e.g., a sequence of acoustic frames) 110 for input to the ASR system 118. Thereafter, the ASR model 200 receives as input a sequence of acoustic frames 110 corresponding to the utterance 106, and generates / predicts a corresponding transcription 120 (e.g., speech recognition result / predicted text) of the utterance 106 at each output step as the ASR model receives (e.g., processes) each acoustic frame 110 in the sequence of acoustic frames 110.
[0024] In the example shown, the ASR model 200 can perform streaming speech recognition to produce initial speech recognition results 120, 120b, and generate final speech recognition results 120, 120a by improving the initial speech recognition results 120b. The speech recognition result 120 can correspond to a partial speech recognition result or an entire speech recognition result. In other words, the speech recognition result 120 can correspond to a portion of the speech 106 or the entire speech 106. For example, a partial speech recognition result can correspond to a portion of a spoken speech or even a portion of a spoken term. However, as will become apparent, the ASR model 200 performs additional processing on the final speech recognition result 120a, whereby the final speech recognition result 120a may be delayed from the initial speech recognition result 120b.
[0025] The user device 10 and / or the remote computing device 60 also executes a user interface generator 107 that is configured to present a representation of a transcription 120 of the utterance 106 to a user 104 of the user device 10. As described in more detail below, the user interface generator 107 may display an initial speech recognition result 120b in a streaming manner during time 1, and then display a final speech recognition result 120a in a streaming manner during time 2. Notably, the ASR model 200 outputs the final speech recognition result 120a in a streaming manner even if the final speech recognition result 120a improves upon the initial speech recognition result 120b. In some configurations, the transcription 120 output from the ASR system 118 is processed, for example, by a natural language understanding (NLU) module executing on the user device 10 or the remote computing device 60 to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of user device 10 or remote computing device 60) may convert transcription 120 into synthesized speech for audible output by user device 10 and / or another device.
[0026] In the example shown, the user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) of the user device 10 using the ASR system 118. For example, Figure 1 A user 104 is depicted communicating with a digital assistant application 50 and the digital assistant application 50 displays a digital assistant interface 18 on a screen of a user device 10 to depict a conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50 "What time is the concert tonight?" This question from the user 104 is a spoken utterance 106 that is captured by an audio capture device 16a and processed by an audio system 16 of the user device 10. In this example, the audio system 16 receives the spoken utterance 106 and converts it into a sequence of acoustic frames 110 for input to an ASR system 118.
[0027] Continuing with the example, the ASR model 200 encodes the acoustic frame sequence 110 upon receiving the acoustic frame sequence 110 corresponding to the utterance 106 when the user 104 is speaking and then decodes the encoded acoustic frame sequence 110 into an initial speech recognition result 120b. During time 1, the user interface generator 107 presents a representation of the initial speech recognition result 120b of the utterance 106 to the user 104 of the user device 10 in a streaming manner via the digital assistant interface 18, so that once the words, word pieces, and / or individual characters are spoken, they appear on the screen. In some examples, the first look ahead audio context is equal to zero.
[0028] During time 2, the user interface generator 107 presents a representation of the final speech recognition result 120a of the utterance 106 to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming manner so that the words, word chunks, and / or individual characters appear on the screen as they are generated by the ASR model 200. In some implementations, the user interface generator 107 replaces the representation of the initial speech recognition result 120b presented at time 1 with the representation of the final speech recognition result 120a presented at time 2. Here, time 1 and time 2 may include timestamps corresponding to when the user interface generator 107 presented the corresponding speech recognition results 120. In this example, the timestamp at time 1 indicates that the user interface generator 107 presented the initial speech recognition result 120b at an earlier time than the final speech recognition result 120a. For example, since the final speech recognition result 120a is considered to be more accurate than the initial speech recognition result 120b, the final speech recognition result 120a that is ultimately displayed as the transcription 120 can repair any words that may be misrecognized in the initial speech recognition result 120b. In this example, the streaming initial speech recognition result 120b output by the ASR model 200 is displayed on the screen of the user device 10 at time 1, is associated with low latency, and provides responsiveness to the user 104 whose query is being processed, while the final speech recognition result 120a output by the ASR model 200 and displayed on the screen at time 2 utilizes additional speech recognition models and / or language models to improve the quality of speech recognition in terms of accuracy, but increases latency. However, since the initial speech recognition result 120b is displayed when the user speaks the speech 106, the higher latency associated with generating and ultimately displaying the final speech recognition result 120a is not noticeable to the user 104.
[0029] exist Figure 1In the example shown, the digital assistant application 50 can use natural language processing to respond to questions asked by the user 104. Natural language processing generally refers to the process of interpreting written language (e.g., the initial speech recognition result 120b and / or the final speech recognition result 120a) and determining whether the written language prompts any action. In this example, the digital assistant application 50 uses natural language processing to recognize that the question from the user 104 is about the user's schedule and more specifically about the concert on the user's schedule. By using natural language processing to recognize these details, the automated assistant returns a response 19 to the user's query, where the response 19 states "Venue doors open at 6:30PM and concertstarts at8pm." In some configurations, natural language processing occurs on a remote server 60 that communicates with the data processing hardware 12 of the user device 10.
[0030] Reference now Figure 2A and Figure 2B In some embodiments, the ASR model 200 includes a cascade encoder 204, a language identification (ID) predictor 230, and a decoder 240. Here, the first decoder 240, 240a is configured to improve the initial speech recognition result 120b output by the second decoder 240, 240b. The second decoder 240b can be operated in a streaming manner so that the second decoder 240b is configured to generate a partial speech recognition result corresponding to the initial speech recognition result 120b. On the other hand, the first decoder 240a improves the partial speech recognition result by receiving an additional right-context and generating a final speech recognition result 120a. The first decoder 240a and the second decoder 240b each include a corresponding prediction network 300, followed by a corresponding joint network 250. Here, the first prediction network 300, 300a and the first joint network 250, 250a correspond to the first decoder 240a, and the second prediction network 300, 300b and the second joint network 250, 250b correspond to the second decoder 240b. The prediction networks 300a, 300b have the same structure including a prediction network based on long short-term memory (LSTM) or one of the V2 embedding lookup tables. In addition, the corresponding joint networks 250a, 250b have the same structure. However, although the component structures of the first decoder 240a and the second decoder 240b are the same, the corresponding components of each decoder 240 are unique and can be trained independently of the components of another decoder 240.
[0031] A cascaded encoder 204 refers to a model structure in which the encoding path includes two encoders 210, 220 in cascade such that the output of the first encoder 210 is fed to the input of the second encoder 220 before decoding. Here, regardless of the underlying architecture of each encoder, the first encoder 210 and the second encoder 220 can be cascaded. The encoders 210, 220 can each include a stacked multi-head attention layer. In some examples, the first encoder 210 includes a causal encoder having one of a plurality of unidirectional (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers. For example, the first encoder 210 can include twelve (12) conformer layers, each having a multi-head (e.g., eight (8) heads) self-attention mechanism and a convolution kernel size of fifteen (15). In addition, the first encoder 210 can perform a cascade operation after the third conformer layer to achieve a time reduction rate of two, whereby the resulting 1024-dimensional vector is transformed by the fourth conformer layer and then projected back to a 512-dimensional vector using another linear transformation. Thereafter, another eight (8) conformer layers are followed by a final normalization layer. Thus, the first encoder 210 may include 110 million parameters. Each layer of the first encoder 210 receives zero context (eg, receives zero subsequent acoustic frames).
[0032] The second encoder 220 includes a non-causal encoder, which includes one of one or more bidirectional LSTM layers, multiple conformer layers, or multiple transformer layers. For example, the second encoder 220 may include a 512-dimensional linear projection to transform the input features, followed by five (5) 512-dimensional conformer layers and a final linear normalization layer, thereby generating 50 million parameters. Here, the second encoder 220 may receive additional context, for example, a total of fifteen (15) frames of context, whereby each conformer layer receives three (3) frames of context.
[0033] Continue to refer Figure 2A and Figure 2B , the first encoder 210 receives a d-dimensional feature vector sequence (eg, the acoustic frame sequence 110) x = (x1, x2, ..., x T ),in And at each output step, a first high-order feature representation 212 is generated for the corresponding acoustic frame 110 in the acoustic frame sequence 110. Similarly, the second encoder 220 is cascade-connected to the first encoder 210, and receives the first high-order feature representation 212 as input, and generates a second high-order feature representation 222 for the corresponding first high-order feature representation 212 at each output step. In some cases, the second encoder 220 generates the second high-order feature representation 222 without receiving any of the acoustic frames 110 as input. In these cases, the second encoder 220 generates the second high-order feature representation 222 using only the first high-order feature representation 212 as input. Therefore, the first high-order feature representation 212 output from the first encoder 210 is fed to the language ID predictor 230 and the second decoder 240b, while the second high-order feature representation 222 output from the second encoder 220 is fed to the first decoder 240a and the language ID predictor 230. In some configurations, only the second high-order feature representation 222 output from the second encoder 220 is fed to the language ID predictor 230 ( Figure 2B ).
[0034] Continue to refer Figure 2A and Figure 2B , the second decoder 240b includes a second joint network 250b and a second prediction network 300b. The second joint network 250b is configured to receive the dense representation 350 generated by the second prediction network 300b and the first high-order feature representation 212 generated by the first encoder 210 as input, and generate an initial speech recognition result 120b for the corresponding acoustic frame 110 at each output step. Here, the second joint network 250b uses the first high-order feature representation 212 and the dense representation 350 to generate the initial speech recognition result 120b. The second decoder 240b operates in a streaming manner, so that the second decoder 240b allows the initial speech recognition result 120b to correspond to the partial speech recognition result.
[0035] In some implementations, the initial speech recognition result 120b includes a second probability distribution on possible speech recognition prediction text. Therefore, in this article, the initial speech recognition result 120b can be used interchangeably with the second probability distribution 120b on possible speech recognition prediction text. Therefore, the second joint network 250b can generate a second probability distribution 120b on possible speech recognition prediction text at each output step (e.g., time step). Here, "possible speech recognition prediction text" corresponds to a set of output tags / symbols (also referred to as "speech units") of graphemes (symbols / characters) or word blocks of a specified natural language. For example, when the natural language is English, the output tag set may include twenty-seven (27) symbols, for example, one tag for each of the 26 letters in the English alphabet and a tag for a specified space. Accordingly, the second joint network 250b can output a value set indicating the possibility of each output tag appearing in a predetermined output tag set. The value set can be a vector (e.g., a unique hot vector) and can indicate a second probability distribution on the output tag set. In some scenarios, the output labels are graphemes (e.g., single characters, and potential punctuation and other symbols), but the output label set is not so limited. For example, in addition to or in place of graphemes, the output label set may include chunks and / or entire words. The output labels may also be other types of speech units, such as phonemes or subphonemes. The second probability distribution 120b of the second joint network 250b may include a posterior probability value for each of the different output labels. Therefore, if there are 100 different output labels representing different graphemes or other symbols, the output of the joint network 250 may include 100 different probability values, one for each output label. The second probability distribution 120b may then be used to select candidate orthographic elements (e.g., graphemes, chunks and / or words) and assign scores to them in a beam search process (e.g., by the final Softmax layer (not shown) of the second joint network 250b) to determine the initial speech recognition result 120b. For example, the second joint network 250b may select the N-best possible speech recognition prediction text with the highest probability as the output of the initial speech recognition result 120b.
[0036] In some implementations, the second prediction network 300b receives as input the non-blank symbol sequence output by the final softmax layer of the second joint network 250b, and generates a dense representation 350 at each output step. That is, the second joint network 250b receives the dense representation 350 of the previous initial speech recognition result 120b, and uses the dense representation 350 to generate a subsequent initial speech recognition result 120b.
[0037] Now specific reference Figure 2AIn some configurations, the language ID predictor 230 of the example ASR model 200, 200a is configured to receive as input the first high-order feature representation 212 generated by the first encoder 210 at each of the plurality of output steps and the second high-order feature representation 222 generated by the second encoder 220 at each of the plurality of output steps. In addition, the language ID predictor 230 may generate a concatenation 231 of the first high-order feature representation 212 and the second high-order feature representation 222. Thereafter, the language ID predictor 230 is further configured to generate a language prediction representation 232 based on the concatenation 231 of the first high-order feature representation 212 and the second high-order feature representation 222 at each of the plurality of output steps. Advantageously, upon receiving the concatenation 231, the language ID predictor 230 generates the language prediction representation 232 using a plurality of inputs.
[0038] The language prediction representation 232 indicates the corresponding language of the spoken utterance. For example, since the ASR model 200 is a multilingual ASR model, the spoken utterance can be in any number of languages. Therefore, using the concatenation 231, the language ID predictor 230 can predict the corresponding language of the spoken utterance. The language prediction representation 232 can be used for downstream tasks (e.g., code switching or speech translation) and / or to improve speech recognition results. That is, the first decoder 240a can use the language prediction representation 232 to improve the initial speech recognition result 120b generated by the second decoder 240b. In some examples, the language ID predictor 230 generates the language prediction representation 232 on a frame-by-frame basis. In these examples, the spoken utterance can include multiple utterances, and the language ID predictor 230 generates the language prediction representation 232 for each acoustic frame 110 in the acoustic frame sequence 110. For example, for a first portion of the acoustic frame sequence, the language prediction representation 232 can indicate that a first language is spoken, and for a second portion of the acoustic frame sequence, the language prediction representation 232 indicates that a second language is spoken.
[0039] Continue to refer Figure 2A, the first decoder 240a includes a joint first network 250a and a first prediction network 300a. In some configurations, the first joint network 250a is configured to receive as input the dense representation 350 generated by the first prediction network 300a, the second high-order feature representation 222 generated by the second encoder 220, and the language prediction representation 232 generated by the language ID predictor 230, and generate a final speech recognition 120a for the corresponding acoustic frame 110 at each output step. Here, the joint network 250 uses the second high-order feature representation 222, the language prediction representation 232, and the dense representation 350 to generate the final speech recognition result 120a. In some examples, the first decoder 240a generates a concatenation of the second high-order feature representation 222 and the language prediction representation 232, and uses the concatenation to generate the final speech recognition result 120a.
[0040] In some implementations, the final speech recognition result 120a includes a first probability distribution on possible speech recognition prediction texts. Therefore, in this article, the final speech recognition result 120a can be used interchangeably with the first probability distribution 120a on possible speech recognition prediction texts. Therefore, the first joint network 250a can generate a first probability distribution 120a on possible speech recognition prediction texts at each output step (e.g., time step). Here, "possible speech recognition prediction texts" correspond to a set of output labels / symbols (also referred to as "speech units") that each represent a specified natural language grapheme (symbol / character) or word block. For example, when the natural language is English, the output label set may include twenty-seven (27) symbols, for example, one label for each of the 26 letters in the English alphabet and one label for a specified space. Accordingly, the first joint network 250a can output a value set indicating the possibility of each output label in a predetermined output label set appearing. The value set can be a vector (e.g., a one-hot vector) and can indicate a first probability distribution on the output label set. In some scenarios, the output labels are graphemes (e.g., single characters, and potential punctuation and other symbols), but the output label set is not so limited. For example, in addition to or in place of graphemes, the output label set may include chunks and / or entire words. The output labels may also be other types of speech units, such as phonemes or subphonemes. The first probability distribution 120a of the first joint network 250a may include a posterior probability value for each of the different output labels. Therefore, if there are 100 different output labels representing different graphemes or other symbols, the output of the first joint network 250a may include 100 different probability values, one for each output label. The first probability distribution 120a can then be used to select candidate orthographic elements (e.g., graphemes, chunks and / or words) and assign scores to them in a beam search process (e.g., by the final Softmax layer (not shown) of the first joint network 250a) to determine the final speech recognition result 120a. For example, the first joint network 250a may select the N-best possible speech recognition prediction text with the highest probability as the output of the final speech recognition result 120a.
[0041] In some implementations, the first prediction network 300a receives as input the non-blank symbol sequence output by the final softmax layer of the first joint network 250a, and generates a dense representation 350 at each output step. That is, the first joint network 250a receives the dense representation 350 of the previous final speech recognition result 120a, and uses the dense representation 350 to generate the subsequent final speech recognition result 120a.
[0042] Now specific reference Figure 2BIn some configurations, the language ID predictor 230 of the example ASR model 200, 200b is configured to receive as input the first high-order feature representation 212 generated by the first encoder 210 at each of the plurality of output steps and to generate a language prediction representation 232 at each of the plurality of output steps. Here, the language ID predictor 230 generates the language prediction representation 232 based solely on the first high-order feature representation 212, and outputs the language prediction representation 232 to the second encoder 220. Advantageously, in such a configuration, both the first decoder 240a and the second encoder 220 benefit from using the language prediction representation 232.
[0043] Therefore, in these configurations, the second encoder 220 is also configured to receive the first high-order feature representation 212 and the language prediction representation 232 as input, and generate a second high-order feature representation 222 for the corresponding first high-order feature representation 212 at each output step. The second encoder 220 can generate a concatenation 221 of the first high-order feature representation 212 and the language prediction representation 232, and generate the second high-order feature representation based on the concatenation 221. In some cases, the second encoder 220 generates the second high-order feature representation 222 without receiving any of the acoustic frames 110 as input. In these cases, the second encoder 220 generates the second high-order feature representation 222 using only the first high-order feature representation 212 and the language prediction representation 232.
[0044] Continue to refer Figure 2B , the first decoder 240a includes a joint first network 250a and a first prediction network 300a. In some configurations, the first joint network 250a is configured to receive as input the dense representation 350 generated by the first prediction network 300a and the second high-order feature representation 222 generated by the second encoder 220, and generate a final speech recognition 120a for the corresponding acoustic frame 110 at each output step. Here, the joint network 250 uses the second high-order feature representation 222 and the dense representation 350 to generate the final speech recognition result 120a.
[0045] Reference again Figure 2A and Figure 2B In both cases, the language ID predictor 230 uses more acoustic information (e.g., longer audio features) to generate a more accurate language prediction representation 232. Therefore, in order to utilize all past acoustic frames 110 but still generate the language prediction representation 232 on a frame-by-frame basis, the language ID predictor 230 uses non-parametric statistical pooling. That is, the language ID predictor 230 converts the first high-order feature representation 212 into the mean (μ ) of the first high-order feature representation 212. t ) and standard deviation (σ t). It is worth noting that the language ID predictor 230 determines the mean and standard deviation in a streaming manner, expressed as:
[0046]
[0047] In equations 1 and 2, h1 represents the first high-order feature representation 212. When the first high-order feature representation 212 is converted into a concatenated vector [μ t ; σ t ] After that, the language ID predictor 230 uses two fully connected layers followed by a softmax output layer to convert the concatenated vectors into a language prediction representation 232. Therefore, the frame-synchronized language ID predictor 230 is efficient to operate in a streaming manner and requires only a small amount of computational cost during execution.
[0048] In some implementations, the ASR model 200 jointly trains the first encoder 210, the second encoder 220, and the language ID predictor 230 on a set of multilingual training utterances. Here, a language ID target word-gram is added as the first word-gram of the corresponding ground-truth transcription of each multilingual training utterance in the set of multilingual training utterances. The language ID target word-gram identifies the language of the corresponding multilingual training utterance. That is, the set of multilingual training utterances can include training utterances in any number of different languages, and the language ID target word-gram identifies the actual language (e.g., ground-truth label) of the multilingual training utterance for training purposes.
[0049] During training, the training process generates a first loss for the first encoder 210 and a second loss for the second encoder 220, expressed as:
[0050]
[0051]
[0052] In equations 3 and 4, represents the loss of the decoder 240 (e.g., a recurrent neural network transducer loss), x represents the acoustic frame sequence 110, and y represents the transcription 120. The ASR model 200 uses two separate decoders 240, and therefore, the training loss of the ASR model 200 is expressed as
[0053]
[0054] In equation 5, represents the loss of the first decoder 240a, represents the loss of the second decoder 240b, λ represents the weighting factor of the loss of the first decoder 240a, and (1-λ) represents the weighting factor of the loss of the second decoder 240b. In addition, the training process generates a third loss for the language ID predictor 230, which is expressed as:
[0055]
[0056] In equation 6, represents the third loss of the language ID predictor 230, and l t The one-hot language prediction of t represents the label. Therefore, the training process trains the ASR model 200 using the final training loss according to the following:
[0057]
[0058] In Equation 7, α is a scalar weight of the loss of the language ID predictor 230. Therefore, the training process trains the ASR model 200 by minimizing the weighted sum of the first loss, the second loss, and the third loss.
[0059] Figure 3 The prediction network 300 of the first decoder 240a or the second decoder 240b (Fig. 2) is shown to receive a non-blank symbol sequence as input, which is limited to the N previous non-blank symbols output by the final Softmax layer of the joint network 250. That is, the non-blank symbol sequence 120, 120a to 120n output by the final Softmax layer of the joint network 250 can correspond to the non-blank symbols of the speech recognition result 120. Therefore, in this article, the non-blank symbol sequence 120 and the speech recognition result 120 can be used interchangeably. In some examples, N is equal to two. In other examples, N is equal to five, however, the present disclosure is non-restrictive, and N can be equal to any integer. In some implementations, the prediction network 300 includes a multi-head attention mechanism 302, which shares a shared embedding matrix 304 between each head 302A to 302H of the multi-head attention mechanism. In one example, the multi-head attention mechanism 302 includes four heads. However, any number of heads can be adopted by the multi-head attention mechanism 302. It is noteworthy that the multi-head attention mechanism significantly improves performance with minimal increase in model size. As described in more detail below, each head 302A to 302H includes its own position vector row 308, and rather than causing an increase in model size by concatenating the outputs 318A to 318H from all heads, the outputs 318A to 318H are averaged by the head average module 322.
[0060] Referring to the first head 302A of the multi-head attention mechanism 302, the head 302A uses the shared embedding matrix 304 to generate corresponding embeddings 306, 306a to 306n (e.g., ). It is worth noting that since the shared embedding matrix 304 is shared among all heads of the multi-head attention mechanism 302, the other heads 302B to 302H all generate the same corresponding embedding 306 for each non-blank symbol. Head 302A also generates the corresponding position vector PV Aa–An 308, 308Aa to 308An (e.g., is assigned to each corresponding non-blank symbol in the non-blank symbol sequence 120. The corresponding position vector PV 308 assigned to each non-blank symbol indicates the position in the history of the non-blank symbol sequence (e.g., the N previous non-blank symbols output by the final Softmax layer of the joint network 250). For example, the first position vector PV Aa is assigned to the most recent position in history, and the last position vector PV An Assigned to the last position in the history of the N previous non-whitespace symbols output by the final Softmax layer of the joint network 250. Notably, each of the embeddings 306 may include the same number of dimensions (ie, dimensionality size) as each of the position vectors PV 308.
[0061] Although the corresponding embedding generated by the shared embedding matrix 304 for each of the non-blank symbols 120 in the non-blank symbol sequence 120 is the same at all heads 302A to 302H of the multi-head attention mechanism 302, each head 302A to 302H defines a different set / row of position vectors 308. For example, the first head 302A defines the position vector PV Aa–An In lines 308Aa to 308An, the second head 302B defines different position vectors PV Ba–Bn 308 Ba–Bn row, and the Hth header 302H defines another different position vector PV Ha–Hn 308 Ha–Hn OK.
[0062] For each non-blank symbol in the received non-blank symbol sequence 120, the first head 302A also weights the corresponding embedding 306 via the weight layer 310, the weight being proportional to the similarity between the corresponding embedding and the corresponding position vector PV 308 assigned to the embedding. In some examples, the similarity may include a cosine similarity (e.g., a cosine distance). In the illustrated example, the weight layer 310 outputs a weighted embedding sequence 312, 312Aa to 312An, each associated with a corresponding embedding 306, the corresponding embedding being weighted proportionally to the corresponding position vector PV 308 assigned to the embedding. In other words, the weighted embedding 312 output by the weight layer 310 for each embedding 306 may correspond to the dot product between the embedding 306 and the corresponding position vector PV 308. The weighted embedding 312 may be interpreted as paying attention to the embedding in proportion to the degree of similarity between the embedding and the position associated with their corresponding position vector PV 308. To increase computational speed, the prediction network 300 includes a non-recurrent layer, and therefore, the weighted embedding sequences 312Aa to 312An are not concatenated, but averaged by the weighted average module 316 to generate a weighted average 318A of the weighted embeddings 312Aa to 312An as output from the first head 302A, represented as:
[0063]
[0064] In Equation 8, h represents the index of head 302, n represents the position in the context, and e represents the embedding dimension. Additionally, in Equation 8, H and N include the sizes of the corresponding dimensions. Position vector PV 308 does not have to be trainable and may include random values. It is worth noting that even if weighted embedding 312 is averaged, position vector PV 308 can potentially save position history information, thereby alleviating the need to provide recurrent connections at each layer of prediction network 300.
[0065] The operations described above with respect to the first head 302A are performed by each other head 302B to 302H of the multi-head attention mechanism 302. Due to the different position vector sets PV 308 defined by each head 302, the weight layer 310 outputs a weighted embedding sequence 312Ba to 312Bn, 312Ha to 312Hn different from the weighted embedding sequence 312Aa to 312Aa at the first head 302A at each other head 302B to 302H. Thereafter, the weighted average module 316 generates respective weighted averages 318B to 318H of the corresponding weighted embeddings 312 of the non-blank symbol sequence as outputs from each other corresponding head 302B to 302H.
[0066] In the example shown, the prediction network 300 includes a head average module 322 that averages the weighted averages 318A to 318H output from the corresponding heads 302A to 302H. A projection layer 326 with SWISH can receive an output 324 corresponding to the average of the weighted averages 318A to 318H from the head average module 322 as input and generate a projection output 328 as output. The final layer normalization 330 can normalize the projection output 328 to provide a single embedding vector 350 at a corresponding time step from a plurality of time steps. The prediction network 300 generates only a single embedding vector 350 at each of the plurality of time steps after the initial time step.
[0067] In some configurations, the prediction network 300 does not implement the multi-head attention mechanism 302, and only performs the operations described above with respect to the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa to 312An is simply passed through the projection layer 326 and the layer normalization 330 to provide a single embedding vector 350.
[0068] Figure 4 4 is a flow chart of an example arrangement of operations of a method 400 for streaming end-to-end multilingual speech recognition using joint language identification. The method 400 may use memory stored in the memory hardware 620 ( Figure 6 ) on the data processing hardware 610 ( Figure 6 ) is executed on the . Data processing hardware 610 and memory hardware 620 may reside on Figure 1 on the user device 10 and / or the remote computing device 60.
[0069] At operation 402, the method 400 includes receiving an acoustic frame sequence 110 as an input to the ASR model 200. At operation 404, the method 400 includes generating, by the first encoder 210 of the ASR model 200 and at each of a plurality of output steps, a first high-order feature representation 212 for a corresponding acoustic frame 110 in the acoustic frame sequence 110. At operation 406, the method 400 includes generating, by the second encoder 220 of the ASR model 200 and at each of a plurality of output steps, a second high-order feature representation 222 for a corresponding first high-order feature representation 212. At operation 408, the method 400 includes generating, by the language ID predictor 230 of the ASR model 200 and at each of a plurality of output steps, a language prediction representation 232 based on a concatenation 231 of the first high-order feature representation 212 generated by the first encoder 210 at each of the plurality of output steps and the second high-order feature representation 222 generated by the second encoder 220 at each of the plurality of output steps. At operation 410, the method 400 includes generating, by the first decoder 240, 240a of the ASR model 200 and at each of the plurality of output steps, a first probability distribution 120a over possible speech recognition prediction text based on a concatenation of a second high-order feature representation 222 generated by the second encoder 220 at each of the plurality of output steps and a language prediction representation 232 generated by the language ID predictor 230 at each of the plurality of output steps.
[0070] Figure 5 6 is a flow chart of an example arrangement of operations of a method 500 for streaming end-to-end multilingual speech recognition using joint language identification. The method 500 may use memory stored in memory hardware 620 ( Figure 6 ) on the data processing hardware 610 ( Figure 6 ) is executed on the . Data processing hardware 610 and memory hardware 620 may reside on Figure 1 on the user device 10 and / or the remote computing device 60.
[0071] At operation 502, the method 500 includes receiving an acoustic frame sequence 110 as an input to the ASR model 200. At operation 504, the method 500 includes generating, by the first encoder 210 of the ASR model 200 and at each of a plurality of output steps, a first high-order feature representation 212 for a corresponding acoustic frame 110 in the acoustic frame sequence 110. At operation 506, the method 500 includes generating, by the language ID predictor 230 of the ASR model 200 and at each of a plurality of output steps, a language prediction representation 232 for the corresponding first high-order feature representation 212. At operation 508, the method 500 includes generating, by the second encoder 220 of the ASR model 200 and at each of a plurality of output steps, a second high-order feature representation 222 based on a concatenation 221 of the first high-order feature representation 212 generated by the first encoder 210 at each of the plurality of output steps and a language prediction representation 232 generated by the language ID predictor 230 at each of the plurality of output steps. At operation 510 , the method 500 includes generating, by the first decoder 240 a of the ASR model 200 and at each of a plurality of output steps, a first probability distribution 120 a (eg, a final speech recognition result 120 a ) over possible speech recognition prediction text for the corresponding second high-order feature representation 222 .
[0072] Figure 6 600 that can be used to implement the systems and methods described in this document. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only, and are not intended to limit implementations of the inventions described and / or claimed in this document.
[0073] The computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 connected to the memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 connected to a low-speed bus 670 and the storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 610 can process instructions for execution within the computing device 600, including instructions stored in the memory 620 or on the storage device 630, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 680 coupled to the high-speed interface 640). In other implementations, multiple processors and / or multiple buses can be used as appropriate along with multiple memories and multiple types of memories. In addition, multiple computing devices 600 can be connected, each of which provides a portion of the necessary operations (for example, as a server group, a blade server group, or a multi-processor system).
[0074] The memory 620 stores information non-temporarily within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-temporary memory 620 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.
[0075] The storage device 630 is capable of providing mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various implementations, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.
[0076] The high-speed controller 640 manages bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 640 is coupled to a memory 620, a display 680 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to a storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, through a network adapter.
[0077] The computing device 600 can be implemented in a variety of different forms, as shown. For example, the computing device can be implemented as a standard server 600a or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.
[0078] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuitry, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor that may be special-purpose or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device.
[0079] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0080] The processes and logic flows described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any type of digital computer. Generally, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.
[0081] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.
[0082] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the appended claims.
Claims
1. A multilingual automated speech recognition (ASR) model (200), characterized in that: include: A first encoder (210), wherein the first encoder is configured to: Receiving as input a sequence of acoustic frames (110); as well as generating, at each of a plurality of output steps, a first high-order feature representation (212) for a corresponding acoustic frame (110) in the sequence of acoustic frames (110); A second encoder (220), the second encoder being configured to: receiving as input the first high-order feature representation (212) generated by the first encoder (210) at each of the plurality of output steps; as well as generating, at each of the plurality of output steps, a second high-order feature representation (222) for the corresponding first high-order feature representation (212); A language identification (ID) predictor (230), the language identification predictor being configured to: receiving as input a concatenation (231) of the first high-order feature representation (212) generated by the first encoder (210) at each of the plurality of output steps and the second high-order feature representation (222) generated by the second encoder (220) at each of the plurality of output steps; as well as generating a language prediction representation at each of the plurality of output steps (232); and A first decoder (240a), the first decoder being configured to: receiving as input a concatenation of the second high-order feature representation (222) generated by the second encoder (220) at each of the plurality of output steps and the language prediction representation (232) generated by the language ID predictor (230) at each of the plurality of output steps; and A first probability distribution over possible speech recognition prediction text is generated at each of the plurality of output steps (120a).
2. The ASR model (200) according to claim 1, characterized in that Also includes: A second decoder (240b), the second decoder being configured to: receiving as input the first high-order feature representation (212) generated by the first encoder (210) at each of the plurality of output steps; as well as A second probability distribution over possible speech recognition prediction text is generated at each of the plurality of output steps (120b).
3. The ASR model (200) of claim 1 or 2, wherein the second decoder (240b) is further configured to generate a partial speech recognition result based on the second probability distribution (120b) over possible speech recognition prediction texts.
4. The ASR model (200) of claim 2 or 3, wherein: The first decoder (240a) and the second decoder (240b) each include a corresponding prediction network (300) followed by a corresponding joint network (250); The corresponding prediction networks (300) of the first decoder (240a) and the second decoder (240b) have the same structure including one of the following: Long short-term memory (LSTM) based prediction network; or V2 embeds a lookup table; and The corresponding joint networks (250) of the first decoder (240a) and the second decoder (240b) include the same structure.
5. The ASR model (200) of any one of claims 1 to 4, wherein the second encoder (220) generates the second high-order feature representation (222) without receiving any of the acoustic frames (110) as input.
6. The ASR model (200) of any one of claims 1 to 5, wherein the first encoder (210) comprises a causal encoder, the causal encoder comprising one of: Multiple unidirectional long short-term memory (LSTM) layers; Multiple conformer layers; or Multiple transformer layers.
7. The ASR model (200) of any one of claims 1 to 6, wherein the second encoder (220) comprises a non-causal encoder, the non-causal encoder comprising one of: One or more bidirectional long short-term memory (LSTM) layers; Multiple conformer layers; or Multiple transformer layers.
8. The ASR model (200) of any one of claims 1 to 7, wherein the first encoder (210), the second encoder (220), and the language ID predictor (230) are jointly trained on a multilingual training utterance set by: generating a first loss for the first encoder (210); generating a second loss for the second encoder (220); generating a third loss for the language ID predictor (230); as well as A weighted sum of the first loss, the second loss, and the third loss is minimized.
9. The ASR model (200) of claim 8, wherein a language ID target word-gram is added as the first word-gram of the corresponding ground-truth transcription of each multilingual training utterance in the multilingual training utterance set, the language ID target word-gram identifying the language of the corresponding multilingual training utterance.
10. The ASR model (200) of claim 8 or 9, wherein a language ID target word is added to each position where a code switch occurs in the corresponding ground-truth transcription of each multilingual training utterance in the multilingual training utterance set.
11. A computer-implemented method (400) which, when executed on data processing hardware (610), causes the data processing hardware (620) to perform operations, characterized in that The operations include: Receiving a sequence of acoustic frames (110) as input to an automatic speech recognition (ASR) model (200); Generating, by a first encoder (210) of the ASR model (200), a first high-order feature representation (212) for a corresponding acoustic frame (110) in the sequence of acoustic frames (110) at each of a plurality of output steps; Generating, by a second encoder (220) of the ASR model (200), a second high-order feature representation (222) for the corresponding first high-order feature representation (212) at each of the plurality of output steps; generating, by a language identification (ID) predictor (230) of the ASR model (200), a language prediction representation (232) at each of the plurality of output steps based on a concatenation (231) of the first high-order feature representation (212) generated by the first encoder (210) at each of the plurality of output steps and the second high-order feature representation (222) generated by the second encoder (220) at each of the plurality of output steps; and A first probability distribution (120a) over possible speech recognition prediction texts is generated by a first decoder (240a) of the ASR model (200) at each of the plurality of output steps based on a concatenation of the second high-order feature representation (222) generated by the second encoder (220) at each of the plurality of output steps and the language prediction representation (232) generated by the language ID predictor (230) at each of the plurality of output steps.
12. The computer-implemented method (400) of claim 11, wherein the operation further comprises generating, by a second decoder (240b) of the ASR model (200), at each of the plurality of output steps, a second probability distribution (120b) over possible speech recognition prediction text based on the first high-order feature representation (212) generated by the first encoder (210) at each of the plurality of output steps.
13. The computer-implemented method (400) of claim 12, wherein the operations further comprise generating, by the second decoder (240b), a partial speech recognition result based on the second probability distribution (120b) over possible speech recognition prediction text.
14. The computer-implemented method (400) of claim 12 or 13, wherein: The first decoder (240a) and the second decoder (240b) each include a corresponding prediction network (300) followed by a corresponding joint network (250); The corresponding prediction networks (300) of the first decoder (240a) and the second decoder (240b) have the same structure including one of the following: Long short-term memory (LSTM) based prediction network; or V2 embeds a lookup table; and The corresponding joint networks (250) of the first decoder (240a) and the second decoder (240b) include the same structure.
15. The computer-implemented method (400) of any one of claims 11 to 14, wherein the second encoder (220) generates the second high-order feature representation (220) without receiving any of the acoustic frames (110) as input.
16. The computer-implemented method (400) of any one of claims 11 to 15, wherein the first encoder (210) comprises a causal encoder, the causal encoder comprising one of: Multiple unidirectional long short-term memory (LSTM) layers; Multiple conformer layers; or Multiple transformer layers.
17. The computer-implemented method (400) of any one of claims 11 to 16, wherein the second encoder (220) comprises a non-causal encoder, the non-causal encoder comprising one of: One or more bidirectional long short-term memory (LSTM) layers; Multiple conformer layers; or Multiple transformer layers.
18. The computer-implemented method (400) of any one of claims 11 to 17, wherein the operations further comprise jointly training the first encoder (210), the second encoder (220), and the language ID predictor (230) on a multilingual training utterance set by: generating a first loss for the first encoder (210); generating a second loss for the second encoder (220); generating a third loss for the language ID predictor (230); as well as A weighted sum of the first loss, the second loss, and the third loss is minimized.
19. The computer-implemented method (400) of claim 18, wherein the operations further comprise adding a language ID target word-gram as a first word-gram of a corresponding ground-truth transcription of each training utterance in the set of multilingual training utterances, the language ID target word-gram identifying a language of the corresponding training utterance.
20. The computer-implemented method (400) of claim 18 or 19, wherein the operations further comprise adding a language ID target word-gram to each position where a code-switch occurs in a corresponding ground-truth transcription of each multilingual training utterance in the set of multilingual training utterances.
