Recognizing and mitigating mismatched language signals in multilingual automatic speech recognition

Abstract

A method (400) includes receiving a sequence of acoustic frames (110) and a language code (202), and generating, by a language verification model (200), a language verification result (290) for a corresponding acoustic frame in the sequence of acoustic frames. For each acoustic frame having a corresponding verification result indicating that a spoken language of the corresponding acoustic frame does match a language specified by the input language code, the method includes adding a learnable embedding (203) that maps to the input language code to acoustic features (111) derived from the corresponding acoustic frame, and generating, by an audio encoder (210) that receives the learnable embedding added to the acoustic features as input, a corresponding higher-order feature representation (212). The method further includes generating, by a decoder (220, 230), a probability distribution over possible speech recognition results (120).

Description

Technical Field

[0001] This disclosure relates to identifying and mitigating mismatched language signals in multilingual automatic speech recognition. Background Technology

[0002] Automatic speech recognition (ASR) – the process of acquiring audio input and transcribing it into text – has become a significant technology used extensively in mobile and other devices. Generally, ASR attempts to provide an accurate transcription of what people are saying by acquiring audio input (e.g., spoken words) and transcribing it into text. Modern ASR models, based on the continuous development of deep neural networks, are constantly improving in both accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay between the client speaking and the transcription). Although a large number of people are bilingual, most ASR models are only compatible with one language. Therefore, for a large number of bilingual speakers, an ASR model that is compatible with several different languages ​​while still maintaining the accuracy and latency performance metrics of modern ASR models would be desirable. Summary of the Invention

[0003] One aspect of this disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the hardware to perform operations including: receiving an acoustic frame sequence representing a speech and an input language code specifying the language of the speech as input to a multilingual automatic speech recognition (ASR) model configured to recognize speech in multiple different supported languages. The operation further includes generating a language verification result for a corresponding acoustic frame in the acoustic frame sequence by a language verification model at each of a plurality of output steps. The language verification result indicates whether the spoken language of the corresponding acoustic frame matches or does not match the language specified by the input language code. For each acoustic frame in the acoustic frame sequence having a corresponding verification result indicating that the spoken language of the corresponding acoustic frame does indeed match the language specified by the input language code, the operation further includes: adding a learnable embedding mapped to the input language code to acoustic features derived from the corresponding acoustic frame; and generating a corresponding higher-order feature representation for the corresponding acoustic frame by an audio encoder of the multilingual ASR model that receives the learnable embedding added to the acoustic features derived from the corresponding acoustic frame as input. For each acoustic frame in the acoustic frame sequence that has a corresponding language verification result indicating a spoken language mismatch of the corresponding acoustic frame as specified by the input language code, the operation further includes: generating a corresponding higher-order feature representation for the corresponding acoustic frame by an audio encoder that receives acoustic features derived from the corresponding acoustic frame as input without adding a learnable embedding mapped to the input language code. The operation also includes generating a probability distribution of possible speech recognition results at each output step of the decoder of the multilingual ASR model, the probability distribution of possible speech recognition results being based on the corresponding higher-order feature representation generated by the audio encoder.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the operation further includes: for each acoustic frame in the acoustic frame sequence having a corresponding language verification result indicating a spoken language mismatch between the corresponding acoustic frame and the language specified by an input language code: generating a language prediction representation for the corresponding acoustic frame by a language recognition (LID) predictor model; and adding the language prediction representation to acoustic features derived from the corresponding acoustic frame, wherein generating a corresponding higher-order feature representation includes generating a corresponding higher-order feature representation for the corresponding acoustic frame by an audio encoder that receives the language prediction representation added to the acoustic features derived from the corresponding acoustic frame as input. The LID predictor model may include: a convolutional layer configured to downsample each corresponding acoustic frame to generate a corresponding strided convolution as output; a stack of multi-head self-attention layers, each having a multi-head attention mechanism, wherein the first self-attention layer in the stack is configured to receive a corresponding strided convolution generated as the output from the convolutional layer, and the last self-attention layer in the stack is configured to generate a corresponding multi-head attention output for each corresponding acoustic frame; a temporal pooling layer configured to temporally pool the corresponding multi-head attention output; and a softmax layer configured to derive a language prediction representation for the corresponding acoustic frame from the corresponding temporally pooled multi-head attention output. The stack of multi-head self-attention layers may include a stack of conformer layers or a stack of transformer layers. Additionally or alternatively, the LID predictor model may be initially trained using a recurrent neural network-transducer (RNN-T) loss until convergence, and fine-tuned using a cross-entropy loss to classify across multiple different languages. Multilingual ASR models can be trained jointly with LID predictor models or separately from LID predictor models.

[0005] In some examples, when generating a corresponding higher-order feature representation for each corresponding acoustic frame in the acoustic frame sequence that has a corresponding language verification result indicating a mismatch between the spoken language of the corresponding acoustic frame and the language specified by the input language code, the audio encoder is configured to receive only the acoustic frame derived from the corresponding acoustic frame as input and generate the corresponding higher-order feature representation without using the input language code or any language prediction representation for the corresponding acoustic frame. In some additional implementations, the acoustic frame sequence received as input at the multilingual ASR model represents utterance spoken in at least one of several different languages ​​supported by the multilingual ASR model. In these examples, the utterance may include code-mixed utterance, which includes one or more words spoken in a first language and one or more other words spoken in a second language.

[0006] In some implementations, the decoder includes a prediction network and a joint network. The prediction network is configured to: receive, at each of the multiple output steps, a sequence of non-whitespace symbols output by the final softmax layer as input; and generate a dense representation at each of the multiple output steps. The joint network is configured to: receive, at each of the multiple output steps, the dense representation generated by the prediction network and the corresponding higher-order feature representation generated by the audio encoder at each of the multiple output steps as input; and generate a probability distribution for possible speech recognition results at each of the multiple output steps.

[0007] In some examples, the language verification model is configured to generate language verification results as binary code, which includes: a first value when the language verification model determines that the spoken language of the corresponding acoustic frame matches the language specified by the input language code; or a second value when the language verification model determines that the spoken language of the corresponding acoustic frame does not match the language specified by the input language code.

[0008] The language validation model can be trained on: multiple positive training utterances, each including audio data representing utterance spoken in the corresponding language paired with a positive language code of the same corresponding language as the specified spoken utterance; and multiple negative training utterances, each including audio data representing utterance spoken in the corresponding language paired with a negative language code of a specified language different from the corresponding language of the spoken utterance. The multilingual ASR model can be trained jointly with the language validation model or separately from it.

[0009] In some implementations, the input language code is derived from one of the following: the language settings of a computing device that captures utterances in streaming audio, spoken by a user of the computing device; the country / region where the computing device is located; or the language settings uploaded by the user when an acoustic frame sequence representing the utterance is uploaded from an external source. In some additional implementations, the operation further includes: generating a transcription of the utterance as output from a multilingual ASR model based on a probability distribution of possible speech recognition results generated by the decoder at each of multiple output steps; and providing the transcription for output to downstream applications, wherein the transcription comprises text words in at least one of a variety of supported languages.

[0010] Another aspect of this disclosure provides a system including data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including: receiving an acoustic frame sequence representing a speech and an input language code specifying the language of the speech as input to a multilingual automatic speech recognition (ASR) model configured to recognize speech in multiple different supported languages. The operation further includes generating a language verification result for a corresponding acoustic frame in the acoustic frame sequence by a language verification model at each of a plurality of output steps. The language verification result indicates whether the spoken language of the corresponding acoustic frame matches or does not match the language specified by the input language code. For each acoustic frame in the acoustic frame sequence having a corresponding verification result indicating that the spoken language of the corresponding acoustic frame does indeed match the language specified by the input language code, the operation further includes: adding a learnable embedding mapped to the input language code to acoustic features derived from the corresponding acoustic frame; and generating a corresponding higher-order feature representation for the corresponding acoustic frame by an audio encoder of the multilingual ASR model that receives the learnable embedding added to the acoustic features derived from the corresponding acoustic frame as input. For each acoustic frame in the acoustic frame sequence that has a corresponding language verification result indicating a spoken language mismatch of the corresponding acoustic frame as specified by the input language code, the operation further includes: generating a corresponding higher-order feature representation for the corresponding acoustic frame by an audio encoder that receives acoustic features derived from the corresponding acoustic frame as input without adding a learnable embedding mapped to the input language code. The operation also includes generating a probability distribution of possible speech recognition results at each output step of the decoder of the multilingual ASR model, the probability distribution of possible speech recognition results being based on the corresponding higher-order feature representation generated by the audio encoder.

[0011] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the operation further includes: for each acoustic frame in the acoustic frame sequence having a corresponding language verification result indicating a spoken language mismatch between the corresponding acoustic frame and the language specified by an input language code: generating a language prediction representation for the corresponding acoustic frame by a language recognition (LID) predictor model; and adding the language prediction representation to acoustic features derived from the corresponding acoustic frame, wherein generating a corresponding higher-order feature representation includes generating a corresponding higher-order feature representation for the corresponding acoustic frame by an audio encoder that receives the language prediction representation added to the acoustic features derived from the corresponding acoustic frame as input. The LID predictor model may include: a convolutional layer configured to downsample each corresponding acoustic frame to generate a corresponding strided convolution as output; a stack of multi-head self-attention layers, each having a multi-head attention mechanism, wherein the first self-attention layer in the stack is configured to receive a corresponding strided convolution generated as the output from the convolutional layer, and the last self-attention layer in the stack is configured to generate a corresponding multi-head attention output for each corresponding acoustic frame; a temporal pooling layer configured to temporally pool the corresponding multi-head attention output; and a softmax layer configured to derive a language prediction representation for the corresponding acoustic frame from the corresponding temporally pooled multi-head attention output. The stack of multi-head self-attention layers may include a stack of conformer layers or a stack of transformer layers. Additionally or alternatively, the LID predictor model may be initially trained using a recurrent neural network-transducer (RNN-T) loss until convergence, and fine-tuned using a cross-entropy loss to classify across multiple different languages. Multilingual ASR models can be trained jointly with LID predictor models or separately from LID predictor models.

[0012] In some examples, when generating a corresponding higher-order feature representation for each corresponding acoustic frame in the acoustic frame sequence that has a corresponding language verification result indicating a mismatch between the spoken language of the corresponding acoustic frame and the language specified by the input language code, the audio encoder is configured to receive only the acoustic frame derived from the corresponding acoustic frame as input and generate the corresponding higher-order feature representation without using the input language code or any language prediction representation for the corresponding acoustic frame. In some additional implementations, the acoustic frame sequence received as input at the multilingual ASR model represents utterance spoken in at least one of several different languages ​​supported by the multilingual ASR model. In these examples, the utterance may include code-mixed utterance, which includes one or more words spoken in a first language and one or more other words spoken in a second language.

[0013] In some implementations, the decoder includes a prediction network and a joint network. The prediction network is configured to: receive, at each of the multiple output steps, a sequence of non-whitespace symbols output by the final softmax layer as input; and generate a dense representation at each of the multiple output steps. The joint network is configured to: receive, at each of the multiple output steps, the dense representation generated by the prediction network and the corresponding higher-order feature representation generated by the audio encoder at each of the multiple output steps as input; and generate a probability distribution for possible speech recognition results at each of the multiple output steps.

[0014] In some examples, the language verification model is configured to generate language verification results as binary code, which includes: a first value when the language verification model determines that the spoken language of the corresponding acoustic frame matches the language specified by the input language code; or a second value when the language verification model determines that the spoken language of the corresponding acoustic frame does not match the language specified by the input language code.

[0015] The language validation model can be trained on: multiple positive training utterances, each including audio data representing utterance spoken in the corresponding language paired with a positive language code of the same corresponding language as the specified spoken utterance; and multiple negative training utterances, each including audio data representing utterance spoken in the corresponding language paired with a negative language code of a specified language different from the corresponding language of the spoken utterance. The multilingual ASR model can be trained jointly with the language validation model or separately from it.

[0016] In some implementations, the input language code is derived from one of the following: the language settings of a computing device that captures utterances in streaming audio, spoken by a user of the computing device; the country / region where the computing device is located; or the language settings uploaded by the user when an acoustic frame sequence representing the utterance is uploaded from an external source. In some additional implementations, the operation further includes: generating a transcription of the utterance as output from a multilingual ASR model based on a probability distribution of possible speech recognition results generated by the decoder at each of multiple output steps; and providing the transcription for output to downstream applications, wherein the transcription comprises text words in at least one of a variety of supported languages.

[0017] Details of one or more implementations of this disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the specification, drawings, and claims. Attached Figure Description

[0018] Figure 1This is a schematic diagram of an example speech recognition system that implements a multilingual speech recognition model, a language identifier (ID) verification model, and a language ID predictor model.

[0019] Figure 2 This is a schematic diagram of a multilingual speech recognition model and a language ID verification model.

[0020] Figure 3 This is a schematic diagram of a multilingual speech recognition model and a language ID predictor model.

[0021] Figure 4 This is a flowchart of an example operational setup for a method to identify and mitigate mismatched language signals in multilingual automatic speech recognition.

[0022] Figure 5 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0023] In the various figures, the same reference numerals indicate the same elements. Detailed Implementation

[0024] End-to-end (E2E) automatic speech recognition (ASR) models are traditionally structured to operate in either streaming or non-streaming modes. Typically, an E2E ASR model includes an encoder and a decoder as its main components. Applications involving end-user interaction, such as voice search or on-device dictation, may require the model to perform recognition in a streaming manner. Here, performing recognition in a streaming manner means that the ASR model outputs each word in the utterance with as little latency as possible as the utterance is spoken. Other applications, such as offline video captioning, do not require the model to be streaming and can leverage future context to improve performance.

[0025] In some implementations, E2E ASR models are configured to recognize speech from multiple languages ​​(e.g., E2E multilingual ASR models). Typically, multilingual ASR models use input language codes added to the input audio data, which modulate the multilingual ASR model to transcribe the input audio data into the target language indicated by the input language code. However, the spoken language of the input audio data may not always match the language specified by the input language code. This mismatch frequently occurs in code-switching languages ​​prevalent in multilingual societies. Unfortunately, language mismatch can lead to a significant degradation in the accuracy / quality of the resulting transcription output by the multilingual ASR model. The input language code can be derived from a user-defined language signal. For example, the language setting of the computing device that captured the input speech or the country / region where the computing device is located can indicate the target language indicated by the input language code. Similarly, the input language code can be derived from the user-uploaded language settings of the computing device that captured the input speech.

[0026] The implementation of this paper involves identifying whether the spoken language of the input speech to be transcribed by a multilingual automatic speech recognition (ASR) model matches the input language code of a specified target language, and when the spoken language of the input speech matches the input language code, using the input language code to adjust the multilingual ASR model to transcribe the input speech in the target language. The implementation further involves using a language recognition (LID) predictor model to predict the language of the input speech, and when the spoken language of the input speech does not match the input language code, using the predicted language to adjust the multilingual ASR model to transcribe the input speech in the predicted language.

[0027] Specifically, the multilingual ASR model is configured to recognize speech in multiple different supported languages ​​and can receive a sequence of acoustic frames representing the utterance and an input language code specifying the language of the utterance. At each of the multiple output steps, the language verification model can predict a language verification result for a corresponding acoustic frame in the acoustic frame sequence, where the language verification result indicates whether the spoken language of the corresponding acoustic frame matches or does not match the language specified by the language input code. For each acoustic frame in the acoustic frame sequence with a corresponding verification result indicating that the spoken language of the corresponding acoustic frame does indeed match the language specified by the input language code, the implementation includes: adding a learnable embedding mapped to the input language code to the acoustic features derived from the corresponding acoustic frame; and generating a higher-order feature representation for the corresponding acoustic frame by an audio encoder of the multilingual ASR model that receives the learnable embedding added to the acoustic features derived from the corresponding acoustic frame as input. On the other hand, for each acoustic frame in the acoustic frame sequence with a corresponding language verification result indicating a mismatch between the spoken language of the corresponding acoustic frame and the language specified by the input language code, the implementation alternatively includes: an audio encoder that receives acoustic features derived from the corresponding acoustic frame as input without adding a learnable embedding mapped to the input language code to generate a corresponding higher-order feature representation for the corresponding acoustic frame. Finally, the decoder of the multilingual ASR model is configured to generate a probability distribution regarding possible speech recognition results at each of multiple output steps, based on the corresponding higher-order feature representation generated by the audio encoder. In some examples, for each acoustic frame in the acoustic frame sequence with a corresponding language verification result indicating a mismatch between the spoken language of the corresponding acoustic frame and the language specified by the input language code, the LID predictor model generates a language prediction representation for the corresponding acoustic frame and adds the language prediction representation to the acoustic features derived from the corresponding acoustic frame. In these examples, the audio encoder receives the language prediction representation added to the acoustic features derived from the corresponding acoustic frame as input and generates a corresponding higher-order feature representation for the corresponding acoustic frame.

[0028] Figure 1Example system 100 in a voice environment, where user 104 interacts with a computing device such as user device 10 via voice input. User device 10 (also commonly referred to as device 10) is configured to capture sound (e.g., streaming audio data) from one or more users 104 within the voice environment 100. Streaming audio data may refer to spoken utterances 106 of user 104, which act as an audible query, a command to user device 10, or audible communication captured by device 10. The voice-enabled system of user device 10 can process queries or commands by answering queries and / or causing commands to be executed / fulfilled by one or more downstream applications.

[0029] User device 10 may correspond to any computing device associated with user 104 and capable of receiving audio data. Some examples of user device 10 include, but are not limited to: mobile devices (e.g., smartphones and tablets), wearable devices (e.g., smartwatches and head-mounted devices), smart home appliances, Internet of Things (IoT) devices, in-vehicle infotainment systems, smart displays, smart speakers, etc. User device 10 includes data processing hardware 12 and memory hardware 14 that communicates with and stores instructions to the data processing hardware 12, which, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. User device 10 further includes an audio system 16 having: audio capture devices (e.g., microphones) 16, 16a for capturing spoken speech 106 within a speech environment 100 and converting the spoken speech into electrical signals; and voice output devices (e.g., speakers) 16, 16b for transmitting audible audio signals (e.g., as output data from user device 10). When the user device 10 can implement an array of audio capture devices 16a without departing from the scope of this disclosure, one or more capture devices 16a in the array may not be physically residing on the user device 10, but may communicate with the audio system 16.

[0030] System 100 includes a multilingual automatic speech recognition (ASR) model 201, which integrates a language verification model 200 for verifying whether the spoken language of input speech 110 representing utterance 106 matches an input language code 202 specifying the language of input speech 110. Optionally, the multilingual ASR model 201 may integrate a language recognition (LID) predictor model 300 for generating a language prediction representation 302 for the spoken language of input speech 110. As will become apparent, when the language verification model 200 verifies that the spoken language of input speech 110 matches the input language code 202, the ASR model 201 can use the input language code 202 to transcribe the input speech 110 into a script in the language specified by the input language code 202. Otherwise, when the language verification model 200 verifies that the spoken language of the input speech 110 does not match the input language code 202, the ASR model 201 may not use the input language code 202 and may optionally use the language prediction representation 302 generated by the LID predictor model 300 to transcribe the input speech into text of the language specified by the language prediction representation 302.

[0031] The multilingual ASR model 201, language verification model 200, and LID predictor model 300 may reside on the user device 10 of user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system executing in a cloud computing environment) that communicates with user device 10 via network 40. In some examples, ASR model 201 includes a recurrent neural network-transducer (RNN-T) model. User device 10 and / or remote computing device 60 also include an audio subsystem 108 configured to receive utterance 106 spoken by user 104 and captured by audio capture device 16a, and to convert utterance 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by ASR model 201. In the example shown, the user speaks the corresponding utterance 106, and audio subsystem 108 converts utterance 106 into corresponding audio data (e.g., an acoustic frame sequence) 110 for input to ASR model 201, language verification model 200, and LID predictor model 300. Language verification model 200 generates a language verification result 290 for the corresponding acoustic frame 110 in the acoustic frame sequence 110 at each output step. Here, the language verification result 290 indicates whether the spoken language of the corresponding acoustic frame 110 matches or does not match the language specified by an input language code, which specifies the language of the utterance 106 spoken by user 104. In some examples, the language verification result 290 generated at each output step includes a binary code that includes a first value (e.g., "1") when language verification model 200 determines that the spoken language of the corresponding acoustic frame 110 matches the language specified by input language code 202, or a second value (e.g., "0") when language verification model 200 determines that the spoken language of the corresponding acoustic frame 110 does not match the language specified by input language code 202. See below for reference. Figure 2 In more detail, for each acoustic frame 110 in the acoustic frame sequence 110 that has a corresponding verification result 290 indicating that the spoken language of the corresponding acoustic frame 110 does indeed match the language specified by the input language code 202, the ASR model 201 may receive acoustic features 111 added to the corresponding acoustic frame 110. Figure 2The learnable embedding 203 is used as input. Otherwise, for those acoustic frames with corresponding verification results 290 indicating a mismatch between spoken language and the language specified by the input language code 202, the ASR model 201 can receive the acoustic features 111 derived from those acoustic frames as input without adding the learnable embedding 203 mapped to the input language code. Here, for each acoustic frame 110 with corresponding language verification results 290 indicating a mismatch between spoken language and the language specified by the input language code 202, the LID predictor model 300 can generate a language prediction representation 302 for that corresponding acoustic frame, and the ASR model 201 can receive the language prediction representation 402 added to the acoustic features 111 derived from the corresponding acoustic frame 110 as input.

[0032] In the example shown, ASR model 201 can perform streaming speech recognition to produce initial speech recognition results 120, 120b, and generate final speech recognition results 120, 120a by improving the initial speech recognition result 120b. Speech recognition result 120 can correspond to a partial speech recognition result or the entire speech recognition result. In other words, speech recognition result 120 can correspond to a part or the entire utterance 106. For example, a partial speech recognition result can correspond to a part of spoken utterance or even a part of spoken words. However, as will become apparent, ASR model 201 performs additional processing on the final speech recognition result 120a, thereby potentially delaying the final speech recognition result 120a relative to the initial speech recognition result 120b.

[0033] User device 10 and / or remote computing device 60 also execute user interface generator 107, which is configured to present a representation of the transcription 120 of utterance 106 to user 104 of user device 10. User interface generator 107 may stream an initial speech recognition result 120b during time 1 and subsequently stream a final speech recognition result 120a during time 2. Notably, ASR model 200 outputs the final speech recognition result 120a in a streaming manner, even if the final speech recognition result 120a is an improvement on the initial speech recognition result 120b. In some configurations, the transcription 120 output from ASR model 201 is processed by a downstream application 109, for example, executed on user device 10 or remote computing device 60, to execute a user command / query specified by utterance 106. For example, application 109 may include a large language model (LLM) or natural language understanding (NLU) module. For example, utterance 106 may correspond to an input prompt directed to an LLM-based conversational assistant application 50, and transcription 120 may be used to prompt the LLM-based conversational assistant 50 to generate a corresponding response 19 to the prompt. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executed on any combination of user device 10 or telecomputing device 60) may convert transcription 120 into synthesized speech for audible output by user device 10 and / or another device.

[0034] In the example shown, user 104 interacts with a program or application 50 (e.g., digital assistant application 50) on user device 10 using multilingual ASR model 201. For example, Figure 1 The text depicts a user 104 communicating with a digital assistant application 50, and the digital assistant application 50 displaying a digital assistant interface 18 on the screen of the user device 10 to illustrate the dialogue between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, “What time is the concert tonight?” This question from the user 104 is spoken utterance 106, which is captured by an audio capture device 16a and processed by the audio system 16 of the user device 10. In this example, the audio system 16 receives the spoken utterance 106 and converts it into an acoustic frame sequence 110 for input to the ASR model 201, the language verification model 200, and the LID predictor model 300.

[0035] The language verification model 200 also receives an input language code 202 specifying the language of the utterance 106. The input language code 202 can be derived from the language settings of the user device 10 that captured the utterance 106 in the streaming audio. The input language code 202 can also be derived from the country / region where the user device 10 is located. Alternatively, the input language code 202 can be derived from the language settings uploaded by the user when the acoustic frame sequence 110 representing the utterance 106 is uploaded from an external source.

[0036] exist Figure 1 In the example shown, digital assistant application 50 can use natural language processing (NLP) to respond to a question / prompt from user 104. NLP typically refers to the process of interpreting written language (e.g., initial speech recognition result 120b and / or final speech recognition result 120a) and determining whether the written language prompts any action. In this example, digital assistant application 50 uses LLM or other NLP techniques to identify that the question from user 104 is about the user's schedule, and more specifically, about a concert on the user's schedule. By leveraging NLP to identify these details, the automated assistant returns a response 19 to the user's query, stating: "Venue doors open at 6:30 PM and concert starts at 8 PM." In some configurations, NLP occurs on a remote server 60 communicating with the data processing hardware 12 of user device 10.

[0037] refer to Figure 2 and Figure 3ASR model 201 may include a recurrent neural network-transducer (RNN-T) model architecture that follows latency constraints associated with interactive applications. The use of the RNN-T model architecture is exemplary, and ASR model 200 may include other architectures such as transformer-transducer and conformer-transducer model architectures. RNN-T models offer a small computational footprint and use less memory than conventional ASR architectures, making them suitable for performing speech recognition entirely on user device 10 (e.g., without needing to communicate with a remote server). The RNN-T model architecture of ASR model 201 includes an encoder network 210, a prediction network 220, and a joint network 230. The encoder network 210, broadly similar to the acoustic model (AM) in a conventional ASR system, comprises a recurrent network of stacked or stacked long short-term memory (LSTM) layers of self-attention layers (e.g., conformer or transformer layers). Acoustic frame 110 may include 128-dimensional log-mel features computed at a frame rate of 10 ms over 32 millisecond (ms) frames, and strided convolution 205 may project the acoustic frame onto 1024-dimensional audio features 111. Encoder 210 reads each from the acoustic frame sequence 110 (e.g., x = (x 1 , x 2 , · · · , x T The d-dimensional feature vector sequence 113 corresponding to one acoustic frame in the ) is as follows: x t ∈ Furthermore, a higher-order feature representation 212 is generated at each output step. This higher-order feature representation is represented as... .

[0038] Similarly, prediction network 220 is also an LSTM network, which, like a language model (LM), will output the sequence of non-whitespace symbols so far by the final Softmax layer 240. y 0 ... y ui-1 Processed as dense representation 222 Finally, utilizing the RNN-T model architecture, the representations generated by the encoder network 210 and the predictor / decoder network 220 are combined by a joint network 230. The predictor network 220 can be replaced by an embedding lookup table to improve latency by outputting the looked-up sparse embeddings instead of processing dense representations. The joint network then predicts... This is a distribution of the next output symbol. In other words, the joint network 230 generates a probability distribution of possible speech recognition hypotheses at each output step (e.g., time step). As may be used herein, a “possible speech recognition hypothesis” corresponds to a set of output labels, each representing a symbol / character in a specified natural language. For example, when the natural language is English, this set of output labels may include twenty-seven (27) symbols, such as one label for each of the 26 letters of the English alphabet, and one label for a space. Thus, the joint network 230 may output a set of values ​​indicating the probability of each output label in the predetermined set of output labels occurring. This set of values ​​may be a vector and may indicate a probability distribution of the output labels in that set. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation marks and other symbols), but the set of output labels is not limited to this. For example, in addition to graphemes or alternative graphemes, the set of output labels may also include word fragments, phonemes, and / or entire words. The output distribution of the joint network 230 may include posterior probability values ​​for each of the different output labels. Therefore, if there are 100 different output labels representing different characters or other symbols, then the output y of the joint network 230... i It can include 100 different probability values, one probability value for each output label. The probability distribution can then be used (e.g., by a Softmax layer 240) to select candidate orthographic elements (e.g., graphemes, word fragments, and / or words) during the beam search process and assign them scores for determining transcription 120.

[0039] The Softmax layer 240 can employ any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the ASR model 201 in the corresponding output step. In this way, the ASR model 201 does not make a conditional independence assumption; instead, the prediction of each symbol is conditional not only on acoustics but also on the sequence of labels output so far. The ASR model 201 does assume that the output symbol is independent of future acoustic frames 110, which allows the ASR model 201 to be used in a streaming, non-streaming, or some combination thereof.

[0040] In some examples, the encoder network (i.e., the audio encoder) 210 of the RNN-T model 201 includes a stack of self-attention layers / blocks, such as conformer blocks, each of which includes a multi-head self-attention mechanism. Here, each conformer block includes a series of multi-head self-attention layers, depthwise convolutional layers, and feedforward layers. In some examples, the audio encoder 210 includes 24 multi-head attention layers (e.g., conformer layers). The prediction network 220 may have a 768-dimensional LSTM layer, and the joint network 230 can project the higher-order feature representation 212 output by the audio encoder 210 and the dense representation 222 output by the prediction network 220 onto 640 hidden units to produce a posterior of more than 4096 word fragments. The softmax layer 240 may consist of a uniform set of word fragments or morphemes generated using all unique word fragments or morphemes from multiple training datasets.

[0041] Figure 2 The diagram illustrates how the language verification model 200 generates a corresponding language verification result 290 for the corresponding acoustic frame 110 at each of multiple output steps. When the verification result 290 indicates that the spoken language of the corresponding acoustic frame matches the language specified by the input language code 202 (e.g., the language verification result equals "1"), the ASR model 201 receives a learnable embedding 203 mapped to the input language code 202, such that the learnable embedding 203 is added to the acoustic features 111 derived from the corresponding acoustic frame, and the audio encoder 210 generates a corresponding higher-order feature representation 212 for the corresponding acoustic frame based on the learnable embedding 203 added to the acoustic features 211 derived from the corresponding acoustic frame. When verification result 290 indicates that the spoken language of the corresponding acoustic frame does not match the language specified by the input language code 202 (e.g., the language verification result equals "0"), ASR model 201 may only receive the acoustic features 111 derived from the corresponding acoustic frame without adding learnable embedding 203, such that encoder 210 generates a higher-order feature representation 212 for the corresponding acoustic frame 110 based solely on the acoustic features 111 derived from the corresponding acoustic frame. Optionally, the language prediction representation 302 generated by LID predictor model 300 for the corresponding acoustic frame 110 may be added to the acoustic features 111 derived from the corresponding acoustic frame, such that encoder 210 generates a higher-order feature representation based on the language prediction representation 302 added to the acoustic features 111.

[0042] Acoustic frame 110 may include 128-dimensional log-mel features computed at a frame rate of 10 ms over 32 millisecond (ms) frames, and strided convolution 207 may project the acoustic frame onto 1024-dimensional audio features to match the dimension of audio features 111 input to ASR model 201. Striked convolution 207 also projects learnable embeddings 203 mapped to input language code 202 onto the same dimension and adds learnable embeddings 203 thereto to provide corresponding audio features 211 with the same dimension as audio features 111 input to ASR model 201. Language validation model 200 may include 18 million parameters and includes a stack of multi-head attention blocks 250. Multi-head attention blocks 250 may include conformer blocks or transformer blocks. In some examples, the stack of multi-head attention blocks 250 includes a stack of 12 conformer blocks. The stacked processing of multi-head attention blocks 250 corresponds to audio features 211 to generate attention output 252 at each of the multiple output steps. Then, an average pooling layer 260 performs average pooling on the attention output 252 to provide the average pooled result 262 as input to a final softmax layer 280, which is configured to provide language verification results.

[0043] The language verification model 200 can be trained using labeled training utterances to generate language verification results 290 as binary outputs (e.g., "1" or "0"). Each labeled training utterance includes audio data representing the utterance and a corresponding language label used as the ground truth input language code for the utterance. The language verification model 200 can be trained on both positive and negative training utterances. Here, for a fixed percentage of labeled training utterances, negative training utterances can be generated by flipping the ground truth input language code to a randomly selected incorrect input language code. The language verification model 200 and the ASR model 201 can be trained jointly or separately. The language verification model 200 and the LID predictor model 300 can be trained jointly or separately.

[0044] Figure 3 This illustrates how the LID predictor model 300 generates a corresponding language prediction representation 302 for the corresponding acoustic frame 110 at each of multiple output steps in multiple output steps. When by... Figure 2When the language verification model 200 determines a language verification result 290 for the corresponding acoustic frame 110 indicating that the spoken language of the corresponding acoustic frame 110 does not match the language specified by the input language code 202 (e.g., the language verification result equals "0"), the LID predictor model 300 can activate a switching connection to provide the corresponding predicted language representation 302 as input to the ASR model 200 so that it can be added to the acoustic features 111 derived from the corresponding acoustic frame, thereby providing the corresponding d-dimensional features 113 as input to the encoder 210. Otherwise, the switching connection alternatively provides a learnable embedding 203 mapped to the input language code as input to the ASR model 200 so that it can be added to the acoustic features 111.

[0045] Acoustic frame 110 may include 128-dimensional log-mel features computed at a frame rate of 10 ms over 32 millisecond (ms) frames, and strided convolution 307 may project the acoustic frame onto 1024-dimensional audio features to match the dimension of audio features 311 input to ASR model 201. LID predictor model 300 may include 120 million parameters and includes a stack of multi-head attention blocks 350. Multi-head attention blocks 350 may include conformer blocks or transformer blocks. In some examples, the stack of multi-head attention blocks 350 includes a stack of 5 conformer blocks. The stack of multi-head attention blocks 350 processes the corresponding audio features 311 to generate attention output 352 at each of multiple output steps, followed by an average pooling layer 360 that temporally pools the attention output 352 to provide the pooled result 362 as input to a final softmax layer 380 configured to provide a language prediction representation 302. In some implementations, the language prediction representation 302 generated by the LID predictor model 300 is always used by the ASR model 201 to replace the input language code 202, or in scenarios where there is no input language code 202 available for the incoming utterance 106 to be transcribed.

[0046] It is worth noting that the LID predictor model 300 may be biased by non-linguistic factors such as channel characteristics, background noise, and speaker characteristics. In some examples, the LID predictor model 300 is jointly trained with the ASR model 201 to learn phonological representations, thereby encouraging the LID predictor model 300 to prioritize basic language structures when predicting language representations based on speech. The LID predictor model 300 can be initially trained using RNN_T loss until convergence, and then fine-tuned using cross-entropy loss to classify among a predetermined number of languages. In some examples, the predetermined number of languages ​​includes 10 different Indian languages. For example, the 10 different Indian languages ​​may include Hindi, Indian English, Gujarati, Tamil, Marathi, Telugu, Urdu, Kannada, Bengali, and Malayalam. Training utterances can be segmented into segments with a maximum length equal to 30 seconds. In other examples, the LID predictor model 300 and the ASR model 201 are trained separately.

[0047] Figure 4 This is a flowchart illustrating an example operational setup of a method 400 for identifying and mitigating mismatched language signals in multilingual automatic speech recognition. Method 400 may be based on memory hardware 520 that communicates with data processing hardware 510. Figure 5 Instructions on the data processing hardware 510 ( Figure 5 The data processing hardware 510 may include the data processing hardware 12 of the user device 10, or the data processing hardware 510 may reside on the remote system 60. The memory hardware 520 may include the memory hardware 14 of the user device 10, or the memory hardware 520 may reside on the remote system 60. At operation 402, method 400 includes receiving an acoustic frame sequence 110 representing utterance 106 and an input language code 202 specifying the language of utterance 106 as input to a multilingual automatic speech recognition (ASR) model 201, which is configured to recognize speech in multiple different supported languages. At operation 404, method 400 includes generating a language verification result 290 for a corresponding acoustic frame 110 in the acoustic frame sequence 110 by the language verification model 200 at each of a plurality of output steps. Here, the language verification result 290 indicates whether the spoken language of the corresponding acoustic frame 110 matches or does not match the language specified by the input language code 202.

[0048] For each acoustic frame 110 in the acoustic frame sequence 110 that has a corresponding verification result 290 indicating that the spoken language of the corresponding acoustic frame 110 does indeed match the language specified by the input language code 202, method 400 performs operations 406 and 408. At operation 406, method 400 includes adding a learnable embedding 203 mapped to the input language code 202 to the acoustic features 111 derived from the corresponding acoustic frame 110. At operation 408, method 400 includes generating a corresponding higher-order feature representation 212 for the corresponding acoustic frame 110 by an audio encoder 210 of a multilingual ASR model 201 that receives the learnable embedding 203 added to the acoustic features 111 derived from the corresponding acoustic frame 110 as input. In some examples, the audio encoder 210 includes a causal encoder followed by a non-causal encoder.

[0049] At operation 410, for each acoustic frame 110 in the acoustic frame sequence 110 having a corresponding language verification result 290 indicating a mismatch between the spoken language of the corresponding acoustic frame and the language specified by the input language code 202, method 400 includes generating a corresponding higher-order feature representation 210 for the corresponding acoustic frame 110 by an audio encoder 210 that receives acoustic features 111 derived from the corresponding acoustic frame 110 as input without adding a learnable embedding 203 mapped to the input language code 202. At operation 412, method 400 includes generating a probability distribution for possible speech recognition results by the decoders 220, 230 of the multilingual ASR model 201 at each of a plurality of output steps. Here, the probability distribution for possible speech recognition results is based on the corresponding higher-order feature representation 212 generated by the audio encoder 210 at the corresponding output step.

[0050] Figure 5 This is a schematic diagram of an example computing device 500 that can be used to implement the systems and methods described in this document. The computing device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementations of the invention described and / or claimed in this document.

[0051] Computing device 500 includes a processor 510, memory 520, storage device 530, a high-speed interface / controller 540 connected to memory 520 and high-speed expansion port 550, and a low-speed interface / controller 560 connected to low-speed bus 570 and storage device 530. Each of components 510, 520, 530, 540, 550, and 560 is interconnected using various buses and may be mounted on a common motherboard or otherwise. Processor 510 can process instructions for execution within computing device 500, including instructions stored in memory 520 or storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 580 coupled to high-speed interface 540. In other implementations, multiple processors and / or multiple buses, as well as multiple memories and various types of memory, may be used as appropriate. Moreover, multiple computing devices 500 may be connected, with each device providing a portion of the necessary operation (e.g., as a server group, blade server cluster, or multiprocessor system).

[0052] Memory 520 stores information non-temporarily within computing device 500. Memory 520 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-temporary memory 520 may be a physical means for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0053] Storage device 530 provides mass storage for computing device 500. In some implementations, storage device 530 is a computer-readable medium. In various implementations, storage device 530 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or device array (including devices arranged in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.

[0054] High-speed controller 540 manages bandwidth-intensive operations of computing device 500, while low-speed controller 560 manages lower bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some implementations, high-speed controller 540 is coupled to memory 520, display 580 (e.g., via a graphics processor or accelerator), and high-speed expansion port 550 which can accept various expansion cards (not shown). In some implementations, low-speed controller 560 is coupled to storage device 530 and low-speed expansion port 590. Low-speed expansion port 590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled, for example, via a network adapter to one or more input / output devices such as a keyboard, pointing device, scanner, or a networking device such as a switch or router.

[0055] As shown in the figure, the computing device 500 can be implemented in a variety of different forms. For example, the computing device can be implemented as a standard server 500a or multiple times in a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0056] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system, which includes at least one programmable processor, which may be dedicated or general-purpose, and is coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transfer data and instructions to the storage system, at least one input device, and at least one output device.

[0057] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0058] The processes and logic flows described in this specification can be executed by one or more programmable processors, also known as data processing hardware, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to said mass storage device or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit system.

[0059] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen) and possibly a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser in response to a request received from a web browser on the user's client device.

[0060] Various implementations have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (400) executed on data processing hardware (510), the computer-implemented method causing the data processing hardware (510) to perform operations, the operations including: The acoustic frame sequence (110) representing the utterance (106) and the input language code (202) specifying the language of the utterance (106) are received as input to a multilingual automatic speech recognition (ASR) model (201), which is configured to recognize speech in multiple different supported languages. The language verification model (200) generates a language verification result (290) for the corresponding acoustic frame (110) in the acoustic frame sequence (110) at each of the multiple output steps, the language verification result (290) indicating whether the spoken language of the corresponding acoustic frame (110) matches or does not match the language specified by the input language code (202); For each acoustic frame (110) in the acoustic frame sequence (110) that has a verification result indicating that the spoken language of the corresponding acoustic frame (110) does indeed match the language specified by the input language code (202): The learnable embedding (203) mapped to the input language code (202) is added to the acoustic features (111) derived from the corresponding acoustic frame (110); and The audio encoder (210) of the multilingual ASR model (201), which receives the learnable embedding (203) added to the acoustic features (111) derived from the corresponding acoustic frame (110) as input, generates a higher-order feature representation (212) for the corresponding acoustic frame (110). For each acoustic frame (110) in the acoustic frame sequence (110) having a corresponding language verification result (290) indicating that the spoken language of the corresponding acoustic frame (110) does not match the language specified by the input language code (202), the audio encoder (210) that receives the acoustic features (111) derived from the corresponding acoustic frame (110) as input without adding the learnable embedding (203) mapped to the input language code (202) generates a corresponding higher-order feature representation (212) for the corresponding acoustic frame (110); as well as The decoder (220, 230) of the multilingual ASR model (201) generates a probability distribution of possible speech recognition results (120) at each of the plurality of output steps, the probability distribution of possible speech recognition results (120) being based on the corresponding higher-order feature representation (212) generated by the audio encoder (210).

2. The method (400) as described in claim 1, wherein, The operation further includes, for each acoustic frame (110) in the acoustic frame sequence (110) having a corresponding language verification result (290) indicating that the spoken language of the corresponding acoustic frame (110) does not match the language specified by the input language code (202): The language recognition LID predictor model (300) generates a language prediction representation (302) for the corresponding acoustic frame (110); and The language prediction representation (302) is added to the acoustic features (111) derived from the corresponding acoustic frame (110). The generation of the corresponding higher-order feature representation (212) includes the audio encoder (210) receiving the language prediction representation (302) added to the acoustic features (111) derived from the corresponding acoustic frame (110) as input to generate the corresponding higher-order feature representation (212) for the corresponding acoustic frame (110).

3. The method (400) as described in claim 2, wherein, The LID prediction model (300) includes: A convolutional layer (307) is configured to downsample each corresponding acoustic frame (110) to generate a corresponding strided convolution (311) as output; A stack of multi-head self-attention layers (350), each having a multi-head attention mechanism, wherein a first self-attention layer in the stack of multi-head self-attention layers (350) is configured to receive the corresponding strided convolution generated as the output from the convolutional layer, and the last self-attention layer in the stack of multi-head self-attention layers is configured to generate a corresponding multi-head attention output (352) for each corresponding acoustic frame (110); A temporal pooling layer (360), configured to temporally pool the corresponding multi-head attention output (352); and A softmax layer (380) is configured to derive the language prediction representation (302) for the corresponding acoustic frame (110) from the corresponding time-pooled multi-head attention output (352).

4. The method (400) as described in claim 3, wherein, The stacking of multi-head self-attention layers includes either a stack of conformer layers or a stack of transformer layers.

5. The method (400) according to any one of claims 2 to 4, wherein, The LID predictor model (300) is initially trained using a recurrent neural network-transducer RNN-T loss until convergence, and fine-tuned using cross-entropy loss to classify across multiple different languages.

6. The method (400) according to any one of claims 2 to 5, wherein, The multilingual ASR model (201) performs one of the following: Jointly trained with the LID prediction model (300); or It is trained separately from the LID prediction model (300).

7. The method (400) according to any one of claims 1 to 6, wherein, When generating the corresponding higher-order feature representation (212) for each corresponding acoustic frame (110) in the acoustic frame sequence (110) having a corresponding language verification result (290) indicating that the spoken language of the corresponding acoustic frame (110) does not match the language specified by the input language code (202), the audio encoder (210) is configured to receive only the acoustic frame (110) derived from the corresponding acoustic frame (110) as input and generate the corresponding higher-order feature representation (212) without using the input language code (202) or any language prediction representation (302) for the corresponding acoustic frame (110).

8. The method (400) according to any one of claims 1 to 7, wherein, The acoustic frame sequence (110) received as input at the multilingual ASR model (201) represents utterances (106) spoken in at least one of the many different languages ​​supported by the multilingual ASR model (201).

9. The method (400) as claimed in claim 8, wherein, The discourse (106) includes code-mixed discourse (106), which includes one or more words spoken in a first language and one or more other words spoken in a second language.

10. The method (400) according to any one of claims 1 to 9, wherein, The decoder (220, 230) includes: Prediction network (220), the prediction network being configured as follows: Each of the plurality of output steps receives a sequence of non-blank symbols output by the final softmax layer as input; and Each of the plurality of output steps generates a dense representation; and Federation network (230), the federated network being configured as follows: The system receives the dense representation generated by the prediction network (220) at each of the plurality of output steps and the corresponding higher-order feature representation (212) generated by the audio encoder (210) at each of the plurality of output steps as input; and Each of the plurality of output steps generates the probability distribution for possible speech recognition results (120).

11. The method (400) according to any one of claims 1 to 10, wherein, The language verification model (200) is configured to generate the language verification result (290) as binary code, the binary code comprising: When the language verification model (200) determines that the spoken language of the corresponding acoustic frame (110) matches the language specified by the input language code (202), a first value is obtained; or The second value is determined when the language verification model (200) determines that the spoken language of the corresponding acoustic frame (110) does not match the language specified by the input language code (202).

12. The method (400) according to any one of claims 1 to 11, wherein, The language verification model (200) is trained on the following: Multiple positive training utterances, each positive training utterance including audio data of the utterance (106) spoken in the corresponding language, representing the positive language code of the same corresponding language as the specified spoken utterance (106); as well as Multiple negative example utterances, each negative example utterance including audio data of the utterance (106) spoken in a corresponding language that is paired with a negative example language code that is different from the corresponding language of the spoken utterance (106).

13. The method (400) according to any one of claims 1 to 12, wherein, The multilingual ASR model (201) performs one of the following: Jointly trained with the language verification model (200); or It is trained separately from the language validation model (200).

14. The method (400) according to any one of claims 1 to 13, wherein, The input language code (202) is derived from one of the following: The language settings of the computing device that capture utterances (106) in streaming audio, said utterances (106) spoken by the user of the computing device; The country or region where the computing device is located; or The user uploads the language settings when the acoustic frame sequence (110) representing the utterance (106) is uploaded from an external source.

15. The method (400) according to any one of claims 1 to 14, wherein, The operation further includes: Based on the probability distribution of possible speech recognition results (120) generated by the decoder (220, 230) at each of the plurality of output steps, a transcription (120) of the utterance (106) is generated as the output of the multilingual ASR model (201); and The transcript (120) is provided for output to downstream applications. The transcription (120) includes text words in at least one of the various supported languages.

16. A system (100) comprising: Data processing hardware (510); as well as A memory hardware (520) that communicates with and stores instructions to the data processing hardware (510) causes the data processing hardware (510) to perform operations when executed on the data processing hardware (510), the operations including: The acoustic frame sequence (110) representing the utterance (106) and the input language code (202) specifying the language of the utterance (106) are received as input to a multilingual automatic speech recognition (ASR) model (201), which is configured to recognize speech in multiple different supported languages. The language verification model (200) generates a language verification result (290) for the corresponding acoustic frame (110) in the acoustic frame sequence (110) at each of the multiple output steps, the language verification result (290) indicating whether the spoken language of the corresponding acoustic frame (110) matches or does not match the language specified by the input language code (202); For each acoustic frame (110) in the acoustic frame sequence (110) that has a verification result indicating that the spoken language of the corresponding acoustic frame (110) does indeed match the language specified by the input language code (202): The learnable embedding (203) mapped to the input language code (202) is added to the acoustic features (111) derived from the corresponding acoustic frame (110); and The audio encoder (210) of the multilingual ASR model (201), which receives the learnable embedding (203) added to the acoustic features (111) derived from the corresponding acoustic frame (110) as input, generates a higher-order feature representation (212) for the corresponding acoustic frame (110). For each acoustic frame (110) in the acoustic frame sequence (110) having a corresponding language verification result (290) indicating that the spoken language of the corresponding acoustic frame (110) does not match the language specified by the input language code (202), the audio encoder (210) that receives the acoustic features (111) derived from the corresponding acoustic frame (110) as input without adding the learnable embedding (203) mapped to the input language code (202) generates a corresponding higher-order feature representation (212) for the corresponding acoustic frame (110); as well as The decoder (220, 230) of the multilingual ASR model (201) generates a probability distribution of possible speech recognition results (120) at each of the plurality of output steps, the probability distribution of possible speech recognition results (120) being based on the corresponding higher-order feature representation (212) generated by the audio encoder (210).

17. The system (100) as claimed in claim 16, wherein, The operation further includes, for each acoustic frame (110) in the acoustic frame sequence (110) having a corresponding language verification result (290) indicating that the spoken language of the corresponding acoustic frame (110) does not match the language specified by the input language code (202): The language recognition LID predictor model generates a language prediction representation (302) for the corresponding acoustic frame (110); and The language prediction representation (302) is added to the acoustic features (111) derived from the corresponding acoustic frame (110). The generation of the corresponding higher-order feature representation (212) includes the audio encoder (210) receiving the language prediction representation (302) added to the acoustic features (111) derived from the corresponding acoustic frame (110) as input to generate the corresponding higher-order feature representation (212) for the corresponding acoustic frame (110).

18. The system (100) as claimed in claim 17, wherein, The LID prediction model (300) includes: A convolutional layer (307) is configured to downsample each corresponding acoustic frame (110) to generate a corresponding strided convolution (311) as output; A stack of multi-head self-attention layers (350), each having a multi-head attention mechanism, wherein a first self-attention layer in the stack of multi-head self-attention layers is configured to receive the corresponding strided convolution generated as the output from the convolutional layer, and a last self-attention layer in the stack of multi-head self-attention layers is configured to generate a corresponding multi-head attention output (352) for each corresponding acoustic frame (110). A temporal pooling layer (360), configured to temporally pool the corresponding multi-head attention output (352); and A softmax layer (380) is configured to derive the language prediction representation (302) for the corresponding acoustic frame (110) from the corresponding time-pooled multi-head attention output (352).

19. The system (100) as claimed in claim 18, wherein, The stacking of multi-head self-attention layers includes either a stack of conformer layers or a stack of transformer layers.

20. The system (100) as claimed in any one of claims 17 to 19, wherein, The LID predictor model (300) is initially trained using a recurrent neural network-transducer RNN-T loss until convergence, and fine-tuned using cross-entropy loss to classify across multiple different languages.

21. The system (100) as claimed in any one of claims 17 to 20, wherein, The multilingual ASR model (201) performs one of the following: Jointly trained with the LID prediction model (300); or It is trained separately from the LID prediction model (300).

22. The system (400) as claimed in any one of claims 16 to 21, wherein, When generating the corresponding higher-order feature representation (212) for each corresponding acoustic frame (110) in the acoustic frame sequence (110) having a corresponding language verification result (290) indicating that the spoken language of the corresponding acoustic frame (110) does not match the language specified by the input language code (202), the audio encoder (210) is configured to receive only the acoustic frame (110) derived from the corresponding acoustic frame (110) as input and generate the corresponding higher-order feature representation (212) without using the input language code (202) or any language prediction representation (302) for the corresponding acoustic frame (110).

23. The system (400) as claimed in any one of claims 16 to 22, wherein, The acoustic frame sequence (110) received as input at the multilingual ASR model (201) represents utterances (106) spoken in at least one of the many different languages ​​supported by the multilingual ASR model (201).

24. The system (100) as claimed in claim 23, wherein, The discourse (106) includes code-mixed discourse (106), which includes one or more words spoken in a first language and one or more other words spoken in a second language.

25. The system (400) as claimed in any one of claims 16 to 24, wherein, The decoder (220, 230) includes: Prediction network (220), the prediction network being configured as follows: Each of the plurality of output steps receives a sequence of non-blank symbols output by the final softmax layer as input; and Each of the plurality of output steps generates a dense representation; and Federation network (230), the federated network being configured as follows: The system receives the dense representation generated by the prediction network (220) at each of the plurality of output steps and the corresponding higher-order feature representation (212) generated by the audio encoder (210) at each of the plurality of output steps as input; and Each of the plurality of output steps generates the probability distribution for possible speech recognition results (120).

26. The system (400) as claimed in any one of claims 16 to 25, wherein, The language verification model (200) is configured to generate the language verification result (290) as binary code, the binary code comprising: When the language verification model (200) determines that the spoken language of the corresponding acoustic frame (110) matches the language specified by the input language code (202), a first value is obtained; or The second value is determined when the language verification model (200) determines that the spoken language of the corresponding acoustic frame (110) does not match the language specified by the input language code (202).

27. The system (400) as claimed in any one of claims 16 to 26, wherein, The language verification model (200) is trained on the following: Multiple positive training utterances, each positive training utterance including audio data of the utterance (106) spoken in the corresponding language, representing the positive language code of the same corresponding language as the specified spoken utterance (106); as well as Multiple negative example utterances, each negative example utterance including audio data of the utterance (106) spoken in a corresponding language that is paired with a negative example language code that is different from the corresponding language of the spoken utterance (106).

28. The system (400) as claimed in any one of claims 16 to 27, wherein, The multilingual ASR model (201) performs one of the following: Jointly trained with the language verification model (200); or It is trained separately from the language validation model (200).

29. The system (400) as claimed in any one of claims 16 to 28, wherein, The input language code (202) is derived from one of the following: The language settings of the computing device that capture utterances (106) in streaming audio, said utterances (106) spoken by the user of the computing device; The country or region where the computing device is located; or The user uploads the language settings when the acoustic frame sequence (110) representing the utterance (106) is uploaded from an external source.

30. The system (400) as claimed in any one of claims 16 to 29, wherein, The operation further includes: Based on the probability distribution of possible speech recognition results (120) generated by the decoder (220, 230) at each of the plurality of output steps, a transcription (120) of the utterance (106) is generated as the output of the multilingual ASR model (201); and The transcript (120) is provided for output to downstream applications. The transcription (120) includes text words in at least one of the various supported languages.

Images