Rare Word Recognition by Language Model (LM)-Aware MWER Training

Abstract

The method (400) uses an audio encoder (210) to generate a high-level feature representation (212) for each audio frame in a sequence of audio frames (110). Use a decoder (215) to generate a plurality of speech recognition hypotheses based on the high-level feature representation, each hypothesis corresponding to a candidate transcription. Each hypothesis has an associated first likelihood score. Use an external language model (230) to generate a second likelihood score for each hypothesis. Use a learnable fusion module (240) to determine a set of fusion weights for each hypothesis based on the high-level feature representation and the hypothesis. The learnable fusion module is used to generate a third likelihood score for each hypothesis based on the first likelihood score, the second likelihood score, and the set of fusion weights. The audio encoder and decoder are trained in the presence of the external language model using minimum additive error rate training.

Description

Technical Field

[0001] The present disclosure relates to an acoustic recognition model for improving rare word recognition.

Background Art

[0002] Automatic speech recognition (ASR) is an important technology that is increasingly being used in mobile devices or other devices. Generally, an automatic speech recognition ASR system can provide an accurate transcription of what a person has said with low latency (e.g., a very short delay between when a person speaks and when the transcription appears).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a need to provide rare word recognition by improved language model (LM)-aware minimum word error rate (MWER) training.

Means for Solving the Problems

[0005] One aspect of the present disclosure provides a computer-implemented method for training an acoustic recognition model with language model aware minimum word error rate training (MWER) to improve rare word recognition. The computer-implemented method, when executed on data processing hardware, causes the data processing hardware to receive, as input, a sequence of acoustic frames corresponding to an utterance; and generate, at each of a plurality of output steps, a high-order feature representation of a corresponding acoustic frame of the sequence of acoustic frames by using an audio encoder. The operations further include generating, at each of the plurality of output steps, a plurality of speech recognition hypotheses based on the high-order feature representation by using a decoder, where each speech recognition hypothesis corresponds to a candidate transcription of the utterance and each speech recognition hypothesis has a first likelihood score; generating, for each speech recognition hypothesis of the plurality of speech recognition hypotheses, a second likelihood score associated with the speech recognition hypothesis by using an external language model; determining, for each speech recognition hypothesis of the plurality of speech recognition hypotheses, a set of fusion weights based on the high-order feature representation and the speech recognition hypothesis by using a learnable fusion module; and generating, for each speech recognition hypothesis of the plurality of speech recognition hypotheses, a third likelihood score based on the first likelihood score, the second likelihood score, and the set of fusion weights by using the learnable fusion module. Here, the audio encoder is trained by using minimum additive error rate training in the presence of an external language model by adjusting the likelihood score according to the minimum additive error rate among the top k speech recognition hypotheses output from the decoder.

[0006] Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the operation also includes a step of selecting a speech recognition hypothesis among a plurality of speech recognition hypotheses as a speech recognition result (speech recognition result) of a sequence of acoustic frames based on a third likelihood score of the plurality of speech recognition hypotheses. In some examples, the first likelihood score includes a conditional likelihood score and an internal language model likelihood score.

[0007] In some examples, the first likelihood score of a particular speech recognition hypothesis includes a plurality of first language model scores. Each language model score among the plurality of first language model scores corresponds to a particular token of the particular speech recognition hypothesis. The second likelihood score of the particular speech recognition hypothesis includes a plurality of second language model scores. Each language model score among the plurality of second language model scores corresponds to a particular token of the particular speech recognition hypothesis. The set of fusion weights for the particular speech recognition hypothesis includes a plurality of first fusion weights and a plurality of second fusion weights. Each fusion weight among the plurality of first fusion weights corresponds to a particular token of the particular speech recognition hypothesis, and each fusion weight among the plurality of second fusion weights corresponds to a particular token of the particular speech recognition hypothesis. Here, by using a learnable fusion module, the step of generating a third likelihood score for a particular speech recognition hypothesis may include determining an internal language model score based on the plurality of first language model scores and the plurality of first fusion weights, determining an external language model score based on the plurality of second language model scores and the plurality of second fusion weights, and combining the internal language model score and the external language model score.

[0008] In some embodiments, an audio encoder, a decoder, and a learnable fusion module are jointly trained. In some examples, an external language model is trained with text-only data. In some examples, the external language model comprises a neural network language model with a stack of conformer layers or transformer layers. In other examples, the external language model comprises a neural network language model with a stack of long short-term memory (LSTM) layers.

[0009] In some examples, the audio encoder comprises a causal encoder with an initial stack of multi-head attention layers and a non-causal encoder with a final stack of multi-head attention layers stacked on top of the initial stack of multi-head attention layers. In some embodiments, the learnable fusion module is configured to calculate per-token fusion weights for an internal language model score of the decoder and an external language model score of the external language model. In some embodiments, the minimum additive error rate comprises a minimum word error rate. In some examples, the external language model is trained with text-only data. The audio encoder and decoder are trained by using hybrid autoregressive transducer (HAT) factorization to facilitate the integration of an external language model trained with text-only data.

[0010] Other aspects of the present disclosure provide an automatic speech recognition (ASR) system, an audio encoder, a decoder, an external language model, and a learnable fusion module. The audio encoder is configured to receive, as input, a sequence of acoustic frames corresponding to an utterance, and to generate, in each of a plurality of output steps, a high-order feature representation of a corresponding acoustic frame among the sequence of acoustic frames. The decoder is configured to receive, as input, the high-order feature representations generated by the audio encoder in each of the plurality of output steps, and to generate, in each of the plurality of output steps, a plurality of speech recognition hypotheses based on the high-order feature representations received in each of the plurality of output steps, wherein each speech recognition hypothesis corresponds to a candidate transcription of the utterance and each speech recognition hypothesis has an associated first likelihood score. The external language model is configured to receive, as input, the plurality of speech recognition hypotheses generated by the decoder, and to generate, for each of the plurality of speech recognition hypotheses, a second likelihood score associated with the speech recognition hypothesis. The learnable fusion module is configured to receive, for each of the plurality of speech recognition hypotheses, as input, the high-order feature representations generated by the audio encoder in each of the plurality of output steps, to determine a set of fusion weights based on the high-order feature representations and the speech recognition hypotheses, and to generate a third likelihood score based on the first likelihood score, the second likelihood score, and the set of fusion weights. Here, the audio encoder is trained by using minimum addition error rate training in the presence of the external language model by adjusting the likelihood score according to the minimum addition error rate among the top k speech recognition hypotheses output from the decoder.

[0011] Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the operation also includes selecting, as an audio recognition result of a sequence of acoustic frames, an audio recognition hypothesis among a plurality of audio recognition hypotheses based on a third likelihood score of the plurality of audio recognition hypotheses. In some examples, the first likelihood score includes a conditional likelihood score and an internal language model likelihood score.

[0012] In some examples, the first likelihood score of a particular audio recognition hypothesis includes a plurality of first language model scores. Each language model score among the plurality of first language model scores corresponds to a particular token of the particular audio recognition hypothesis. The second likelihood score of the particular audio recognition hypothesis includes a plurality of second language model scores. Each language model score among the plurality of second language model scores corresponds to a particular token of the particular audio recognition hypothesis. The set of fusion weights of the particular audio recognition hypothesis includes a plurality of first fusion weights and a plurality of second fusion weights. Each fusion weight among the plurality of first fusion weights corresponds to a particular token of the particular audio recognition hypothesis. Each fusion weight among the plurality of second fusion weights corresponds to a particular token of the particular audio recognition hypothesis. Here, by using a learnable fusion module, the step of generating the third likelihood score of the particular audio recognition hypothesis may include determining an internal language model score based on the plurality of first language model scores and the plurality of first fusion weights, determining an external language model score based on the plurality of second language model scores and the plurality of second fusion weights, and combining the internal language model score and the external language model score.

[0013] In some embodiments, the audio encoder, decoder, and learnable fusion module are jointly trained. In some examples, the external language model is trained with text-only data. In some examples, the external language model includes a neural network language model having a stack of conformer layers or transformer layers. In other examples, the external language model includes a neural network language model having a stack of long short-term memory (LSTM) layers.

[0014] In some examples, the audio encoder comprises a causal encoder with an initial stack of multi-head attention layers and a non-causal encoder with a final stack of multi-head attention layers stacked on top of the initial stack of multi-head attention layers. In some embodiments, the learnable fusion module is configured to calculate per-token fusion weights for the internal language model scores of the decoder and the external language model scores of the external language model. In some embodiments, the minimum addition error rate comprises a minimum word error rate. In some examples, the external language model is trained on text-only data. The audio encoder and decoder are trained using a hybrid autoregressive transducer (HAT) factorization to facilitate integration of an externally trained text-only data language model.

[0015] Details of one or more embodiments of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings and from the claims.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

DETAILED DESCRIPTION OF THE INVENTION

[0017] Like reference symbols in the various drawings refer to like elements. Automatic speech recognition (ASR) is an important technology that is increasingly being used on mobile devices and other devices. Generally, an automatic speech recognition ASR system is enabled to provide an accurate transcription of what a person has said with low latency (e.g., a very short delay between when a person speaks and when the transcription appears). However, conventional end-to-end (E2E) automatic speech recognition ASR models are trained with audio-text pairs that represent only a small portion of the available text-only data. Thus, conventional E2E automatic speech recognition ASR models may not accurately recognize rare words. In contrast, a language model (LM) that can be trained by using text-only data is enabled to recognize rare words more accurately. The language model LM is combined with the E2E automatic speech recognition ASR model to improve the recognition accuracy of rare words. For example, the E2E automatic speech recognition ASR model score and the language model LM score can be fused during inference by using a hybrid autoregressive transducer (HAT) factorization.

[0018] The embodiments disclosed herein are directed to incorporating a language model (LM) score into an end-to-end automatic speech recognition (ASR) loss term during training. However, end-to-end ASR models are conventionally trained against ground truth labels, which is different from how the LM is incorporated during inference, making it difficult to incorporate the LM score into the training of the ASR model. Thus, the embodiments disclosed herein enable the ASR model score to be fused with the LM score during training, in a manner similar to how the ASR model score and the LM score are fused during inference, by operating the end-to-end ASR model in beam search mode during training to perform LM-aware minimum addition error rate training (e.g., LM-aware minimum word error rate (MWER) training). Here, the ASR model score and the LM score may be fused by using a hybrid autoregressive transducer (HAT) factorization. However, conventional methods for determining and tuning the fusion weights (e.g., parameter sweep) are complex and computationally expensive. Further, after training, the fusion weights are conventionally fixed after training. In contrast, the embodiments disclosed herein construct and train a learnable fusion module (LFM) for determining token-by-token fusion weights for speech recognition hypotheses in a data-dependent manner. Specifically, the LFM determines a fusion weight for each token of a speech recognition hypothesis for an utterance, based on a generated high-order feature representation (e.g., acoustic encoding) for the acoustic frames of the utterance. That is, the fusion weights are determined dynamically during inference. In the disclosed embodiments, the LFM is trained by using LM-aware minimum addition error rate training (e.g., LM-aware minimum word error rate (MWER) training).

[0019] FIG. 1 is a schematic diagram of an example of an acoustic environment 100 and a system 101. In the acoustic environment 100, the way a user 104 interacts with a computing device such as a user device 10 can be via voice input. The user device 10 (generally also referred to as device 10) is configured to capture sound (e.g., streaming audio data) from one or more users 104 within the acoustic environment 100. Here, the streaming audio data can refer to an audible query, a command to the device 10, or a spoken utterance (spoken atalance) 106 by the user 104 that functions as an audible communication already captured by the device 10. The voice-responsive system of the device 10 can handle a query or command by responding to the query and / or causing the command to be performed / carried out by one or more downstream applications.

[0020] The user device 10 can correspond to any computing device that is associated with the user 104 and is capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smartwatches), smart appliances, Internet of Things (IoT) devices, vehicle-mounted infotainment systems, smart displays, smart speakers, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 that communicates with the data processing hardware 12. The memory hardware 114 stores instructions that cause the data processing hardware 12 to perform one or more operations when executed by the data processing hardware 12. The user device 10 further includes an audio system 16 having an audio capture device (e.g., a microphone) 16, 16a for capturing a voice utterance 106 within the acoustic environment 100 and converting it into an electrical signal, and an audio output device (e.g., a speaker) 16, 16b for communicating an audible audio signal (e.g., as output audio data from the device 10). In the example shown, the user device 10 implements a single audio capture device 16a, but without departing from the scope of the present disclosure, the user device 10 may implement an array of audio capture devices 16a, whereby one or more capture devices 16a of the array are not physically present on the user device 10 but can communicate with the audio system 16.

[0021] In an acoustic environment 100, an automatic speech recognition (ASR) system 118 implementing an end-to-end (E2E) automatic speech recognition ASR model 200 and an external language model 230 is present in a remote computing device 70 (e.g., one or more remote servers of a distributed system running in a cloud computing environment) that communicates with a user device 10 via the user 104's user device 10 and / or network 40. In some examples, the automatic speech recognition ASR model 200 comprises a hybrid autoregressive transducer (HAT) model. The user device 10 and / or the remote computing device 70 also receive an utterance 106 spoken by the user 104 and captured by an audio capture device 16a, and are configured to convert the utterance 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by the automatic speech recognition ASR system 118. In the example shown, the user speaks each utterance 106, and the audio subsystem 108 converts the utterance 106 into a corresponding audio data (e.g., acoustic frame) 110 for input to the automatic speech recognition ASR system 118. Thereafter, the automatic speech recognition ASR model 200 receives, as input, an acoustic frame 110 corresponding to the utterance 106 and generates / predicts, as output, a corresponding transcription 120 (e.g., recognition result / hypothesis) of the utterance 106. In the example shown, the automatic speech recognition ASR model 200 may perform streaming speech recognition to generate initial speech recognition results 120, 120a. The external language model 230 may update (i.e., re-score) the initial speech recognition result 120a to generate final speech recognition results 120, 120b. The server 70 comprises data processing hardware 72 and memory hardware 74 that communicates with the data processing hardware 72. The memory hardware 74 stores instructions that, when executed by the data processing hardware 72, cause the data processing hardware 72 to perform one or more operations such as those described herein.

[0022] The user device 10 and / or the remote computing device 70 also execute a user interface generator 107 configured to present the representation of the transcription 120 of the utterance 106 to the user 104 of the user device 10. As will be described in more detail below, the user interface generator 107 may display the initial speech recognition result 120a in a streaming format during a first time 1 and then display the final speech recognition result 120b during a second time 2. In some configurations, the transcription 120 already output from the automatic speech recognition ASR system 118 is processed by, for example, a natural language processing / understanding (NLP / NLU) module executed on the user device 10 or the remote computing device 70 to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executed in any combination of the user device 10 or the remote computing device 70) may convert the transcription to synthetic speech for audible output by the user device 10 and / or other devices.

[0023] In the example shown, user 104 interacts with program or application 50 (e.g., digital assistant application 50) of user device 10 that uses automatic speech recognition ASR system 118. For example, FIG. 1 depicts digital assistant application 50 that displays digital assistant interface 18 on the screen of user 104 and user device 10 that communicates with digital assistant application 50 to depict a conversation between user 104 and digital assistant application 50. In this example, user 104 asks digital assistant application 50, "What time is the concert tonight?" This question from user 104 is a voice utterance 106 that is captured by audio capture device 16a and processed by audio system 16 of user device 10. In this example, audio system 16 receives voice utterance 106 and converts it to acoustic frame 110 for input to automatic speech recognition ASR system 118.

[0024] Continuing the example, automatic speech recognition ASR model 200 encodes acoustic frame 110 while receiving acoustic frame 110 corresponding to utterance 106 as user 104 speaks, and then decodes the encoded acoustic frame 110 into initial speech recognition result 120a. During a first pass, user interface generator 107 presents, via digital assistant interface 18, a representation of initial speech recognition result 120a of utterance 106 in a streaming format to user 104 of user device 10 such that words, parts of words, and / or individual characters appear on the screen as soon as they are spoken. In some examples, the first look-ahead audio context is equal to zero.

[0025] During the second time 2, the user interface generator 107 presents, via the digital assistant interface 18, the representation of the final speech recognition result 120b of the utterance 106, which has been re-scored by the external language model 230, to the user 104 of the user device 10. In some embodiments, the user interface generator 107 replaces the representation of the initial speech recognition result 120a presented at the first time 1 with the representation of the final speech recognition result 120b presented at the second time 2. Here, the first time 1 and the second time 2 may include corresponding timestamps when the user interface generator 107 presents each speech recognition result 120. In this example, the timestamp of the first time 1 indicates that the user interface generator 107 presents the initial speech recognition result 120a at a point in time earlier than the final speech recognition result 120b. For example, since the final speech recognition result 120b is estimated to be more accurate than the initial speech recognition result 120a, the user interface generator 107 ultimately displays the final speech recognition result 120b as the transcription 120 in order to correct any terms that may have been misrecognized in the initial speech recognition result 120a. In this example, the streaming initial speech recognition result 120a already output by the automatic speech recognition ASR model 200 is displayed on the screen of the user device 10 at the first time 1 and is associated with low latency, providing the user 104 with the responsiveness that the user's query is being processed. On the other hand, the final speech recognition result 120b output by the external language model 230 and displayed on the screen at the second time 2 improves the speech recognition quality in terms of accuracy by utilizing additional speech recognition models and / or language models, but the latency increases. However, since the initial speech recognition result 120a is displayed when the user speaks the utterance 106, the higher latency associated with generating and finally displaying the final recognition result is not recognized by the user 104.

[0026] In the example shown in FIG. 1, the digital assistant application 50 may respond to a question posed by the user 104 by using natural language processing / understanding NLP / NLU. Natural language processing / understanding NLP / NLU generally interprets the text language (e.g., the initial speech recognition result 120a and / or the final speech recognition result 120b) and refers to a process of determining whether the text language prompts any action. In this example, the digital assistant application 50 uses natural language processing / understanding NLP / NLU to recognize that the question from the user 104 relates to the user's schedule and more specifically to the concert in the user's schedule. By recognizing these details with natural language processing / understanding NLP / NLU, the automatic assistant returns a response 19 to the user's query, and the response 19 states that "the doors of the venue will open at 7:30 p.m. and the concert will start at 9:00 p.m." In some configurations, natural language processing / understanding NLP / NLU occurs on a remote server 70 that communicates with the data processing hardware 12 of the user device 10.

[0027] FIG. 2 is a schematic diagram of an exemplary architecture of the automatic speech recognition ASR model 200 of FIG. 1. The exemplary automatic speech recognition ASR model 200 may include a hybrid autoregressive transducer HAT model based on a variation of the recurrent neural network - transducer (RNN - T) model. The automatic speech recognition ASR model 200 provides a small computational footprint and, by utilizing fewer memory requirements than conventional automatic speech recognition ASR architectures, makes the automatic speech recognition ASR model 200 suitable for performing complete speech recognition on the user device 10 (e.g., communication with a remote server is not required).

[0028] As shown, the automatic speech recognition ASR model 200 is integrated with an external language model (LM) 230 and includes an audio encoder network 210, a prediction network 300, a joint network 220, and a final softmax layer 250. The prediction network 300 and the joint network 220 may collectively provide a decoder 215. In some examples, the external language model LM230 is trained separately from the audio encoder network 210 and the decoder 215. The audio encoder network 210 is generally similar to an acoustic model (AM) in a conventional automatic speech recognition ASR system and may include a recurrent network of stacked long short-term memory (LSTM) layers or a stack of multi-head attention layers. For example, the audio encoder network 210 reads a sequence X=(x1,x2,···,x T ) of d-dimensional feature vectors (e.g., acoustic frame 110), where

[0029]

Number

[0030] and generates a high-order feature representation 212 at each time step. This high-order feature representation 212 is

[0031]

Number

[0032] is shown as. In some examples, the audio encoder network 210 includes a multi-head attention encoder having 12 layers and 512 attention dimensions. For example, the encoder network 210 may include a stack of multi-head attention layers such as conformer layers or transformer layers. Another exemplary audio encoder network 210 includes a causal (cozarl) encoder having an initial stack of conformer layers (or other types of multi-head attention layers), and a non-causal (non-cozarl) encoder having a final stack of conformer layers (or other types of multi-head attention layers) stacked on top of the initial stack of conformer layers.

[0033] The prediction network 300 may be an LSTM network, and like the language model LM, sequences y0, ···, y of non-blank symbols 252 that have been output by the final softmax layer 250 so far ui-1 into dense or hidden representations

[0034]

Number

[0035] are processed to become 350. As will be described in more detail below in connection with FIG. 3, the representations

[0036]

Number

[0037] 350 has a single embedding vector. In particular, the sequence of non-blank symbols 252 received by the prediction network 300 captures the linguistic dependencies between the non-blank symbols 252 predicted during the previous time steps up to now, and the joint network 220 assists in predicting the probability of the next output symbol or blank symbol during the current time step. As will be described in more detail below, to contribute to techniques for reducing the size of the prediction network 300 without sacrificing the accuracy / performance of the automatic speech recognition ASR model 200, the prediction network 300 has a limited history sequence y ui-n ,···,y ui-1 and may receive this, which is limited to the N previous non-blank symbols 252 output by the final softmax layer 250. In some examples, the prediction network 300 calculates a language model LM likelihood score based on two previous non-blank tokens 252. The prediction network 300 may have two 2048-dimensional LSTM layers, followed by a 740-dimensional projection layer and a 128-unit embedding layer after each layer.

[0038] The joint network 220 combines the high-order feature representation

[0039]

Number

[0040] 212 generated by the audio encoder network 210 with the representation

[0041]

Number

[0042] 350 (i.e., the single embedding vector 350). The joint network 220 combines the acoustic frames

[0043]

Number

[0044] The conditional distribution for the following output symbols conditioned on 110

[0045] [Number]

[0046] Predict 222. In other words, the joint network 220 generates, at each time step, a probability distribution 222 of conditional likelihood scores for the possible speech recognition hypotheses. Here, the "possible speech recognition hypotheses" correspond to a set of output labels each representing a specified natural language symbol / character. For example, when the natural language is English, the set of output labels can include 27 symbols, for example, one label for each of the 26 letters of the English alphabet and one label for specifying a space. Thus, the joint network 220 may output a set of values indicating the occurrence likelihood of each of the predetermined sets of output labels. This set of values can be a vector and can represent a probability distribution for the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, potential punctuation marks, and other symbols), but the set of output labels is not so restricted. For example, the set of output labels can comprise word fragments and / or whole words in addition to, or instead of, graphemes. The output distribution of the joint network 220 can have posterior probability values for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output of the joint network 220

[0047] [Number]

[0048] 222 can have 100 different probability values, one for each output label. Next, by using the probability distribution, candidate orthographic elements (e.g., graphemes, word pieces, and / or words) for use in beam search processing (e.g., by the final softmax layer 250) to determine the transcription 120 are selected and can be assigned scores.

[0049] The joint network 220 also predicts a likelihood score P K {Y1,···,Y ILM (Y k )224 for each of the top K speech recognition hypotheses {Y1,···,Y ILM (Y k )224 that have been generated by beam search performed by the joint network 220. Here, the joint network 200 predicts the likelihood score P

[0050]

Number

[0051] without using 212 or by using a null higher-order feature representation 212. Thus, the likelihood score P ILM (Y k )224 represents the internal language model LM score of the decoder 215 based on a limited set of audio-text training data used to train the audio encoder network 210 and the decoder network 215 and thus may be biased. In some embodiments, the joint network 220 has 740 hidden units and performs beam search using a beam size of 8.

[0052] The external language model LM230 predicts a likelihood score P K {Y1,···,Y ELM (Y k)Predict 232. Here, the external language model LM320 is trained based on a large corpus of text-only data, and the audio encoder network 210 and the decoder network 215 are trained separately. Since the external language model LM230 is trained on a large corpus of text-only data, the internal language model LM likelihood score P ILM (Y k )The bias of 224 can be corrected. An exemplary external language model LM230 includes a neural network language model LM having a stack of 2048-unit LSTM layers (e.g., 4 layers). Another exemplary external language model LM230 includes a neural network language model LM having a conformer with 12 layers and 384 attention dimensions. In yet another example, the external language model LM230 includes a neural network language model LM having a stack of transformer layers.

[0053] Embodiments of this specification incorporate a learnable fusion module (LFM) LFM240 that performs a hybrid autoregressive transducer HAT factorization to fuse or combine the first likelihood scores (i.e., those with likelihood scores 222 and 224) determined by the joint network and the second likelihood score 232 determined by the external language model LM230 for each of the top K speech recognition hypotheses {Y1, ···, Y K} generated by beam search performed by the joint network 220. For each hypothesis {Y1, ···, Y K}, to determine the third likelihood score P(Y k |X)242, the likelihood score P(Y k |X)242 may be expressed as follows.

[0054] logP(Y k |X):=logP E2E (Y k |X)-λ·logP ILM (Y k )+γ·logP ELM (Y k) + C (1) where λ and γ are the internal language model LM scores P ILM (Y k )224 and the external language model LM score and P ELM (Y k )232, and is a vector of fusion weights used to fuse them. C is

[0055]

Number

[0056] a normalization constant that can be selected as such. λ = [λ1, ···, λ L and logP ILM (Y k ) = [s1, ···, s L , then, where L is the number of symbols or tokens of the speech recognition hypothesis Y k . The total contribution score of the internal language model LM may be expressed as follows.

[0057]

Number

[0058] Similarly, γ = [γ1, ···, γ L and logP ELM (Y k ) = [p1, ···, p L , then the total contribution score of the external language model LM230 may be expressed as follows.

[0059]

Number

[0060] Each fusion weight of the plurality of first fusion weights μ = [μ1, ···, μ L corresponds to a specific token of the hypothesis Y k . Similarly, each fusion weight of the plurality of second fusion weights λ = [λ1, ···, λ L corresponds to the hypothesis Y kcorresponds to a specific token. In other words, the plurality of fusion weights μ = [μ1, ···, μ L and λ = [λ1, ···, λ L are token-wise fusion weights. This is in stark contrast to the conventional hybrid autoregressive transducer HAT factorization where the same scalar fusion weight is applied to all tokens of hypothesis Y k . Similarly, each likelihood score P ILM (Y k )224 and P ELM (Y k )232 corresponds to a specific token of hypothesis Y k .

[0061] The audio encoder network 210 and the decoder network 215 may be trained by using an additional error rate-based loss, such as a minimum word error rate MWER-based loss, to incorporate a previously separately trained external language model LM230. An exemplary minimum word error rate MWER-based loss that may be used to train the audio encoder network 210 and the decoder network 215 is expressed as follows.

[0062] [Number]

[0063] where NWE(Y k , Y * ) represents the number of word errors between Y k and Y * . In some embodiments,[[]]

[0064] [Number]

[0065] The composite loss of is minimized during training to improve the stability of learning. An exemplary value of θ is 0.04. An exemplary minimum word error rate MWER loss for training is the top 8 hypotheses Yk It is calculated by using

[0066] The learnable fusion module LFM240 is based on the high-order feature representation 212 and the hypothesis Y k Based on the features of, for each speech recognition hypothesis Y k For, the fusion weight vector μ = [μ1, ···, μ L of the internal language model LM score 224 of the decoder 215 and the fusion weight vector γ = [γ1, ···, γ L of the external language model LM score 232 of the external language model LM230 are configured to be determined, and it is equipped with a trained fusion weight determination model 244. In particular, here, the fusion weight vectors μ = [μ1, ···, μ L and γ = [γ1, ···, γ L are dynamically determined during inference based on data (that is, based on the high-order feature representation 212 and the hypothesis Y k ). That is, the fusion weight vectors μ = [μ1, ···, μ L and γ = [γ1, ···, γ L are not determined during training and are then fixed during inference. In some examples, the fusion weight determination model 244 has a similar transformer architecture to an attention-based decoder, and by performing causal self-attention on the features of the hypothesis Y k and cross-attention on the high-order feature representation 212, the fusion weight vectors μ = [μ1, ···, μ k and γ = [γ1, ···, γ L of the hypothesis Y LDetermine []. The exemplary fusion weight determination model 244 includes three transformer layers and 256 or 320 attention dimensions. In some examples, the fusion weight determination model 244 is trained in a speech recognition task based on the minimum word error rate MWER-based loss of Equation (4) and / or Equation (5) after the audio encoder network 210 and the decoder network 215 are trained and while the coefficients of the audio encoder network 210 and the decoder network 215 are fixed. In other examples, the audio encoder network 210, the decoder network 215, and the fusion weight determination model 244 are co-trained in a speech recognition task based on the minimum word error rate MWER-based loss of Equation (4) and / or Equation (5).

[0067] The final softmax layer 250 receives the likelihood score P(Y k |X)242 and selects the speech recognition hypothesis Y that has the highest likelihood score P(Y k |X)242 to generate the transcription 120. The automatic speech recognition ASR model 200 does not make an assumption of conditional independence. Rather, the prediction of each symbol y k 252 is conditioned not only on the acoustic frame 110 but also on the sequence of labels 252 that have been output so far, y u ,···,y ui-n ,···,y ui-1 . The automatic speech recognition ASR model 200 assumes that the output symbol 252 is independent of future acoustic frames 110, which enables the automatic speech recognition ASR model 200 to be used in a streaming manner.

[0068] FIG. 3 is a schematic diagram of an exemplary prediction network 300 of the automatic speech recognition ASR model 200. The prediction network 300 takes as input a sequence y of non-blank symbols 252a~252n ui-n ,···,y ui-1has received, which is limited to the N previous non-blank symbols 252a~252n that have been output by the final softmax layer 250. In some examples, N is equal to 2. In other examples, N is equal to 5, but the disclosure is non-limiting and N may be equal to any integer. The sequence of non-blank symbols 252a~252n represents the initial speech recognition result 120a (Figure 1). In some embodiments, the prediction network 300 includes a multi-head attention mechanism 302 that shares an embedding matrix 304 across each head 302A~302H of the multi-head attention mechanism. In one example, the multi-head attention mechanism 302 includes four heads. However, any number of heads may be used by the multi-head attention mechanism 302. In particular, the multi-head attention mechanism significantly improves performance while minimizing an increase in model size. As will be described in more detail below, each head 302A~302H includes a row of its own position vector 308, and instead of increasing the model size by concatenating the outputs 318A~318H from all heads, the outputs 318A~318H are averaged by the head average module 322.

[0069] Referring to the first head 302A of the multi-head attention mechanism 302, the head 302A uses the shared embedding matrix 304 to receive, as input, the sequence y of non-blank symbols 252a~252n that have been received at corresponding time steps from a plurality of time steps ui-n ,···,y ui-1 For each non-blank symbol in, the corresponding embeddings 306, 306a~306n (e.g.,

[0070]

Number

[0071] generates. In particular, since the shared embedding matrix 304 is shared among all the heads of the multi-head attention mechanism 302, all the other heads 302B to 302H all generate the same corresponding embedding 306 for each non-blank symbol. Head 302A also, for the sequence y of non-blank symbols 252a to 252n ui-n , ···, y ui-1 for each corresponding non-blank symbol, respective position vectors PV Aa~An 308, 308Aa to 308An (for example

[0072]

Number

[0073] ) are assigned. Each position vector PV308 assigned to each non-blank symbol indicates the position of the history of the sequence of non-blank symbols (for example, the N previous non-blank symbols 252a to 252n already output by the final softmax layer 250). For example, the first position vector PV Aa is assigned to the latest position in the history, while the last position vector PV An is assigned to the last position of the history of the N previous non-blank symbols already output by the final softmax layer 250. In particular, each of the embeddings 306 may include the same dimension (i.e., dimension size) as each of the position vectors PV308.

[0074] For the sequence y of non-blank symbols 252a to 252n ui-n , ···, y ui-1 the corresponding embeddings already generated by the shared embedding matrix 304 for each non-blank symbol are the same in all the heads 302A to 302H of the multi-head attention mechanism 302, but each head 302A to 302H defines a different set / row of the position vectors 308. For example, the first head 302A defines the rows 308Aa to 308An of the position vector PV Aa~An . The second head 302B defines different rows of 308Ba to 308Bn of the position vector PV Ba~Bn , ···, and the H-th head 302H defines the position vector PVHa~Hn Define other different rows from 308Ha to 308Hn of .

[0075] For each non - blank symbol in the sequence of received non - blank symbols 252a to 252n, the first head 302A also weights the corresponding embedding 306 in proportion to the similarity between the corresponding embedding and each of the position vectors PV308 assigned thereto via the weight layer 310. In some examples, the similarity may include cosine similarity (e.g., cosine distance). In the example shown, the weight layer 310 outputs a sequence 312Aa to 312An of weighted embeddings 312, each of which is associated with the corresponding embedding 306 weighted in proportion to the respective position vector PV308 assigned thereto. In other words, for each embedding 306, the weighted embedding 312 already output by the weight layer 310 may correspond to the dot product between the embedding 306 and each position vector PV308. The weighted embedding 312 can be interpreted as attending to the embedding in proportion to how similar the embedding is to the positioning associated with each position vector PV308. To increase the calculation speed, the prediction network 300 comprises a non - recurrent layer, and thus, the sequence 312Aa to 312An of weighted embeddings is not concatenated but instead is averaged by a weighted average module 316, and as the output from the first head 302A, a weighted average 318A of the weighted embeddings 312Aa to 312An represented by the following equation is generated.

[0076]

Equation

[0077] In Equation (6), h represents the index of the head 302, n represents the position within the context, and e represents the embedding dimension. Further in Equation (6), H, N, and d e, has a size corresponding to the dimension. The position vector PV308 does not need to be trainable and may include random values. In particular, even if the weighted embedding 312 is averaged, the position vector PV308 can potentially store position history information, thus reducing the need to provide recurrent connections in each layer of the prediction network 300.

[0078] The operations described above for the first head 302A are similarly performed for each of the other heads 302B - 302H of the multi - head attention mechanism 302. By different sets of the positioned vectors PV308 defined by each head 302, the weight layer 310 outputs sequences of weighted embeddings 312Ba - 312Bn, 312Ha - 312Hn in each of the other heads 302B - 302H that are different from the sequence of weighted embeddings 312Aa - 312Aa in the first head 302A. Then, the weighted average module 316 generates weighted averages 318B - 318H of the corresponding weighted embeddings 312 of the non - blank symbol sequences as outputs from the other corresponding heads 302B - 302H.

[0079] In the example shown, the prediction network 300 includes a head average module 322 that averages the weighted averages 318A - 318H already output from the corresponding heads 302A - 302H. The projection layer 326 having SWISH receives the output 324 from the head average module 322 corresponding to the average of the weighted averages 318A - 318H as input and may generate a projection output 328 as output. The final layer normalization 330 may normalize the projection output 328 to provide a single embedding vector

[0080]

Number

[0081] 350. The prediction network 300 provides a single embedding vector at each of a plurality of time steps following the initial time step.

[0082]

Number

[0083] Generate only 350. In some configurations, the prediction network 300 does not implement the multi-head attention mechanism 302 and only performs the operations described above with respect to the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa to 312An is a single embedding vector

[0084]

Number

[0085] It is simply passed through the projection layer 326 and the layer normalization 330 to provide 350. In some embodiments, a parameter binding between the prediction network 300 and the joint network 220 is applied to further reduce the size of the RNN-T decoder, i.e., the prediction network 300 and the joint network 220. Specifically, for the vocabulary size |V| and the embedding dimension d e in the case of, the shared embedding matrix 304 in the prediction network is

[0086]

Number

[0087] is. On the other hand, the last hidden layer has a dimension size d in the joint network 220, and the weights of the feed-forward projection from the hidden layer to the output logits are h and the weights of the feed-forward projection from the hidden layer to the output logits are

[0088]

Number

[0089] As a result, the vocabulary contains additional blank tokens. Therefore, the feed-forward layer corresponding to the last layer of the joint network 220 has a weight matrix [d h , |V|]. By associating the embedding dimension d e of the prediction network 300 with the dimension d h of the last hidden layer of the joint network 220, the feed-forward projection weights of the joint network 220 and the shared embedding matrix 304 of the prediction network 300 can share their weights for all non-blank symbols via a simple transpose transformation. Since the two matrices share all values, the RNN-T decoder only needs to store the values once in memory instead of storing two individual matrices. By setting the size of the embedding dimension d e to be equal to the size of the hidden layer dimension d h , the RNN-T decoder reduces the number of parameters equal to the product of the embedding dimension d e and the vocabulary size |V|. This weight association corresponds to a regularization technique.

[0090] Figure 4 is a flowchart of an exemplary arrangement of the operations of a computer-implemented method 400 for improving rare word recognition. In operation 402, the method 400 includes receiving a sequence of acoustic frames 110 corresponding to the utterance 106. In operation 404, the method 400 includes generating, using the audio encoder 210, a high-order feature representation 212 of the corresponding acoustic frame 110 among the received sequence of acoustic frames 110 at each of a plurality of output steps.

[0091] In operation 406, the method 400 includes generating, using the decoder 215, a plurality of speech recognition hypotheses {Y1, ···, Y K} based on the high-order feature representation 212 at each of a plurality of output steps. Each speech recognition hypothesis Y k corresponds to a candidate transcription of the utterance 106. Each speech recognition hypothesis Y kIt has a first likelihood score associated with likelihood scores 222 and 224.

[0092] In operation 408, method 400 uses an external language model 230 to generate a second likelihood score P ELM (Y k ) 232 for each speech recognition hypothesis Y in a plurality of speech recognition hypotheses {Y1, ···, Y k}. K} among each speech recognition hypothesis Y k For speech recognition hypothesis Y k There is a second likelihood score P ELM (Y k )232.

[0093] In operation 410, method 400 uses a learnable fusion module LFM 240 to determine a set of fusion weight sets λ = {λ1, ···, λ L} and γ = {γ1, ···, γ L} for each speech recognition hypothesis Y in a plurality of speech recognition hypotheses {Y1, ···, Y K} based on the high-order feature representation 212 and the speech recognition hypothesis Y k . In operation 412, method 400 uses the learnable fusion module LFM 240 to generate a third likelihood score 242 for each speech recognition hypothesis Y in a plurality of speech recognition hypotheses {Y1, ···, Y K} based on the first likelihood scores 222, 224, the second likelihood score 232, and the fusion weight sets λ = {λ1, ···, λ L} and γ = {γ1, ···, γ L}. K} among each speech recognition hypothesis Y k For high-order feature representation 212 and speech recognition hypothesis Y k Based on the fusion weight set λ = {λ1, ···, λ L} and γ = {γ1, ···, γ L} of the set. In operation 412, method 400 uses the learnable fusion module LFM 240 to generate a third likelihood score 242 for each speech recognition hypothesis Y in a plurality of speech recognition hypotheses {Y1, ···, Y K} among each speech recognition hypothesis Y k Based on the first likelihood scores 222, 224, the second likelihood score 232, and the fusion weight set λ = {λ1, ···, λ L} and γ = {γ1, ···, γ L} of the set.

[0094] In operation 414, method 400 selects a speech recognition hypothesis Y from a plurality of speech recognition hypotheses {Y1, ···, Y K} as the speech recognition result 120 of the sequence of acoustic frames 110 based on the third likelihood score 242 of the plurality of speech recognition hypotheses {Y1, ···, Y K}. K} Based on the third likelihood score 242 of the plurality of speech recognition hypotheses {Y1, ···, Y K} among each speech recognition hypothesis Y k For the speech recognition hypothesis Y of the set.

[0095] FIG. 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems and methods described in this document. Computing device 500 is intended to represent various forms of digital computers, such as a laptop, desktop, workstation, personal digital assistant, server, blade server, mainframe, and other appropriate computers. The components shown here, their connections and relationships, and their functions are for purposes of illustration only and are not intended to limit the embodiments of the invention described and / or claimed in this document.

[0096] The computing device 500 includes a processor 510 (i.e., data processing hardware) that can be used to implement data processing hardware 12 and / or 72, a memory 520 (i.e., memory hardware) that can be used to implement memory hardware 14 and / or 74, a storage device 530 (i.e., memory hardware) that can be used to implement memory hardware 14 and / or 74, a high-speed interface / controller 540 connected to the memory 520 and the high-speed expansion port 550, and a low-speed interface / controller 560 connected to the low-speed bus 570 and the storage device 530. Each of the components (510, 520, 530, 540, 550, and 560) is interconnected by using various buses and may be installed on a common motherboard or exist in other ways as needed. The processor 510 is capable of displaying graphical information of a graphical user interface (GUI) on an external input / output device such as a display 580 coupled to the high-speed interface 540 by processing instructions for execution within the computing device 500 that include instructions stored in the memory 520 or the storage device 530. In other embodiments, multiple memories and multiple types of memories may be used, along with multiple processors and / or multiple buses as needed. Also, multiple computing devices 500 may be connected, and each device may perform a part of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0097] Memory 520 stores information non - temporarily within computing device 500. Memory 520 may be a computer - readable medium, a volatile memory unit(s), or a non - volatile memory unit(s). The non - temporary memory 520 may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) temporarily or permanently for use by computing device 500. Examples of non - volatile memory include, but are not limited to, flash memory and read - only memory (ROM) / programmable read - only memory (PROM) / erasable programmable read - only memory (EPROM) / electrically erasable programmable read - only memory (EEPROM) (e.g., typically used for firmware such as a boot program). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase - change memory (PCM), and disks or tapes.

[0098] Storage device 530 can provide large - capacity storage for computing device 500. In some embodiments, storage device 530 is a computer - readable medium. In various different embodiments, storage device 530 may be a floppy (registered trademark) disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid - state memory device, or an array of devices comprising a storage area network or other configuration of devices. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product comprises instructions that, when executed, perform one or more of the methods as described above. The information carrier is a computer - readable medium or a machine - readable medium such as memory 520, storage device 530, or memory on processor 510.

[0099] The high-speed controller 540 manages the bandwidth-intensive operations of the computing device 500, and the low-speed controller 560 manages the low-bandwidth-intensive operations. Such an assignment of duties is merely an example. In some embodiments, the high-speed controller 540 is coupled to a high-speed expansion port 550 that can receive the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and various expansion cards (not shown). In some embodiments, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (USB, Bluetooth®, Ethernet®, wireless Ethernet®, etc.), or may be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., via a network adapter.

[0100] As shown in the figure, the computing device 500 can be implemented in many different forms. For example, it may be implemented as a standard server 500a, or multiple times within a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0101] Various embodiments of the systems and techniques described herein can be implemented in digital electronics and / or optical circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can be special or general purpose, and can include at least one programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, and executable and / or interpretable in a programmable system comprising one or more computer programs.

[0102] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an “application,” an “app,” or a “program.” Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0103] These computer programs (also known as programs, software, software applications, or code) comprise machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" are used to refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic device (PLD)) that provide machine instructions and / or data to a programmable processor comprising a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0104] The processes and logical flows described in this specification can be performed by one or more programmable processors, also referred to as data processing hardware, acting on input data and performing functions by generating output, by executing one or more computer programs. The processes and logical flows can also be performed by special-purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read only memory, a random access memory, or both. The basic elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operatively coupled to receive data from, or transmit data to, or both, such devices. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special-purpose logic circuitry.

[0105] To interact with a user, aspects of the present invention can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touch screen and optionally a keyboard and a pointing device, such as a mouse or trackball, by which the user can input to the computer. Other types of devices can also be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input from the user can be received in any form, including acoustic, voice, or tactile input. Further, the computer can interact with the user by sending and receiving documents to and from the devices used by the user, for example, by sending a web page to a web browser on the user's client device in response to a received request from a web browser.

[0106] Unless otherwise stated to the contrary, the phrase "at least one of A, B, or C" is intended to refer to any combination or subset of A, B, and C, such as (1) at least one A alone, (2) at least one B alone, (3) at least one C alone, (4) at least one A having at least one B, (5) at least one A having at least one C, (6) at least one B having at least one C, (7) at least one A having at least one B and at least one C, and so on. Further, unless otherwise stated to the contrary, the phrase "at least one of A, B, and C" is intended to refer to any combination or subset of A, B, and C, such as (1) at least one A alone, (2) at least one B alone, (3) at least one C alone, (4) at least one A having at least one B, (5) at least one A having at least one C, (6) at least one B having at least one C, (7) at least one A having at least one B and at least one C, and so on. Further, unless otherwise stated to the contrary, "A or B" is intended to refer to any combination of A and B, such as (1) A alone, (2) B alone, and (3) A and B.

[0107] Some embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (400) to cause data processing hardware (510) to perform operations, the operations comprising: receiving, as input, a sequence of acoustic frames (110) corresponding to an utterance (106); generating, for each of a plurality of output steps, a high-level feature representation (212) of a corresponding acoustic frame (110) of the sequence of acoustic frames (110) by using an audio encoder (210); generating, for each of the plurality of output steps, a plurality of speech recognition hypotheses based on the high-level feature representation (212) in each of the plurality of output steps, each speech recognition hypothesis corresponding to a candidate transcription of the utterance (106) and each speech recognition hypothesis having an associated first likelihood score; generating, for each of the plurality of speech recognition hypotheses, a second likelihood score associated with the speech recognition hypothesis by using an external language model (230); determining, for each of the plurality of speech recognition hypotheses, a set of fusion weights based on the high-level feature representation (212) and the speech recognition hypothesis by using a learnable fusion module (240); generating, for each of the plurality of speech recognition hypotheses, a third likelihood score based on the first likelihood score, the second likelihood score, and the set of fusion weights by using the learnable fusion module (240); wherein the audio encoder (210) is trained by using minimum addition error rate training in the presence of the external language model (230) by adjusting a likelihood score according to a minimum addition error rate among the top k speech recognition hypotheses output from the decoder (215); A computer-implemented method (400).

2. The operations further comprise selecting, as a speech recognition result (120) of the sequence of acoustic frames (110), one of the plurality of speech recognition hypotheses based on the third likelihood score of the plurality of speech recognition hypotheses. The computer-implemented method (400) according to claim 1.

3. The first likelihood score includes a conditional likelihood score and an internal language model likelihood score. The computer-implemented method (400) according to claim 1.

4. The first likelihood score of the specific speech recognition hypothesis includes a plurality of first language model scores, and each of the plurality of first language model scores corresponds to a specific token of the specific speech recognition hypothesis. The second likelihood score of the specific speech recognition hypothesis includes a plurality of second language model scores, and each of the plurality of second language model scores corresponds to a specific token of the specific speech recognition hypothesis. The set of fusion weights of the specific speech recognition hypothesis includes a plurality of first fusion weights and a plurality of second fusion weights. Each of the plurality of first fusion weights corresponds to a specific token of the specific speech recognition hypothesis, and each of the plurality of second fusion weights corresponds to a specific token of the specific speech recognition hypothesis. The step of generating the third likelihood score of the specific speech recognition hypothesis by using the learnable fusion module (240) includes: Determining an internal language model score based on the plurality of first language model scores and the plurality of first fusion weights; Determining an external language model score based on the plurality of second language model scores and the plurality of second fusion weights; Combining the internal language model score and the external language model score. It includes. The computer-implemented method (400) according to any one of claims 1 to 3.

5. The audio encoder (210), the decoder (215), and the learnable fusion module (240) are co-trained. The computer-implemented method (400) according to any one of claims 1 to 3.

6. The external language model (230) is trained with text-only data. The computer-implemented method (400) according to any one of claims 1 to 3.

7. The external language model (230) includes a neural network language model having a stack of conformer layers or transformer layers. The computer-implemented method (400) according to any one of claims 1 to 3.

8. The external language model (230) comprises a neural network language model having a stack of long short-term memory (LSTM) layers. The computer-implemented method (400) according to any one of claims 1 to 3. **Claim 9** The audio encoder (210) A causal encoder having an initial stack of multi-head attention layers, and A non-causal encoder having a final stack of multi-head attention layers stacked on top of the initial stack of the multi-head attention layers, Comprising The computer-implemented method (400) according to any one of claims 1 to 3. **Claim 10** The learnable fusion module (240) is configured to calculate per-token fusion weights for the internal language model score of the decoder (215) and the external language model score of the external language model (230). The computer-implemented method (400) according to any one of claims 1 to 3. **Claim 11** The minimum addition error rate has a minimum word error rate. The computer-implemented method (400) according to any one of claims 1 to 3. **Claim 12** The external language model (230) is trained with text-only data, The audio encoder (210) and the decoder (215) are trained by using hybrid autoregressive transducer (HAT) factorization to facilitate the integration of the external language model (230) trained with the text-only data. The computer-implemented method (400) according to any one of claims 1 to 3. **Claim 13** An automatic speech recognition ASR system (118), wherein the automatic speech recognition ASR system (118) An audio encoder (210), wherein the audio encoder (210) Receiving, as input, a sequence of acoustic frames (110) corresponding to an utterance (106); and Generating, at each of a plurality of output steps, a high-order feature representation (212) of a corresponding acoustic frame (110) of the sequence of acoustic frames (110). The audio encoder (210) configured to perform A decoder (215), wherein the decoder (215) receiving, as input, the high-order feature representation (212) generated by the audio encoder (210) in each of the plurality of the output steps; generating a plurality of speech recognition hypotheses based on the high-order feature representation (212) received in each of the plurality of the output steps, wherein each speech recognition hypothesis corresponds to a candidate transcription of the utterance (106), and each speech recognition hypothesis has an associated first likelihood score; the decoder (215) configured to perform; an external language model (230), wherein the external language model (230) receiving, as input, the plurality of speech recognition hypotheses generated by the decoder (215); generating, for each of the plurality of speech recognition hypotheses, a second likelihood score associated with the speech recognition hypothesis; the external language model (230) configured to perform; a learnable fusion module (240), wherein the learnable fusion module (240) for each of the plurality of speech recognition hypotheses receiving, as input, the high-order feature representation (212) generated by the audio encoder (210) in each of the plurality of the output steps; determining a set of fusion weights based on the high-order feature representation (212) and the speech recognition hypothesis; generating a third likelihood score based on the first likelihood score, the second likelihood score, and the set of fusion weights; the learnable fusion module (240) configured to perform; comprising the audio encoder (210) is trained by using minimum added error rate training in the presence of the external language model (230) by adjusting a likelihood score according to a minimum added error rate among the top k speech recognition hypotheses output from the decoder (215); an automatic speech recognition ASR system (118).

14. The operation further comprises selecting, as a speech recognition result of the sequence of the acoustic frames (110), one of the plurality of speech recognition hypotheses based on the third likelihood score of the plurality of speech recognition hypotheses. The automatic speech recognition ASR system (118) according to claim 13.

15. The first likelihood score includes a conditional likelihood score and an internal language model likelihood score. The automatic speech recognition ASR system (118) according to claim 13. **Claim 16** The first likelihood score of a specific speech recognition hypothesis includes a plurality of first language model scores, and each of the plurality of first language model scores corresponds to a specific token of the specific speech recognition hypothesis. The second likelihood score of the specific speech recognition hypothesis includes a plurality of second language model scores, and each of the plurality of second language model scores corresponds to a specific token of the specific speech recognition hypothesis. The set of fusion weights of the specific speech recognition hypothesis includes a plurality of first fusion weights and a plurality of second fusion weights. Each of the plurality of first fusion weights corresponds to a specific token of the specific speech recognition hypothesis, and each of the plurality of second fusion weights corresponds to a specific token of the specific speech recognition hypothesis. The learnable fusion module (240) determines an internal language model score based on the plurality of first language model scores and the plurality of first fusion weights; determines an external language model score based on the plurality of second language model scores and the plurality of second fusion weights; combines the internal language model score and the external language model score; and is configured to generate the third likelihood score of the specific speech recognition hypothesis thereby. The automatic speech recognition ASR system (118) according to any one of claims 13 to 15. **Claim 17** The audio encoder (210), the decoder (215), and the learnable fusion module (240) are co-trained. The automatic speech recognition ASR system (118) according to any one of claims 13 to 15. **Claim 18** The external language model (230) is trained with text-only data. The automatic speech recognition ASR system (118) according to any one of claims 13 to 15. **Claim 19** The external language model (230) includes a neural network language model having a stack of conformer layers or transformer layers. The automatic speech recognition ASR system (118) according to any one of claims 13 to 15. **Claim 20** The external language model (230) comprises a neural network language model having a stack of long short-term memory (LSTM) layers. The automatic speech recognition ASR system (118) according to any one of claims 13 to 15. **Claim 21** The audio encoder (210) A causal encoder having an initial stack of multi-head attention layers, A non-causal encoder having a final stack of multi-head attention layers stacked on top of the initial stack of multi-head attention layers, comprises The automatic speech recognition ASR system (118) according to any one of claims 13 to 15. **Claim 22** The trainable fusion module (240) is configured to calculate per-token fusion weights for the internal language model score of the decoder (215) and the external language model score of the external language model (230). The automatic speech recognition ASR system (118) according to any one of claims 13 to 15. **Claim 23** The minimum addition error rate has a minimum word error rate. The automatic speech recognition ASR system (118) according to any one of claims 13 to 15. **Claim 24** The external language model (230) is trained with text-only data, The audio encoder (210) and the decoder (215) are trained by using hybrid autoregressive transducer (HAT) factorization to facilitate the integration of the external language model (230) trained with the text-only data. The automatic speech recognition ASR system (118) according to any one of claims 13 to 15.

Images