Rare Word Recognition by Language Model (LM)-Aware MWER Training

Abstract

The method (400) uses an audio encoder (210) to generate a high-level feature representation (212) for each audio frame in a sequence of audio frames (110). Use a decoder (215) to generate a plurality of speech recognition hypotheses based on the high-level feature representation, each hypothesis corresponding to a candidate transcription. Each hypothesis has an associated first likelihood score. Use an external language model (230) to generate a second likelihood score for each hypothesis. Use a learnable fusion module (240) to determine a set of fusion weights for each hypothesis based on the high-level feature representation and the hypothesis. The learnable fusion module is used to generate a third likelihood score for each hypothesis based on the first likelihood score, the second likelihood score, and the set of fusion weights. The audio encoder and decoder are trained in the presence of the external language model using minimum additive error rate training.

Description

[Technical field]

[0001] This disclosure relates to speech recognition models for improving rare word recognition. [Background technology]

[0002] Automatic speech recognition (ASR) is an important technology that is increasingly used in mobile and other devices. Generally, automatic speech recognition (ASR) systems are enabled to provide accurate transcriptions of what a person says with low latency (e.g., a very short delay between what the person says and the appearance of the transcription). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2022 / 005465 Summary of the Invention [Problem to be solved by the invention]

[0004] It is desired to provide rare word recognition with improved Language Model (LM)-aware Minimum Word Error Rate (MWER) training. [Means for solving the problem]

[0005] One aspect of the present disclosure provides a computer-implemented method for training a speech recognition model to improve rare word recognition with language model aware minimum word error rate (MWER) training. The computer-implemented method, when executed on data processing hardware, causes the data processing hardware to perform operations comprising: receiving as an input a sequence of acoustic frames corresponding to an utterance; and generating, at each of a plurality of output steps, a high-order feature representation of a corresponding acoustic frame of the sequence of acoustic frames using an audio encoder. The operations further comprise: generating, at each of the plurality of output steps, a plurality of speech recognition hypotheses based on the high-order feature representation using a decoder, each speech recognition hypothesis corresponding to a candidate transcription of the utterance, each speech recognition hypothesis having a first likelihood score; and generating, for each speech recognition hypothesis of the plurality of speech recognition hypotheses, a second likelihood score associated with the speech recognition hypothesis using an external language model. The operations also include determining, for each speech recognition hypothesis of the plurality of speech recognition hypotheses, a set of fusion weights based on the high-level feature representation and the speech recognition hypothesis using a learnable fusion module, and generating, for each speech recognition hypothesis of the plurality of speech recognition hypotheses, a third likelihood score based on the first likelihood score, the second likelihood score, and the set of fusion weights using the learnable fusion module, where the audio encoder is trained using minimum additive error rate training in the presence of an external language model by adjusting the likelihood scores according to a minimum additive error rate among the top-k speech recognition hypotheses output from the decoder.

[0006] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operations also include selecting a speech recognition hypothesis from the plurality of speech recognition hypotheses as a speech recognition result for the sequence of acoustic frames based on a third likelihood score of the plurality of speech recognition hypotheses. In some examples, the first likelihood score comprises a conditional likelihood score and an internal language model likelihood score.

[0007] In some examples, the first likelihood score for the particular (particular) speech recognition hypothesis comprises a plurality of first language model scores. Each language model score of the plurality of first language model scores corresponds to a particular token of the particular speech recognition hypothesis. The second likelihood score for the particular speech recognition hypothesis comprises a plurality of second language model scores. Each language model score of the plurality of second language model scores corresponds to a particular token of the particular speech recognition hypothesis. The fusion weight set for the particular speech recognition hypothesis comprises a plurality of first fusion weights and a plurality of second fusion weights. Each fusion weight of the plurality of first fusion weights corresponds to a particular token of the particular speech recognition hypothesis, and each fusion weight of the plurality of second fusion weights corresponds to a particular token of the particular speech recognition hypothesis. Here, using the learnable fusion module to generate the third likelihood score for the particular speech recognition hypothesis may include determining an internal language model score based on the plurality of first language model scores and the plurality of first fusion weights, determining an external language model score based on the plurality of second language model scores and the plurality of second fusion weights, and combining the internal language model score and the external language model score.

[0008] In some implementations, the audio encoder, the decoder, and the learnable fusion module are jointly trained. In some examples, the external language model is trained with text-only data. In some examples, the external language model comprises a neural network language model comprising a stack of conformer or transformer layers. In other examples, the external language model comprises a neural network language model comprising a stack of long short-term memory (LSTM) layers.

[0009] In some examples, the audio encoder comprises a causal encoder comprising an initial stack of multi-head attention layers, and a non-causal encoder comprising a final stack of multi-head attention layers layered on the initial stack of multi-head attention layers. In some implementations, the learnable fusion module is configured to calculate per-token fusion weights for the internal language model scores of the decoder and the external language model scores of the external language model. In some implementations, the minimum additive error rate comprises a minimum word error rate. In some examples, the external language model is trained on text-only data. The audio encoder and decoder are trained using hybrid autoregressive transducer (HAT) factorization to facilitate integration of the external language model trained on text-only data.

[0010] Another aspect of the disclosure provides an automatic speech recognition (ASR) system, an audio encoder, a decoder, an external language model, and a learnable fusion module. The audio encoder is configured to receive as an input a sequence of acoustic frames corresponding to an utterance and generate, at each of a plurality of output steps, a high-level feature representation of a corresponding acoustic frame of the sequence of acoustic frames. The decoder is configured to receive as an input the high-level feature representation generated by the audio encoder at each of the plurality of output steps and generate, based on the high-level feature representation received at each of the plurality of output steps, a plurality of speech recognition hypotheses, each of which corresponds to a candidate transcription of the utterance and each of which has an associated first likelihood score. The external language model is configured to receive as an input the plurality of speech recognition hypotheses generated by the decoder and generate, for each of the plurality of speech recognition hypotheses, a second likelihood score associated with the speech recognition hypothesis. The learnable fusion module is configured to receive, for each speech recognition hypothesis of the plurality of speech recognition hypotheses, as an input, a high-level feature representation generated by the audio encoder at each of the plurality of output steps, determine a fusion weight set based on the high-level feature representation and the speech recognition hypothesis, and generate a third likelihood score based on the first likelihood score, the second likelihood score, and the fusion weight set, where the audio encoder is trained using minimum additive error rate training in the presence of an external language model by adjusting the likelihood score according to a minimum additive error rate among the top-k speech recognition hypotheses output from the decoder.

[0011] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operations also include selecting a speech recognition hypothesis from the plurality of speech recognition hypotheses as a speech recognition result for the sequence of acoustic frames based on a third likelihood score of the plurality of speech recognition hypotheses. In some examples, the first likelihood score comprises a conditional likelihood score and an internal language model likelihood score.

[0012] In some examples, the first likelihood score for the particular speech recognition hypothesis comprises a plurality of first language model scores. Each language model score of the plurality of first language model scores corresponds to a particular token of the particular speech recognition hypothesis. The second likelihood score for the particular speech recognition hypothesis comprises a plurality of second language model scores. Each language model score of the plurality of second language model scores corresponds to a particular token of the particular speech recognition hypothesis. The fusion weight set for the particular speech recognition hypothesis comprises a plurality of first fusion weights and a plurality of second fusion weights. Each fusion weight of the plurality of first fusion weights corresponds to a particular token of the particular speech recognition hypothesis. Each fusion weight of the plurality of second fusion weights corresponds to a particular token of the particular speech recognition hypothesis. Wherein using the learnable fusion module to generate a third likelihood score for the particular speech recognition hypothesis may include determining an internal language model score based on the plurality of first language model scores and the plurality of first fusion weights, determining an external language model score based on the plurality of second language model scores and the plurality of second fusion weights, and combining the internal language model score and the external language model score.

[0013] In some implementations, the audio encoder, the decoder, and the learnable fusion module are jointly trained. In some examples, the external language model is trained with text-only data. In some examples, the external language model comprises a neural network language model comprising a stack of conformer or transformer layers. In other examples, the external language model comprises a neural network language model comprising a stack of long short-term memory (LSTM) layers.

[0014] In some examples, the audio encoder comprises a causal encoder comprising an initial stack of multi-head attention layers and an a-causal encoder comprising a final stack of multi-head attention layers layered on the initial stack of multi-head attention layers. In some implementations, the learnable fusion module is configured to calculate per-token fusion weights for the internal language model scores of the decoder and the external language model scores of the external language model. In some implementations, the minimum additive error rate comprises a minimum word error rate. In some examples, the external language model is trained on text-only data. The audio encoder and decoder are trained using hybrid autoregressive transducer (HAT) factorization to facilitate integration of the external language model trained on text-only data.

[0015] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a schematic diagram of an exemplary audio environment using an automatic speech recognition (ASR) system to transcribe audio. [Diagram 2] FIG. 2 is a schematic diagram of an example hybrid autoregressive transducer (HAT) model architecture for the automatic speech recognition ASR system of FIG. [Diagram 3] FIG. 1 is a schematic diagram of an exemplary linked reduced prediction network. [Figure 4] 1 is a flowchart of an exemplary arrangement of operations of a computer-implemented method for improving rare word recognition. [Diagram 5] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Like reference symbols in the various drawings indicate like elements. Automatic speech recognition (ASR) is an important technology that is increasingly used in mobile devices and other devices. Generally, automatic speech recognition ASR systems are enabled to provide accurate transcriptions of what a person says with low latency (e.g., negligible delay between what the person says and the transcription appearing). However, conventional end-to-end (E2E) automatic speech recognition ASR models are trained with audio-text pairs that represent a small portion of the available text-only data. Thus, conventional E2E automatic speech recognition ASR models may not accurately recognize rare words. In contrast, language models (LMs), which may be trained using text-only data, are enabled to more accurately recognize rare words. The language models LMs are combined with E2E automatic speech recognition ASR models to improve the recognition accuracy of rare words. For example, the E2E automatic speech recognition ASR model scores and the language model LM scores may be fused during inference using hybrid autoregressive transducer (HAT) factorization.

[0018] The embodiments disclosed herein are directed to incorporating language model LM scores into E2E ASR loss terms during training. However, since E2E ASR models are traditionally trained on ground truth labels, which is different from the way language model LM is incorporated during inference, it may be difficult to incorporate language model LM scores into the training of the ASR model. Thus, the embodiments disclosed herein enable E2E ASR model scores to be fused with language model LM scores during training, similar to the way E2E ASR model scores and language model LM scores are fused during inference, by performing language model LM-aware minimum additive error rate training (e.g., language model LM-aware minimum word error rate (MWER)) by operating the E2E ASR model in beam search mode during training. Here, the E2E ASR model scores and language model LM scores may be fused by using hybrid autoregressive transducer HAT factorization. However, conventional methods for determining and adjusting fusion weights (e.g., parameter sweep) are complex and computationally expensive. Moreover, after training, the fusion weights are customarily fixed after training. In contrast, embodiments disclosed herein configure and train a learnable fusion module (LFM) for determining per-token fusion weights for speech recognition hypotheses in a data-dependent manner. Specifically, the learnable fusion module LFM determines fusion weights for each token of a speech recognition hypothesis of an utterance based on a generated high-level feature representation (e.g., an acoustic encoding) for an acoustic frame of the utterance. That is, the fusion weights are dynamically determined during inference. In the disclosed embodiments, the learnable fusion module LFM is trained using a language model LM-aware minimum additive error rate training (e.g., a language model LM-aware minimum word error rate (MWER) training).

[0019] 1 is a schematic diagram of an example audio environment 100 and system 101. In the audio environment 100, a user 104's way of interacting with a computing device, such as a user device 10, may be through voice input. The user device 10 (also generally referred to as device 10) is configured to capture sounds (e.g., streaming audio data) from one or more users 104 in the audio environment 100. Here, the streaming audio data may refer to spoken utterances 106 by the user 104 that function as audible queries, commands to the device 10, or audible communications captured by the device 10. The voice-enabled system of the device 10 may address the queries or commands by responding to the queries and / or causing the commands to be performed / fulfilled by one or more downstream applications.

[0020] The user device 10 may correspond to any computing device associated with a user 104 and enabled to receive audio data. Some examples of the user device 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, Internet of Things (IoT) devices, vehicle-mounted infotainment systems, smart displays, smart speakers, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 in communication with the data processing hardware 12. The memory hardware 114 stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 having audio capture devices (e.g., microphones) 16, 16a for capturing and converting audio utterances 106 into electrical signals within the audio environment 100, and audio output devices (e.g., speakers) 16, 16b for communicating audible audio signals (e.g., as output audio data from the device 10). In the illustrated example, the user device 10 implements a single audio capture device 16a, however, without departing from the scope of this disclosure, the user device 10 may implement an array of audio capture devices 16a, whereby one or more capture devices 16a of the array may not be physically present on the user device 10 but may communicate with the audio system 16.

[0021] In the speech environment 100, an automatic speech recognition (ASR) system 118 implementing an end-to-end (E2E) automatic speech recognition ASR model 200 and an external language model 230 resides on a user device 10 of a user 104 and / or a remote computing device 70 (e.g., one or more remote servers of a distributed system executing in a cloud computing environment) that communicates to the user device 10 via a network 40. In some examples, the automatic speech recognition ASR model 200 comprises a hybrid autoregressive transducer (HAT) model. The user device 10 and / or the remote computing device 70 also comprises an audio subsystem 108 configured to receive an utterance 106 spoken by the user 104 and captured by the audio capture device 16a, and to convert the utterance 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by the automatic speech recognition ASR system 118. In the illustrated example, a user speaks each utterance 106, and an audio subsystem 108 converts the utterance 106 into corresponding audio data (e.g., acoustic frames) 110 for input to an automatic speech recognition ASR system 118. An automatic speech recognition ASR model 200 then receives the acoustic frames 110 corresponding to the utterance 106 as input and generates / predicts a corresponding transcription 120 (e.g., recognition result / hypothesis) of the utterance 106 as output. In the illustrated example, the automatic speech recognition ASR model 200 may perform streaming speech recognition to generate initial speech recognition results 120, 120a. An external language model 230 may update (i.e., rescore) the initial speech recognition results 120a to generate final speech recognition results 120, 120b. The server 70 includes data processing hardware 72 and memory hardware 74 in communication with the data processing hardware 72. Memory hardware 74 stores instructions that, when executed by data processing hardware 72, cause data processing hardware 72 to perform one or more operations, such as those described herein.

[0022] The user device 10 and / or the remote computing device 70 also execute a user interface generator 107 configured to present a representation of the transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 may display initial speech recognition results 120a in a streaming format during a first time 1 and then display final speech recognition results 120b during a second time 2. In some configurations, the transcription 120 output from the automatic speech recognition ASR system 118 is processed by a natural language processing / understanding (NLP / NLU) module executing, for example, on the user device 10 or the remote computing device 70 to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device 10 or the remote computing device 70) may convert the transcription into synthetic speech for audible output by the user device 10 and / or other devices.

[0023] In the illustrated example, a user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) of a user device 10 that uses an automatic speech recognition ASR system 118. For example, FIG. 1 depicts a user 104 communicating with the digital assistant application 50 and the digital assistant application 50 displaying a digital assistant interface 18 on the screen of the user device 10 to depict a conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, "What time is the concert tonight?" This question from the user 104 is a speech utterance 106 that is captured by the audio capture device 16a and processed by the audio system 16 of the user device 10. In this example, the audio system 16 receives the speech utterance 106 and converts it into acoustic frames 110 for input to the automatic speech recognition ASR system 118.

[0024] Continuing with the example, the automatic speech recognition ASR model 200 receives acoustic frames 110 corresponding to an utterance 106 as the user 104 speaks, while encoding the acoustic frames 110, and then decodes the encoded acoustic frames 110 into initial speech recognition results 120a. During the first time 1, the user interface generator 107 presents to the user 104 of the user device 10, via the digital assistant interface 18, a representation of the initial speech recognition results 120a of the utterance 106 in a streaming format such that words, parts of words, and / or individual characters appear on the screen as they are spoken. In some examples, the first look-ahead audio context is equal to zero.

[0025] During the second time 2, the user interface generator 107 presents, via the digital assistant interface 18, a representation of the final speech recognition result 120b of the utterance 106, rescored by the external language model 230, to the user 104 of the user device 10. In some implementations, the user interface generator 107 replaces the representation of the initial speech recognition result 120a presented at the first time 1 with the representation of the final speech recognition result 120b presented at the second time 2, where the first time 1 and the second time 2 may include timestamps corresponding to when the user interface generator 107 presents each speech recognition result 120. In this example, the timestamp for the first time 1 indicates that the user interface generator 107 presents the initial speech recognition result 120a at an earlier time than the final speech recognition result 120b. For example, since the final speech recognition result 120b is estimated to be more accurate than the initial speech recognition result 120a, the user interface generator 107 will eventually display the final speech recognition result 120b as the transcription 120 to correct any terms that may have been misrecognized in the initial speech recognition result 120a. In this example, the streaming initial speech recognition result 120a output by the automatic speech recognition ASR model 200 is displayed on the screen of the user device 10 at a first time 1, associated with low latency and providing the user 104 with responsiveness that the user's query is being processed. Meanwhile, the final speech recognition result 120b output by the external language model 230 and displayed on the screen at a second time 2 utilizes additional speech recognition and / or language models to improve speech recognition quality in terms of accuracy but increase latency. However, because the initial speech recognition results 120a are displayed as the user speaks the utterance 106, the higher latency associated with generating and ultimately displaying the final recognition results is not perceived by the user 104.

[0026] In the example shown in FIG. 1, the digital assistant application 50 may respond to a question posed by the user 104 by using natural language processing / understanding NLP / NLU. Natural language processing / understanding NLP / NLU generally refers to the process of interpreting written language (e.g., initial speech recognition results 120a and / or final speech recognition results 120b) and determining whether the written language prompts some action. In this example, the digital assistant application 50 uses natural language processing / understanding NLP / NLU to recognize that the question from the user 104 is about the user's schedule and more specifically, a concert on the user's schedule. By recognizing these details with natural language processing / understanding NLP / NLU, the automated assistant returns a response 19 to the user's query, which states, "The venue doors open at 7:30 p.m. and the concert begins at 9 p.m." In some configurations, the natural language processing / understanding NLP / NLU occurs on a remote server 70 that communicates to the data processing hardware 12 of the user device 10.

[0027] 2 is a schematic diagram of an example architecture of the automatic speech recognition ASR model 200 of FIG. 1. The example automatic speech recognition ASR model 200 may include a hybrid autoregressive transducer HAT model based on a variation of a recurrent neural network-transducer (RNN-T) model. The automatic speech recognition ASR model 200 provides a small computational footprint and utilizes smaller memory requirements than conventional automatic speech recognition ASR architectures, making the automatic speech recognition ASR model 200 suitable for performing speech recognition entirely on the user device 10 (e.g., no communication with a remote server is required).

[0028] As shown, the automatic speech recognition ASR model 200 includes an audio encoder network 210, a prediction network 300, a collaboration network 220, and a final softmax layer 250, along with integration with an external language model (LM) 230. The prediction network 300 and the collaboration network 220 may collectively provide a decoder 215. In some examples, the external language model LM 230 is trained separately from the audio encoder network 210 and the decoder 215. The audio encoder network 210 is generally similar to an acoustic model (AM) in a conventional automatic speech recognition ASR system and may include a recurrent network of stacked long short-term memory (LSTM) layers or a stack of multi-head attention layers. For example, the audio encoder network 210 may be configured to recognize a sequence X=(x1, x2, . . . , x T ) is read out, where:

[0029]

number

[0030] and at each time step generates a high-level feature representation 212, which can be expressed as:

[0031]

number

[0032] In some examples, the audio encoder network 210 includes a multi-head attention encoder having 12 layers and an attention dimension of 512. For example, the encoder network 210 may include a stack of multi-head attention layers, such as conformer layers or transformer layers. Another example audio encoder network 210 includes a causal encoder having an initial stack of conformer layers (or other type of multi-head attention layer) and a non-causal encoder having a final stack of conformer layers (or other type of multi-head attention layer) layered on the initial stack of conformer layers.

[0033] The prediction network 300 may be an LSTM network, and as a language model LM, it is assumed that the sequence of non-blank symbols 252 y 0 , . . . , y ui-1 , dense or hidden representation

[0034]

number

[0035] 350. As will be explained in more detail below in relation to FIG.

[0036]

number

[0037] 350 comprises a single embedding vector. In particular, the sequence of non-blank symbols 252 received by the prediction network 300 captures the linguistic dependencies between the non-blank symbols 252 predicted during previous time steps, helping the collaboration network 220 predict the probability of the next output symbol or blank symbol during the current time step. As will be described in more detail below, to contribute to the technique for reducing the size of the prediction network 300 without sacrificing the accuracy / performance of the automatic speech recognition ASR model 200, the prediction network 300 uses a limited historical sequence of non-blank symbols 252 y ui-n ,···,y ui-1 may receive N, which is limited to the N previous non-blank symbols 252 output by the final softmax layer 250. In some examples, the prediction network 300 calculates a language model LM likelihood score based on the two previous non-blank tokens 252. The prediction network 300 may have two 2048-dimensional LSTM layers, each followed by a 740-dimensional projection layer, as well as a 128-unit embedding layer.

[0038] The collaborative network 220 uses the high-level feature representation generated by the audio encoder network 210.

[0039]

number

[0040] 212 and the representation generated by the prediction network 300

[0041]

number

[0042] 350 (i.e., a single embedding vector 350). The joint network 220 combines the acoustic frame

[0043]

number

[0044] Conditional distribution for the next output symbol conditioned on 110

[0045]

number

[0046] 222. In other words, the collaborative network 220 generates a probability distribution 222 of conditional likelihood scores for possible speech recognition hypotheses at each time step, where a "probable speech recognition hypothesis" corresponds to a set of output labels each representing a symbol / character of a specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters of the English alphabet and one label specifying a space. Thus, the collaborative network 220 may output a set of values ​​indicating the likelihood of occurrence of each of the set of given output labels. This set of values ​​may be a vector and may indicate a probability distribution for the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, potential punctuation, and other symbols), but the set of output labels is not so limited. For example, the set of output labels may comprise word fragments and / or whole words in addition to or instead of graphemes. The output distribution of the collaborative network 220 may comprise posterior probability values ​​for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output of the collaborative network 220

[0047]

number

[0048] 222 may be provided with 100 different probability values, one for each output label. The probability distributions are then used to select candidate orthographic elements (e.g., graphemes, word fragments, and / or words) for the beam search process (e.g., by the final softmax layer 250) to determine the transcription 120, and scores may be assigned to them.

[0049] The collaborative network 220 also generates the top-K speech recognition hypotheses {Y1, . . . , Y K}, a likelihood score P ILM (Y k ) 224, where the collaboration network 200 predicts the likelihood score P ILM (Y k )224 as a high-level feature representation

[0050]

number

[0051] 212 or by using a null high-level feature representation 212. Thus, the likelihood score P ILM (Y k ) 224 represents the internal language model LM score of the decoder 215 that is based on a restricted set of audio-text training data used to train the audio encoder network 210 and the decoder network 215, and may therefore be biased. In some implementations, the collaboration network 220 has 740 hidden units and performs a beam search using a beam size of 8.

[0052] The external language model LM230 is generated by the beam search performed by the collaborative network 220 using the top K speech recognition hypotheses {Y1, . . . , Y K}, a likelihood score P ELM (Y k) 232, where the external language model LM 320 is trained on a large corpus of text-only data and is trained separately from the audio encoder network 210 and the decoder network 215. Because the external language model LM 230 is trained on a large corpus of text-only data, the internal language model LM likelihood score P ILM (Y k ) 224. An exemplary external language model LM 230 comprises a neural network language model LM having a stack of 2048-unit LSTM layers (e.g., 4 layers). Another exemplary external language model LM 230 comprises a neural network language model LM having a conformer with 12 layers and an attention dimension of 384. In yet another example, the external language model LM 230 comprises a neural network language model LM having a stack of transformer layers.

[0053] The embodiment herein uses the top-K speech recognition hypotheses {Y1, . . . , Y K}, for each hypothesis {Y1, ,Y K}, the third likelihood score P(Y k The network incorporates a learnable fusion module (LFM) LFM 240 that performs a hybrid autoregressive transducer HAT factorization to fuse or combine the first likelihood score determined by the collaborative network (i.e., comprising likelihood scores 222 and 224) and the second likelihood score determined by the external language model LM 230 232 to determine likelihood score P(Y |X) 242. k |X) 242 may be expressed as follows:

[0054] logP(Y k |X):=logP E2E (Y k |X)-λ logP ILM (Y k )+γ logP ELM (Y k)+C (1) where λ and γ are the internal language model LM scores P ILM (Y k )224 and the external language model LM score and P ELM (Y k ) 232 and C is a vector of fusion weights used to fuse

[0055]

number

[0056] is a normalization constant that can be chosen to be λ = [λ1,...,λ L ] and logP ILM (Y k ) = [s1, ,s L ], where L is the number of speech recognition hypotheses Y k The total contribution score of the internal language model LM may be expressed as:

[0057]

number

[0058] Similarly, γ = [γ1, ,γ L ] and logP ELM (Y k )=[p1, ,p L ], then the total contribution score of the external language model LM230 may be expressed as:

[0059]

number

[0060] Multiple first fusion weights μ = [μ1, , μ L Each fusion weight of ] is a hypothesis Y k Similarly, multiple second fusion weights λ = [λ1, , λ L Each fusion weight of ] is a hypothesis Y kIn other words, multiple fusion weights μ = [μ1, , μ L ] and λ = [λ1, , λ L ] is the per-token fusion weight, which is the weight of the hypothesis Y k This is in stark contrast to conventional hybrid autoregressive transducer (HAT) factorization, where the same scalar fusion weight is applied to all tokens in . Similarly, multiple likelihood scores P ILM (Y k )224 and P ELM (Y k )232 likelihood scores are the likelihood of the hypothesis Y k Corresponds to a specific token.

[0061] The audio encoder network 210 and the decoder network 215 may be trained using an additive error rate based loss, such as a minimum word error rate (MWER) based loss, to incorporate a previously separately trained external language model LM 230. An exemplary minimum word error rate (MWER) based loss that may be used to train the audio encoder network 210 and the decoder network 215 is expressed as follows:

[0062]

number

[0063] During the ceremony, NWE(Y k ,Y * ) is Y k and Y * In some implementations,

[0064]

number

[0065] The composite loss of is minimized during training to improve the stability of learning. An exemplary value of θ is 0.04. An exemplary minimum word error rate (MWER) loss for training isk It is calculated by using

[0066] The learnable fusion module LFM 240 combines the high-level feature representation 212 and the hypotheses Y k Based on the features of each speech recognition hypothesis Y k , the fusion weight vector μ=[μ1, . . . , μ L ] and the fusion weight vector γ = [γ1, . . . , γ L ] and includes a trained fusion weight determination model 244. In particular, here, the fusion weight vector μ=[μ1, . . . , μ L ] and γ = [γ1, ,γ L ] is based on the data (i.e., the high-level feature representation 212 and the hypothesis Y k The fusion weight vector μ = [μ1, ,μ L ] and γ = [γ1, ,γ L ] is not determined during training, but is then fixed during inference. In some examples, the fusion weight determination model 244 comprises a transformer architecture similar to that of the attention-based decoder, and is based on the hypothesis Y k By performing causal self-attention on the features of and cross-attention on the high-level feature representation 212, we can obtain the hypothesis Y k The fusion weight vector μ=[μ1, ,μ L ] and γ = [γ1, ,γ L]. An exemplary fusion weight determination model 244 has three transformer layers and an attention dimension of 256 or 320. In some examples, the fusion weight determination model 244 is trained on a speech recognition task based on the minimum word error rate MWER-based loss in equation (4) and / or equation (5) after the audio encoder network 210 and the decoder network 215 have been trained and while the coefficients of the audio encoder network 210 and the decoder network 215 are fixed. In other examples, the audio encoder network 210, the decoder network 215, and the fusion weight determination model 244 are jointly trained on a speech recognition task based on the minimum word error rate MWER-based loss in equation (4) and / or equation (5).

[0067] The final softmax layer 250 calculates the likelihood score P(Y k |X)242 and the highest likelihood score P(Y k Speech recognition hypothesis Y has |X)242 k to generate the transcription 120. The automatic speech recognition ASR model 200 does not make any conditional independence assumptions, but rather selects each symbol y u The prediction of 252 is based not only on the acoustic frame 110 but also on the sequence y ui-n ,···,y ui-1 The automatic speech recognition ASR model 200 assumes that the output symbols 252 are independent of future acoustic frames 110, allowing the automatic speech recognition ASR model 200 to be used in a streaming manner.

[0068] 3 is a schematic diagram of an exemplary prediction network 300 of the automatic speech recognition ASR model 200. The prediction network 300 receives as input a sequence y of non-blank symbols 252a-252n. ui-n ,···,y ui-1, which is limited to the N previous non-blank symbols 252a-252n output by the final softmax layer 250. In some examples, N is equal to 2. In other examples, N is equal to 5, but the disclosure is non-limiting and N may be equal to any integer. The sequence of non-blank symbols 252a-252n represents the initial speech recognition result 120a (FIG. 1). In some implementations, the prediction network 300 includes a multi-head attention mechanism 302 that shares a shared embedding matrix 304 across each head 302A-302H of the multi-head attention mechanism. In one example, the multi-head attention mechanism 302 includes four heads. However, any number of heads may be used by the multi-head attention mechanism 302. Notably, the multi-head attention mechanism significantly improves performance while minimizing the increase in model size. As explained in more detail below, each head 302A-302H has its own row of position vectors 308, and rather than increasing the model size by concatenating the outputs 318A-318H from all heads, the outputs 318A-318H are instead averaged by a head averaging module 322.

[0069] Referring to the first head 302A of the multi-head attention mechanism 302, the head 302A uses a shared embedding matrix 304 to estimate the sequence y of non-empty symbols 252a-252n received as input at corresponding time steps from multiple time steps. ui-n ,···,y ui-1 For each non-blank symbol in

[0070]

number

[0071] Notably, the shared embedding matrix 304 is shared among all heads of the multi-head attention mechanism 302, so that all other heads 302B-H generate the same corresponding embedding 306 for each non-whitespace symbol. Head 302A also generates a sequence y ui-n ,···,y ui-1 For each corresponding non-blank symbol, Aa~An 308, 308Aa-308An (e.g.

[0072]

number

[0073] Each position vector PV 308 assigned to each non-whitespace symbol indicates a historical position in the sequence of non-whitespace symbols (e.g., N previous non-whitespace symbols 252a-252n output by the final softmax layer 250). For example, the first position vector PV Aa is assigned to the most recent position in the history, while the last position vector PV An is assigned to the last position in the history of the N previous non-empty symbols output by the final softmax layer 250. Notably, each of the embeddings 306 may include the same dimensionality (i.e., dimension size) as each of the position vectors PV 308.

[0074] A sequence y of non-blank symbols 252a through 252n ui-n ,···,y ui-1 The corresponding embedding generated by the shared embedding matrix 304 for each non-blank symbol in is the same for all heads 302A-H of the multi-head attention mechanism 302, but each head 302A-H defines a different set / row of position vectors 308. For example, the first head 302A defines a position vector PV Aa~An The second head 302B defines rows 308Aa to 308An of the position vector PV Ba~Bn The H-th head 302H defines different rows of the position vector PVHa~Hn Define other different lines from 308Ha to 308Hn.

[0075] For each non-blank symbol in the sequence of received non-blank symbols 252a-252n, the first head 302A also weights the corresponding embedding 306 via the weighting layer 310 in proportion to the similarity between the corresponding embedding and the respective position vector PV 308 assigned thereto. In some examples, the similarity may include a cosine similarity (e.g., cosine distance). In the example shown, the weighting layer 310 outputs a sequence 312Aa-312An of weighted embeddings 312, each associated with a corresponding embedding 306 weighted in proportion to the respective position vector PV 308 assigned thereto. In other words, the weighted embedding 312 output by the weighting layer 310 for each embedding 306 may correspond to a dot product between the embedding 306 and the respective position vector PV 308. The weighted embedding 312 may be interpreted as attending to the embedding in proportion to how similar the embedding is to the positioning associated with the respective position vector PV 308. To increase computational speed, the prediction network 300 includes a non-recurrent layer, so that the sequence of weighted embeddings 312Aa-312An are not concatenated but instead averaged by a weighted average module 316 to produce as output from the first head 302A a weighted average 318A of the weighted embeddings 312Aa-312An, expressed as follows:

[0076]

number

[0077] In equation (6), h represents the index of the head 302, n represents the position in the context, and e represents the embedding dimension. e, with the size of the corresponding dimension. The position vector PV 308 does not need to be trainable and may contain random values. Notably, even if the weighted embedding 312 is averaged, the position vector PV 308 can potentially preserve position history information, alleviating the need to provide recurrent connections at each layer of the prediction network 300.

[0078] The operations described above with respect to the first head 302A are performed similarly for each of the other heads 302B-302H of the multi-head attention mechanism 302. Due to the different set of positioning vectors PV 308 defined by each head 302, the weight layer 310 outputs a sequence of weighted embeddings 312Ba-312Bn, 312Ha-312Hn for each of the other heads 302B-302H that is different from the sequence of weighted embeddings 312Aa-312Aa for the first head 302A. The weighted average module 316 then generates a weighted average 318B-318H of each of the corresponding weighted embeddings 312 of the sequences of non-blank symbols as output from each of the other corresponding heads 302B-302H.

[0079] In the illustrated example, the prediction network 300 includes a head averaging module 322 that averages the weighted averages 318A-H output from corresponding heads 302A-H. A projection layer 326 with SWISH may receive as an input an output 324 from the head averaging module 322 corresponding to the average of the weighted averages 318A-H and generate as an output a projection output 328. A final layer normalization 330 may normalize a single embedding vector at a corresponding time step from multiple time steps.

[0080]

number

[0081] The prediction network 300 may normalize the projection outputs 328 to provide a single embedding vector 350 at each of multiple time steps following the initial time step.

[0082]

number

[0083] Generates only 350. In some configurations, the prediction network 300 does not implement a multi-head attention mechanism 302, and performs only the operations described above with respect to the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-312An is calculated as a single embedding vector

[0084]

number

[0085] It is simply passed through a projection layer 326 and layer normalization 330 to provide 350. In some implementations, parameter binding between the prediction network 300 and the collaborative network 220 is applied to further reduce the size of the RNN-T decoder, i.e., the prediction network 300 and the collaborative network 220. Specifically, the vocabulary size |V| and the embedding dimension d e In this case, the shared embedding matrix 304 in the prediction network is

[0086]

number

[0087] Meanwhile, the last hidden layer has a dimension size of d h and the weights of the feedforward projections from the hidden layer to the output logits are

[0088]

number

[0089] and the vocabulary includes an additional blank token. Therefore, the feedforward layer corresponding to the last layer of the collaborative network 220 has a weight matrix [d h , |V]|. The prediction network 300 has an embedding dimension d e The size of the last hidden layer of the collaborative network 220 is the dimension d h By combining the embedding dimension d, the feedforward projection weights of the joint network 220 and the shared embedding matrix 304 of the prediction network 300 can share their weights for all non-blank symbols via a simple transpose transformation. Because the two matrices share all values, the RNN-T decoder only needs to store the values ​​once in memory instead of storing two individual matrices. e The size of the hidden layer dimension d h By setting the embedding dimension d equal to the size of e We reduce the number of parameters equal to the product of |V| and the vocabulary size |V|. This weight binding corresponds to a regularization technique.

[0090] 4 is a flowchart of an exemplary arrangement of operations of a computer-implemented method 400 for improving rare word recognition. At operation 402, the method 400 comprises receiving a sequence of acoustic frames 110 corresponding to speech 106. At operation 404, the method 400 comprises generating, using an audio encoder 210, a high-level feature representation 212 of a corresponding acoustic frame 110 of the received sequence of acoustic frames 110 at each of a plurality of output steps.

[0091] In operation 406, the method 400 uses the decoder 215 to generate a plurality of speech recognition hypotheses {Y1, . . . , Y K} for each speech recognition hypothesis Y k corresponds to a candidate transcription of utterance 106. khas an associated first likelihood score comprising likelihood scores 222, 224.

[0092] In operation 408, the method 400 generates a plurality of speech recognition hypotheses {Y1, . . . , Y K} for each speech recognition hypothesis Y k For speech recognition hypothesis Y k The second likelihood score P associated with ELM (Y k ) 232.

[0093] The method 400, in operation 410, uses a learnable fusion module LFM 240 to generate a plurality of speech recognition hypotheses {Y1, . . . , Y K} for each speech recognition hypothesis Y k For the high-level feature representation 212 and speech recognition hypothesis Y k Based on the fusion weight set λ={λ1, , λ L} and γ = {γ1, , γ L In operation 412, the method 400 uses the learnable fusion module LFM 240 to determine a set of speech recognition hypotheses {Y1, . . . , Y K} for each speech recognition hypothesis Y k , λ = {λ 1 , . . . , λ 1 , . . . , λ 2 ...2 , . . . , λ 2 , . . L} and γ = {γ1, , γ L} and generating a third likelihood score 242 based on

[0094] At operation 414, the method 400 generates a plurality of speech recognition hypotheses {Y1, . . . , Y K} as a speech recognition result 120 for the sequence of acoustic frames 110. K} of speech recognition hypotheses Y k The method includes a step of selecting:

[0095] 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems and methods described herein. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functions are for illustrative purposes only and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0096] Computing device 500 includes a processor 510 (i.e., data processing hardware) that may be used to implement data processing hardware 12 and / or 72, a memory 520 (i.e., memory hardware) that may be used to implement memory hardware 14 and / or 74, a storage device 530 (i.e., memory hardware) that may be used to implement memory hardware 14 and / or 74, a high-speed interface / controller 540 that connects to memory 520 and a high-speed expansion port 550, a low-speed interface / controller 560 that connects to a low-speed bus 570 and storage device 530. Each of the components (510, 520, 530, 540, 550, and 560) are interconnected using various buses and may be mounted on a common motherboard or present in other manners as needed. The processor 510 processes instructions, comprising instructions stored in memory 520 or storage device 530, for execution within the computing device 500, enabling it to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 coupled to a high speed interface 540. In other implementations, multiple processors and / or multiple buses may be used, as well as multiple memories and types of memory, as needed. Also, multiple computing devices 500 may be connected, with each device performing some of the required operations (e.g., as a bank of servers, a group of blade servers, or a multi-processor system).

[0097] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0098] The storage device 530 can provide mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be an array of devices, including floppy disk devices, hard disk devices, optical disk devices, or tape devices, flash memory or other similar solid-state memory devices, or devices in a storage area network or other configuration. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product comprises instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.

[0099] The high-speed controller 540 manages the bandwidth-intensive operations of the computing device 500, and the low-speed controller 560 manages the low-bandwidth intensive operations. Such an allocation of duties is merely an example. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (USB, Bluetooth, Ethernet, wireless Ethernet, etc.) or may be coupled, for example, via a network adapter, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router.

[0100] The computing device 500 can be implemented in many different forms, as shown in the figure: for example, it may be implemented as a standard server 500a, or multiple times within a cluster of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0101] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general purpose and may include implementations in one or more computer programs executable and / or interpretable by a programmable system having at least one programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from the storage system and to transmit data and instructions to the storage system.

[0102] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0103] These computer programs (also known as programs, software, software applications, or codes) comprise machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor comprising a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0104] The process and logic flow described herein may be performed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to operate on input data and generate output to perform functions. The process and logic flow may also be performed by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for performing instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0105] To interact with a user, aspects of the invention can be implemented in a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touch screen for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request previously received from the web browser.

[0106] Unless expressly stated to the contrary, the phrase "at least one of A, B, or C" is intended to refer to any combination or subset of A, B, C, such as (1) at least one A alone, (2) at least one B alone, (3) at least one C alone, (4) at least one A with at least one B, (5) at least one A with at least one C, (6) at least one B with at least one C, (7) at least one A with at least one B and at least one C, etc. Further, unless expressly stated to the contrary, the phrase "at least one of A, B, and C" is intended to refer to any combination or subset of A, B, C, such as (1) at least one A alone, (2) at least one B alone, (3) at least one C alone, (4) at least one A with at least one B, (5) at least one A with at least one C, (6) at least one B with at least one C, (7) at least one A with at least one B and at least one C. Further, unless expressly stated to the contrary, "A or B" is intended to refer to any combination of A and B, such as (1) A alone, (2) B alone, and (3) A and B.

[0107] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (400) executed by data processing hardware (510) to cause the data processing hardware (510) to perform an operation, the operation comprising: receiving as input a sequence of acoustic frames (110) corresponding to an utterance (106); generating, at each of a plurality of output steps, a high-level feature representation (212) of a corresponding audio frame (110) of the sequence of audio frames (110) using an audio encoder (210); generating a plurality of speech recognition hypotheses based on the high-level feature representation (212) at each of a plurality of the output steps using a decoder (215), each of the speech recognition hypotheses corresponding to a candidate transcription of the utterance (106), each of the speech recognition hypotheses having an associated first likelihood score; generating, for each of the plurality of speech recognition hypotheses using an external language model (230), a second likelihood score associated with the speech recognition hypothesis; determining, for each speech recognition hypothesis of the plurality of speech recognition hypotheses, a set of fusion weights based on the high-level feature representation and the speech recognition hypothesis using a learnable fusion module; generating, for each of the plurality of speech recognition hypotheses, a third likelihood score based on the first likelihood score, the second likelihood score, and the set of fusion weights using the learnable fusion module (240); Equipped with the audio encoder (210) is trained using minimum additive error rate training in the presence of the external language model (230) by adjusting likelihood scores according to a minimum additive error rate among the top-k speech recognition hypotheses output from the decoder (215); A computer-implemented method (400).

2. the operations further comprise selecting the speech recognition hypothesis from the plurality of speech recognition hypotheses as a speech recognition result (120) for the sequence of acoustic frames (110) based on the third likelihood scores of the plurality of speech recognition hypotheses. The computer-implemented method (400) of claim 1.

3. the first likelihood scores comprise a conditional likelihood score and an internal language model likelihood score. The computer-implemented method (400) of claim 1 or claim 2.

4. the first likelihood score for a particular speech recognition hypothesis comprises a plurality of first language model scores, each of the first language model scores corresponding to a particular token of the particular speech recognition hypothesis; the second likelihood score for the particular speech recognition hypothesis comprises a plurality of second language model scores, each second language model score of the plurality of second language model scores corresponding to a particular token of the particular speech recognition hypothesis; the fusion weight set for the particular speech recognition hypothesis comprises a plurality of first fusion weights and a plurality of second fusion weights, each of the first fusion weights among the plurality of first fusion weights corresponding to a particular token of the particular speech recognition hypothesis, and each of the second fusion weights among the plurality of second fusion weights corresponding to a particular token of the particular speech recognition hypothesis; Utilizing the learnable fusion module (240) to generate the third likelihood score for the particular speech recognition hypothesis comprises: determining an internal language model score based on a plurality of the first language model scores and a plurality of the first fusion weights; determining an external language model score based on the second language model scores and the second fusion weights; combining the internal language model scores and the external language model scores; Equipped with A computer implemented method (400) according to any one of claims 1 to 3.

5. The audio encoder (210), the decoder (215), and the learnable fusion module (240) are jointly trained. A computer implemented method (400) according to any one of claims 1 to 4.

6. the external language model (230) is trained on text-only data; A computer implemented method (400) according to any one of claims 1 to 5.

7. the external language model (230) comprises a neural network language model comprising a stack of conformer or transformer layers; A computer implemented method (400) according to any one of the preceding claims.

8. the external language model (230) comprises a neural network language model comprising a stack of long short-term memory (LSTM) layers; A computer implemented method (400) according to any one of the preceding claims.

9. The audio encoder (210): a causal encoder with an initial stack of multi-head attention layers; a non-causal encoder comprising a final stack of multi-head attention layers superimposed on the initial stack of multi-head attention layers; Equipped with A computer implemented method (400) according to any one of the preceding claims.

10. the learnable fusion module (240) is configured to compute per-token fusion weights for the internal language model scores of the decoder (215) and the external language model scores of the external language model (230); A computer implemented method (400) according to any one of the preceding claims.

11. the minimum additive error rate comprises a minimum word error rate. A computer implemented method (400) according to any one of the preceding claims.

12. the external language model (230) is trained on text-only data; the audio encoder (210) and the decoder (215) are trained using a hybrid autoregressive transducer (HAT) factorization to facilitate integration of the external language model (230) trained on the text-only data; A computer implemented method (400) according to any one of claims 1 to 5 and 7 to 11.

13. An automatic speech recognition (ASR) system (118), comprising: An audio encoder (210), comprising: receiving as input a sequence of acoustic frames (110) corresponding to an utterance (106); generating, at each of a plurality of output steps, a high-level feature representation (212) of a corresponding acoustic frame (110) of the sequence of acoustic frames (110); The audio encoder (210) configured to: A decoder (215), comprising: receiving as input the high-level feature representation (212) generated by the audio encoder (210) at each of a plurality of the output steps; generating a plurality of speech recognition hypotheses based on the received high-level feature representation (212) at each of a plurality of the output steps, each of the speech recognition hypotheses corresponding to a candidate transcription of the utterance (106), each of the speech recognition hypotheses having an associated first likelihood score; The decoder (215) is configured to: An external language model (230), comprising: receiving as input a plurality of said speech recognition hypotheses generated by said decoder (215); for each speech recognition hypothesis of the plurality of speech recognition hypotheses, generating a second likelihood score associated with the speech recognition hypothesis; the external language model (230) configured to a trainable fusion module (240), for each speech recognition hypothesis of the plurality of speech recognition hypotheses: receiving as input the high-level feature representation (212) generated by the audio encoder (210) at each of a plurality of the output steps; determining a set of fusion weights based on the high-level feature representation (212) and the speech recognition hypotheses; generating a third likelihood score based on the first likelihood score, the second likelihood score, and the set of fusion weights; The learnable fusion module (240) configured to: Equipped with the audio encoder (210) is trained using minimum additive error rate training in the presence of the external language model (230) by adjusting likelihood scores according to a minimum additive error rate among the top-k speech recognition hypotheses output from the decoder (215); An automatic speech recognition (ASR) system (118).

14. and selecting a speech recognition hypothesis from among the plurality of speech recognition hypotheses as a speech recognition result for the sequence of acoustic frames (110) based on the third likelihood scores of the plurality of speech recognition hypotheses.

14. The automatic speech recognition (ASR) system (118) of claim 13.

15. the first likelihood scores comprise a conditional likelihood score and an internal language model likelihood score. An automatic speech recognition (ASR) system (118) according to claim 13 or claim 14.

16. the first likelihood score for a particular speech recognition hypothesis comprises a plurality of first language model scores, each of the first language model scores corresponding to a particular token of the particular speech recognition hypothesis; the second likelihood score for the particular speech recognition hypothesis comprises a plurality of second language model scores, each second language model score of the plurality of second language model scores corresponding to a particular token of the particular speech recognition hypothesis; the fusion weight set for the particular speech recognition hypothesis comprises a plurality of first fusion weights and a plurality of second fusion weights, each of the first fusion weights among the plurality of first fusion weights corresponding to a particular token of the particular speech recognition hypothesis, and each of the second fusion weights among the plurality of second fusion weights corresponding to a particular token of the particular speech recognition hypothesis; The learnable fusion module (240) determining an internal language model score based on a plurality of the first language model scores and a plurality of the first fusion weights; determining an external language model score based on the second language model scores and the second fusion weights; combining the internal language model scores and the external language model scores; generating the third likelihood score for the particular speech recognition hypothesis by An automatic speech recognition (ASR) system (118) according to any one of claims 13 to 15.

17. The audio encoder (210), the decoder (215), and the learnable fusion module (240) are jointly trained. An automatic speech recognition (ASR) system (118) according to any one of claims 13 to 16.

18. the external language model (230) is trained on text-only data; An automatic speech recognition (ASR) system (118) according to any one of claims 13 to 17.

19. the external language model (230) comprises a neural network language model comprising a stack of conformer or transformer layers; An automatic speech recognition (ASR) system (118) according to any one of claims 13 to 18.

20. the external language model (230) comprises a neural network language model comprising a stack of long short-term memory (LSTM) layers; An automatic speech recognition (ASR) system (118) according to any one of claims 13 to 18.

21. The audio encoder (210): a causal encoder with an initial stack of multi-head attention layers; a non-causal encoder comprising a final stack of multi-head attention layers superimposed on the initial stack of multi-head attention layers; Equipped with An automatic speech recognition (ASR) system (118) according to any one of claims 13 to 20.

22. the learnable fusion module (240) is configured to compute per-token fusion weights for the internal language model scores of the decoder (215) and the external language model scores of the external language model (230); An automatic speech recognition (ASR) system (118) according to any one of claims 13 to 21.

23. the minimum additive error rate comprises a minimum word error rate. An automatic speech recognition (ASR) system (118) according to any one of claims 13 to 22.

24. the external language model (230) is trained on text-only data; the audio encoder (210) and the decoder (215) are trained using a hybrid autoregressive transducer (HAT) factorization to facilitate integration of the external language model (230) trained on the text-only data; An automatic speech recognition (ASR) system (118) according to any one of claims 13 to 17 and 19 to 23.

Images