Rare word recognition using LM aware MWER training
By introducing learnable fusion modules and minimum additive error rate training in the speech recognition model, the fusion weight is dynamically determined, which solves the problem of insufficient accuracy of rare word recognition for normal-scale types, and improves the accuracy and computing efficiency of speech recognition.
Patent Information
- Application Number
- CN202380035242.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-22
- Filing Date
- 2023-03-21
- Publication Date
- 2024-11-29
AI Technical Summary
The conventional end-to-end speech recognition model is difficult to effectively fuse external language model scores during training, resulting in insufficient accuracy of rare word recognition. The traditional method's fusion weight determination is complex and has high computational cost. The fusion weight is usually kept fixed and cannot be adjusted dynamically. .
The higher-order feature representation of the speech recognition assumption is dynamically determined by the Learnable Fusion Module (LFM) based on the acoustic frame generation, and the likelihood score is adjusted through minimum additive error rate training (such as minimum word error rate). The E2E ASR model scores were fused with the LM scores in a manner similar to the inference period.
The speech recognition accuracy of the speech recognition model for rare words is improved, the fusion weight is dynamically adjusted to adapt to different speech inputs, reducing the computational complexity and cost, and achieving the fractional fusion effect during training.
Smart Images

Figure CN119054016A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to speech recognition models for improving rare word recognition. Background Art
[0002] Automatic speech recognition (ASR) is an important technology that is increasingly used in mobile devices and other devices. Generally speaking, ASR systems can provide accurate transcriptions of what a person says with low latency (eg, only a short delay between when the person speaks and when the transcription appears). Summary of the invention
[0003] One aspect of the present disclosure provides a computer-implemented method for training a speech recognition model to improve rare word recognition using language model-perceived minimum word error rate training. The computer-implemented method, when executed on data processing hardware, causes the data processing hardware to perform operations, the operations comprising: receiving as input a sequence of acoustic frames corresponding to an utterance; and generating, using an audio encoder, a higher-order feature representation of a corresponding acoustic frame in the sequence of acoustic frames at each of a plurality of output steps. The operations also include: generating, using a decoder based on the higher-order feature representation at each of the plurality of output steps, a plurality of speech recognition hypotheses, each speech recognition hypothesis corresponding to a candidate transcription of the utterance and having an associated first likelihood score; and generating, for each of the plurality of speech recognition hypotheses, using an external language model, a second likelihood score associated with the speech recognition hypothesis. The operations also include: using a learnable fusion module, for each of the multiple speech recognition hypotheses, determining a fusion weight set based on the higher-order feature representation and the speech recognition hypothesis; and using the learnable fusion module, for each of the multiple speech recognition hypotheses, generating a third likelihood score based on the first likelihood score, the second likelihood score, and the fusion weight set. Here, in the presence of an external language model, the audio encoder is trained using minimum additive error rate training by adjusting the likelihood score according to the minimum additive error rate among the first k speech recognition hypotheses output from the decoder.
[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operations also include selecting a speech recognition hypothesis from the multiple speech recognition hypotheses as the speech recognition result of the acoustic frame sequence based on a third likelihood score of the multiple speech recognition hypotheses. In some examples, the first likelihood score includes a conditional likelihood score and an internal language model likelihood score.
[0005] In some examples, a first likelihood score for a specific speech recognition hypothesis includes a first plurality of language model scores, each of which corresponds to a specific word element of the specific speech recognition hypothesis; a second likelihood score for the specific speech recognition hypothesis includes a second plurality of language model scores, each of which corresponds to a specific word element of the specific speech recognition hypothesis; a fusion weight set for the specific speech recognition hypothesis includes a first plurality of fusion weights and a second plurality of fusion weights, each of which corresponds to a specific word element of the specific speech recognition hypothesis, and each of which corresponds to a specific word element of the specific speech recognition hypothesis. Here, generating a third likelihood score for the specific speech recognition hypothesis using the learnable fusion module may include: determining an internal language model score based on the first plurality of language model scores and the first plurality of fusion weights; determining an external language model score based on the second plurality of language model scores and the second plurality of fusion weights; and combining the internal language model score and the external language model score.
[0006] In some implementations, the audio encoder, the decoder, and the learnable fusion module are jointly trained. In some examples, the external language model is trained on plain text data. In some examples, the external language model includes a neural network language model that includes a stack of conformer layers or transformer layers. In other examples, the external language model includes a neural network language model that includes a stack of long short-term memory (LSTM) layers.
[0007] In some examples, the audio encoder includes: a causal encoder comprising an initial multi-head attention layer stack; and a non-causal encoder comprising a final multi-head attention layer stack overlaid on the initial multi-head attention layer stack. In some implementations, the learnable fusion module is configured to calculate a per-word fusion weight of the internal language model score of the decoder and the external language model score of the external language model. In some implementations, the minimum additive error rate includes a minimum word error rate. In some examples, the external language model is trained for plain text data; and the audio encoder and the decoder are trained using hybrid autoregressive transducer (HAT) decomposition to facilitate the integration of the external language model trained for the plain text data.
[0008] Another aspect of the present disclosure provides an automatic speech recognition (ASR) system, which includes an audio encoder, a decoder, an external language model, and a learnable fusion module. The audio encoder is configured to receive a sequence of acoustic frames corresponding to an utterance as input; and generate a higher-order feature representation of a corresponding acoustic frame in the sequence of acoustic frames at each of a plurality of output steps. The decoder is configured to receive a higher-order feature representation generated by the audio encoder at each of the plurality of output steps as input, and generate a plurality of speech recognition hypotheses based on the higher-order feature representation received at each of the plurality of output steps, each speech recognition hypothesis corresponding to a candidate transcription of the utterance and having an associated first likelihood score. The external language model is configured to receive a plurality of speech recognition hypotheses generated by the decoder as input, and generate a second likelihood score associated with the speech recognition hypothesis for each of the plurality of speech recognition hypotheses. The learnable fusion module is configured to, for each of the multiple speech recognition hypotheses: receive as input a higher order feature representation generated by the audio encoder at each of the multiple output steps; determine a fusion weight set based on the higher order feature representation and the speech recognition hypothesis; and generate a third likelihood score based on the first likelihood score, the second likelihood score, and the fusion weight set. Here, in the presence of an external language model, the audio encoder is trained using minimum additive error rate training by adjusting the likelihood score according to the minimum additive error rate among the top k speech recognition hypotheses output from the decoder.
[0009] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operations also include selecting a speech recognition hypothesis from the multiple speech recognition hypotheses as the speech recognition result of the acoustic frame sequence based on a third likelihood score of the multiple speech recognition hypotheses. In some examples, the first likelihood score includes a conditional likelihood score and an internal language model likelihood score.
[0010] In some examples, a first likelihood score for a specific speech recognition hypothesis includes a first plurality of language model scores, each of which corresponds to a specific word element of the specific speech recognition hypothesis; a second likelihood score for the specific speech recognition hypothesis includes a second plurality of language model scores, each of which corresponds to a specific word element of the specific speech recognition hypothesis; a fusion weight set for the specific speech recognition hypothesis includes a first plurality of fusion weights and a second plurality of fusion weights, each of which corresponds to a specific word element of the specific speech recognition hypothesis, and each of which corresponds to a specific word element of the specific speech recognition hypothesis. Here, generating a third likelihood score for the specific speech recognition hypothesis using the learnable fusion module may include: determining an internal language model score based on the first plurality of language model scores and the first plurality of fusion weights; determining an external language model score based on the second plurality of language model scores and the second plurality of fusion weights; and combining the internal language model score and the external language model score.
[0011] In some implementations, the audio encoder, the decoder, and the learnable fusion module are jointly trained. In some examples, the external language model is trained on plain text data. In some examples, the external language model includes a neural network language model that includes a stack of conformer layers or transformer layers. In other examples, the external language model includes a neural network language model that includes a stack of long short-term memory (LSTM) layers.
[0012] In some examples, the audio encoder includes: a causal encoder comprising an initial multi-head attention layer stack; and a non-causal encoder comprising a final multi-head attention layer stack overlaid on the initial multi-head attention layer stack. In some implementations, the learnable fusion module is configured to calculate a per-word fusion weight of the internal language model score of the decoder and the external language model score of the external language model. In some implementations, the minimum additive error rate includes a minimum word error rate. In some examples, the external language model is trained for plain text data; and the audio encoder and the decoder are trained using hybrid autoregressive transducer (HAT) decomposition to facilitate the integration of the external language model trained for the plain text data.
[0013] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 is a schematic diagram of an example speech environment in which speech is transcribed using an automatic speech recognition (ASR) system.
[0015] Figure 2 yes Figure 1 Schematic diagram of an example hybrid autoregressive transducer (HAT) model architecture of an ASR system.
[0016] Figure 3 is a schematic diagram of an example tied and reduced prediction network.
[0017] Figure 4 is a flow diagram of an example operational arrangement of a computer-implemented method for improving rare word recognition.
[0018] Figure 5 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.
[0019] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION
[0020] Automatic speech recognition (ASR) is an important technology that is increasingly used in mobile devices and other devices. In general, ASR systems can provide accurate transcription of what a person says with low latency (e.g., only a short delay between when the person speaks and when the transcription occurs). However, conventional end-to-end (E2E) ASR models are trained for audio-text pairs, which represent only a small portion of the available plain text data. Therefore, conventional E2E ASR models may not accurately recognize rare words. In contrast, language models (LMs) that can be trained using plain text data can more accurately recognize rare words. LMs have been combined with E2E ASR models to improve rare word recognition accuracy. For example, E2E ASR model scores and LM scores can be fused using hybrid autoregressive transducer (HAT) decomposition during reasoning.
[0021] The implementation disclosed herein is intended to incorporate the LM score into the E2E ASR loss term during training. However, it may be difficult to incorporate the LM score into the training of the ASR model because the E2E ASR model is traditionally trained for true value labels, which is different from the way the LM is incorporated during reasoning. Therefore, the implementation disclosed herein performs LM-aware minimum additive error rate training (e.g., LM-aware minimum word error rate (MWER)) by operating the E2E ASR model in a beam search mode during training, which allows the E2E ASR model score to be fused with the LM score during training in a manner similar to the way the E2E ASR model score is fused with the LM score during reasoning. Here, the E2E ASR model score and the LM score can be fused using HAT decomposition. However, conventional methods (e.g., parameter sweeps) for determining and tuning fusion weights are complex and computationally expensive. In addition, after training, the fusion weights are traditionally kept fixed after training. In contrast, the implementation disclosed herein configures and trains a learnable fusion module (LFM) to determine the per-word fusion weights of the speech recognition hypothesis in a data-dependent manner. Specifically, the LFM determines the fusion weights for each word-memory of the speech recognition hypothesis of the utterance based on higher-order feature representations (e.g., acoustic encoding) generated for the acoustic frames of the utterance. That is, the fusion weights are dynamically determined during inference. In the disclosed implementation, the LFM is trained using LM-perceptual minimum additive error rate training (e.g., LM-perceptual MWER training).
[0022] Figure 1 1 is a schematic diagram of an example of a speech environment 100 and system 101. In the speech environment 100, the way in which a user 104 interacts with a computing device (such as a user device 10) can be performed through voice input. The user device 10 (also generally referred to as the device 10) is configured to capture sound (e.g., streaming audio data) from one or more users 104 within the speech environment 100. Here, the streaming audio data can refer to the spoken words 106 of the user 104, which serve as audible queries, commands to the device 10, or audible communications captured by the device 10. The speech-enabled system of the device 10 can process the query or command by answering the query and / or causing one or more downstream applications to execute / fulfill the command.
[0023] The user device 10 may correspond to any computing device associated with the user 104 and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablet computers, laptop computers, etc.), computers, wearable devices (e.g., smart watches), smart home appliances, Internet of Things (IoT) devices, in-vehicle infotainment systems, smart displays, smart speakers, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 in communication with the data processing hardware 12. The memory hardware 114 stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 also includes an audio system 16 having: an audio capture device (e.g., a microphone) 16, 16a, which is used to capture the spoken speech 106 in the speech environment 100 and convert it into an electrical signal; and a speech output device (e.g., a speaker) 16, 16b, which is used to transmit an audible audio signal (e.g., as output audio data from the device 10). Although in the illustrated example, the user device 10 implements a single audio capture device 16 a, the user device 10 may implement an array of audio capture devices 16 a without departing from the scope of the present disclosure, wherein one or more capture devices 16 a in the array may not physically reside on the user device 10 but may communicate with the audio system 16 .
[0024] In the speech environment 100, an ASR system 118 implementing an end-to-end (E2E) automatic speech recognition (ASR) model 200 and an external language model 230 resides on a user device 10 of a user 104 and / or on a remote computing device 70 (e.g., one or more remote servers of a distributed system executed in a cloud computing environment) that communicates with the user device 10 via a network 40. In some examples, the ASR model 200 includes a hybrid autoregressive transducer (HAT) model. The user device 10 and / or the remote computing device 70 also includes an audio subsystem 108 that is configured to receive an utterance 106 spoken by the user 104 and captured by the audio capture device 16a, and convert the utterance 106 into a corresponding digital format associated with an input acoustic frame 110 so that it can be processed by the ASR system 118. In the example shown, the user speaks a corresponding utterance 106, and the audio subsystem 108 converts the utterance 106 into corresponding audio data (e.g., acoustic frame) 110 for input to the ASR system 118. Thereafter, the ASR model 200 receives as input an acoustic frame 110 corresponding to the utterance 106 and generates / predicts as output a corresponding transcription 120 (e.g., a recognition result / hypothesis) of the utterance 106. In the example shown, the ASR model 200 may perform streaming speech recognition to produce initial speech recognition results 120, 120a, and the external language model 230 may update the initial speech recognition results 120a (i.e., re-score them) to produce final speech recognition results 120, 120b. The server 70 includes data processing hardware 72 and memory hardware 74 in communication with the data processing hardware 72. The memory hardware 74 stores instructions that, when executed by the data processing hardware 72, cause the data processing hardware 72 to perform one or more operations, such as those disclosed herein.
[0025] The user device 10 and / or the remote computing device 70 also executes a user interface generator 107 that is configured to present a representation of a transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 can display the initial speech recognition results 120a in a streaming manner during time 1, and then display the final speech recognition results 120b during time 2. In some configurations, the transcription 120 output from the ASR system 118 is processed by a natural language processing / understanding (NLP / NLU) module, for example, executing on the user device 10 or the remote computing device 70, to execute the user command / query specified by the utterance 106. In addition or alternatively, a text-to-speech system (not shown) (e.g., executed on any combination of the user device 10 or the remote computing device 70) can convert the transcription into synthesized speech for audible output by the user device 10 and / or another device.
[0026] In the example shown, the user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) of the user device 10 using the ASR system 118. For example, Figure 1 A user 104 is depicted communicating with a digital assistant application 50 and the digital assistant application 50 displays a digital assistant interface 18 on a screen of a user device 10 to depict a conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50 "What time is the concert tonight?" This question from the user 104 is a spoken utterance 106 that is captured by an audio capture device 16a and processed by an audio system 16 of the user device 10. In this example, the audio system 16 receives the spoken utterance 106 and converts it into acoustic frames 110 for input to an ASR system 118.
[0027] Continuing with the example, the ASR model 200 encodes the acoustic frames 110 corresponding to the utterance 106 as the user 104 speaks, and then decodes the encoded acoustic frames 110 into initial speech recognition results 120a. During time 1, the user interface generator 107 presents a representation of the initial speech recognition results 120a of the utterance 106 to the user 104 of the user device 10 in a streaming manner via the digital assistant interface 18, so that words, word fragments, and / or individual characters appear on the screen immediately as they are spoken. In some examples, the first look ahead audio context is equal to zero.
[0028] During time 2, the user interface generator 107 presents a representation of the final speech recognition result 120b of the utterance 106 rescored by the external language model 230 to the user 104 of the user device 10 via the digital assistant interface 18. In some implementations, the user interface generator 107 replaces the representation of the initial speech recognition result 120a presented at time 1 with the representation of the final speech recognition result 120b presented at time 2. Here, time 1 and time 2 may include timestamps corresponding to the times at which the user interface generator 107 presents the corresponding speech recognition results 120. In this example, the timestamp of time 1 indicates that the user interface generator 107 presents the initial speech recognition result 120a at an earlier time than the final speech recognition result 120b. For example, because it is assumed that the final speech recognition result 120b is more accurate than the initial speech recognition result 120a, the user interface generator 107 will eventually display the final speech recognition result 120b as the transcription 120 to repair any terms that may have been misrecognized in the initial speech recognition result 120a. In this example, the streaming initial speech recognition result 120a output by the ASR model 200 is displayed on the screen of the user device 10 at time 1, is associated with low latency, and provides the user 104 with a response that its query is being processed, while the final speech recognition result 120b output by the external language model 230 and displayed on the screen at time 2 utilizes additional speech recognition models and / or language models to improve the quality of speech recognition (in terms of accuracy), but increases latency. However, because the initial speech recognition result 120a is displayed while the user is speaking the utterance 106, the higher latency associated with generating and ultimately displaying the final recognition result is not noticeable to the user 104.
[0029] exist Figure 1 In the example shown in , the digital assistant application 50 can use NLP / NLU to respond to questions raised by the user 104. NLP / NLU generally refers to the process of interpreting written language (e.g., initial speech recognition results 120a and / or final speech recognition results 120b) and determining whether the written language prompts any action. In this example, the digital assistant application 50 uses NLP / NLU to recognize that the question from the user 104 involves the user's schedule, and more specifically, the concert on the user's schedule. By using NLP / NLU to recognize these details, the automatic assistant returns a response 19 to the user's query, wherein the response 19 states "Venue doors open at 7:30PM and concert starts at 9pm (venue doors open at 7:30PM, and the concert starts at 9pm)". In some configurations, NLP / NLU appears on a remote server 70 that communicates with the data processing hardware 12 of the user device 10.
[0030] Figure 2 yes Figure 1 Schematic diagram of an example architecture of an ASR model 200. The example ASR model 200 may include a HAT model based on a variation of a recurrent neural network-transducer (RNN-T) model. The ASR model 200 provides a small computational footprint and utilizes less memory requirements than conventional ASR architectures, making the ASR model 200 suitable for performing speech recognition entirely on the user device 10 (e.g., without the need to communicate with a remote server).
[0031] As shown, the ASR model 200 is integrated with an external language model (LM) 230 and includes an audio encoder network 210, a prediction network 300, a joint network 220, and a final softmax layer 250. The prediction network 300 and the joint network 220 can jointly provide a decoder 215. In some examples, the external LM 230 is trained separately from the audio encoder network 210 and the decoder 215. The audio encoder network 210 is roughly similar to the acoustic model (AM) in a traditional ASR system, and can include a recurrent network of stacked long short-term memory (LSTM) layers or a stack of multi-head attention layers. For example, the audio encoder network 210 reads a d-dimensional feature vector sequence (e.g., acoustic frame 110) X = (x1, x2, ..., x T ),in And a higher-order feature representation 212 is generated at each time step. The higher-order feature representation 212 is represented as In some examples, the audio encoder network 210 includes a multi-headed attention encoder having twelve layers and an attention dimension of 512. For example, the encoder network 210 can include a stack of multi-headed attention layers (such as conformer layers or transformer layers). Another example audio encoder network 210 includes a causal encoder having an initial stack of conformer layers (or other types of multi-headed attention layers), and a non-causal encoder having a final stack of conformer layers (or other types of multi-headed attention layers) overlaid on the initial stack of conformer layers.
[0032] The prediction network 300 may be an LSTM network that, like an LM, converts the sequence (y0, ..., y ui-1 ) is processed into dense or hidden representation 350. Combined with the following Figure 3 Describe in more detail 350 includes a single embedding vector. Notably, the sequence of non-blank symbols 252 received at the prediction network 300 captures the linguistic correlations between the non-blank symbols 252 currently predicted during the previous time step to assist the joint network 220 in predicting the probability of the next output symbol or blank symbol during the current time step. As described in more detail below, in order to facilitate a technique for reducing the size of the prediction network 300 without sacrificing the accuracy / performance of the ASR model 200, the prediction network 300 may receive a limited historical sequence y of non-blank symbols 252 ui-n ,...,y ui-1 , the finite history sequence is limited to the first N non-blank symbols 252 output by the final softmax layer 250. In some examples, the prediction network 300 calculates the LM likelihood score based on the first two non-blank tokens 252. The prediction network 300 can have two 2,048-dimensional LSTM layers, each followed by a 740-dimensional projection layer and an embedding layer with 128 units.
[0033] The joint network 220 represents the higher-order features produced by the audio encoder network 210. 212 and the representation produced by the prediction network 300 350 (i.e., a single embedding vector 350). The joint network 220 is based on the acoustic frame 110Predict the conditional distribution of the next output symbol 222. In other words, the joint network 220 generates a probability distribution 222 of conditional likelihood scores for possible speech recognition hypotheses at each time step. Here, a "possible speech recognition hypothesis" corresponds to a set of output labels, each output label representing a symbol / character in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, for example, there is a label for each of the 26 letters in the English alphabet, and a label specifies a space. Therefore, the joint network 220 can output a set of values indicating the likelihood of each of the predetermined set of output labels occurring. The set of values may be a vector and may indicate a probability distribution about the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation and other symbols), but the set of output labels is not limited to this. For example, in addition to or in lieu of graphemes, the set of output labels may include word fragments and / or entire words. The output distribution of the joint network 220 may include a posterior probability value for each of the different output labels. Therefore, if there are 100 different output labels representing different graphemes or other symbols, the output of the joint network 220 is 222 may include 100 different probability values, one for each output label. The probability distribution may then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, word fragments, and / or words) in a beam search process (e.g., by a softmax layer 250) to determine the transcription 120.
[0034] The joint network 220 also searches for the top K speech recognition hypotheses {Y1, ..., Y K Each of the predicted likelihood scores P ILM (Y k )224. Here, the joint network 200 does not use higher-order feature representation 212 or use an empty higher-order feature representation to predict the likelihood score P ILM (Y k )224. Therefore, the likelihood score P ILM (Y k ) 224 represent the internal LM scores of the decoder 215, which are based on a limited set of audio-text training data used to train the audio encoder network 210 and the decoder network 215, and thus may be biased. In some implementations, the joint network 220 has 740 hidden units and uses a beam size of 8 to perform beam search.
[0035] The outer LM 230 performs a search on the top K speech recognition hypotheses {Y1, . . . , Y K Each of the predicted likelihood scores P ELM (Y k ) 232. Here, the external LM 320 is trained based on a large corpus of plain text data and is trained separately from the audio encoder network 210 and the decoder network 215. Because the external LM 230 is trained based on a large corpus of plain text data, it can correct the internal LM likelihood score P ILM (Y k ) 224. An example external LM 230 includes a neural network LM having a stack of LSTM layers (e.g., four layers) with 2048 units. Another example external LM 230 includes a neural network LM having a conformer with twelve layers and an attention dimension of 384. In yet another example, the external LM 230 includes a neural network LM having a stack of transformer layers.
[0036] The implementation herein incorporates a learnable fusion module (LFM) LFM 240 that performs HAT decomposition to fuse or combine the first likelihood scores (i.e., including likelihood scores 224 and 224) determined by the joint network and the second likelihood scores 232 determined by the external LM 230 to generate a first K speech recognition hypothesis {Y1, ..., Y K Each of the following determines each hypothesis {Y1, ..., Y K The third likelihood score P(Y k |X)242. Likelihood score P(Y k |X)242 can be expressed as:
[0037] logP(U k |X):=log P E2E (Y k |k)-λ·logP ILM (Y k )+γ·logP ELM (Y k )+C(1)
[0038] where λ and γ are the fractions used to fuse the internal LM ILM (Y k )224 and external LM score P ELM (Y k )232 fusion weight vector, and C can be selected so that Normalization constant. Let λ = [λ1,…,λ L ] and logP ILM (Y k )=[s1,…,s L ], where L is the speech recognition hypothesis Y k The total contribution score of the internal LM can be expressed as
[0039]
[0040] Similarly, let γ = [γ1, ..., γ L ] and logP ELM (Y k )=[p1,…,p L ], the total contribution fraction of the external LM 230 can be expressed as
[0041]
[0042] The first plurality of fusion weights μ=[μ1,…,μ L Each fusion weight in ] is consistent with the hypothesis Y kSimilarly, the second plurality of fusion weights λ=[λ1,…,λ L Each fusion weight in ] is consistent with the hypothesis Y k In other words, multiple fusion weights μ = [μ1, ..., μ L ] and λ=[λ1,…,λ L ] is the per-word fusion weight. This is in stark contrast to the conventional HAT decomposition, in which the same scalar fusion weight is applied to hypothesis Y k Similarly, multiple likelihood scores P ILM (Y k )224 and P ELM (Y k )232 each likelihood score and hypothesis Y k Specific word correspondence.
[0043] The audio encoder network 210 and the decoder network 215 may be trained using an additive error rate based loss, such as a MWER based loss, to incorporate a previously and separately trained external LM 230. An example MWER based loss that may be used to train the audio encoder network 210 and the decoder network 215 is expressed as
[0044]
[0045] Among them, NWE(Y k , Y * ) indicates Y k With Y * In some implementations, the compound loss
[0046]
[0047] is minimized during training to improve learning stability. An example value for θ is 0.04. Using the first eight hypotheses Y k Compute the example MWER loss used for training.
[0048] LFM 240 includes a fusion weight determination model 244 that is configured and trained to determine the weights of each speech recognition hypothesis Y. k , based on higher-order feature representation 212 and the assumption that Y k The fusion weight vector μ = [μ1, ..., μ L ] and the fusion weight vector γ = [γ1, ..., γ L ]. It is worth noting that here, the fusion weight vector μ = [μ1,…,μ L] and γ=[γ1,…,γ L ] During inference, based on the data (i.e., based on the higher-order feature representation 212 and the hypothesis Y k ) is dynamically determined. That is, the fusion weight vector μ = [μ1, ..., μ L ] and γ=[γ1,…,γ L ] is not determined during training and then remains fixed during inference. In some examples, the fusion weight determination model 244 includes a transformer architecture similar to an attention-based decoder and assumes Y k The features of Y perform causal self-attention and cross-attention on higher-order feature representations 212 to determine the hypothesis Y k The fusion weight vector μ = [μ1, ..., μ L ] and γ=[γ1,…,γ L ]. The example fusion weight determination model 244 includes three transformer layers and an attention dimension 256 or 320. In some examples, after training the audio encoder network 210 and the decoder network 215 and while the coefficients of the audio encoder network 210 and the decoder network 215 remain fixed, the fusion weight determination model 244 is trained for a speech recognition task based on the MWER-based loss of equation (4) and / or equation (5). In other examples, the audio encoder network 210, the decoder network 215, and the fusion weight determination model 244 are jointly trained for a speech recognition task based on the MWER-based loss of equation (4) and / or equation (5).
[0049] The final softmax layer 250 receives the likelihood score P(Y k |X)242 and select the one with the highest likelihood score P(Y k |X)242 Speech Recognition Hypothesis Y k to generate the transcription 120. The ASR model 200 does not make conditional independence assumptions, but rather for each symbol y u The prediction of 252 depends not only on the acoustic frame 110, but also on the sequence y of the currently output labels 252 ui-n ,…,y ui-1 The ASR model 200 does assume that the output symbols 252 are independent of future acoustic frames 110, which allows the ASR model 200 to be employed in a streaming manner.
[0050] Figure 3 2 is a schematic diagram of an example prediction network 300 for use with the ASR model 200. The prediction network 300 receives a sequence of non-blank symbols 252a-ny. ui-n ,…,y ui-1As input, the non-blank symbol sequence is limited to the first N non-blank symbols 252a-n output by the final softmax layer 250. In some examples, N is equal to 2. In other examples, N is equal to 5, however, the present disclosure is non-limiting and N can be equal to any integer. The non-blank symbol sequence 252a-n indicates the initial speech recognition result 120a ( Figure 1 ). In some implementations, the prediction network 300 includes a multi-head attention mechanism 302 that shares a shared embedding matrix 304 across each head 302A-302H of the multi-head attention mechanism. In one example, the multi-head attention mechanism 302 includes four heads. However, the multi-head attention mechanism 302 can employ any number of heads. Notably, the multi-head attention mechanism significantly improves performance with minimal increase in model size. As described in more detail below, each head 302A-H includes its own row of position vectors 308, and rather than concatenating the outputs 318A-H of all heads to increase model size, the outputs 318A-H are averaged by a head averaging module 322.
[0051] Referring to the first head 302A of the multi-head attention mechanism 302, the head 302A uses the shared embedding matrix 304 to receive as input the non-blank symbol sequence 252a-ny at corresponding time steps in the plurality of time steps. ui-n ,…,y ui-1 Each non-whitespace symbol in generates a corresponding embedding 306, 306a-n (e.g., ). It is worth noting that since the shared embedding matrix 304 is shared across all heads of the multi-head attention mechanism 302, the other heads 302B-H all generate the same corresponding embedding 306 for each non-blank symbol. Head 302A also generates the corresponding position vector PV Aa-An 308, 308Aa-An (e.g., Assigned to non-whitespace character sequence 252a-ny ui-n , ..., y ui-1 Each corresponding non-blank symbol in . The corresponding position vector PV 308 assigned to each non-blank symbol indicates the position in the history of the non-blank symbol sequence (e.g., the first N non-blank symbols 252a-n output by the final softmax layer 250). For example, the first position vector PV Aa is assigned to the most recent position in history, and the last position vector PV An Assigned to the last position in the history of the first N non-whitespace symbols output by the final softmax layer 250. Notably, each of the embeddings 306 may include the same number of dimensions (ie, dimensional size) as each of the position vectors PV 308.
[0052] Although the shared embedding matrix 304 is a non-blank symbol sequence 252a-ny ui-n ,...,y ui-1 The corresponding embedding generated for each non-whitespace symbol in is the same across all heads 302A-H of the multi-head attention mechanism 302, but each head 302A-H defines a different set / row of position vectors 308. For example, the first head 302A defines a row of position vectors PV Aa-An 308Aa-An, the second head 302B defines different position vector rows PV Ba-Bn 308 Ba-Bn , ..., and the Hth header 302H defines another different position vector row PV Ha-Hn 308 Ha-Hn .
[0053] For each non-blank symbol in the received non-blank symbol sequence 252a-n, the first head 302A also weights the corresponding embedding 306 via the weight layer 310, and the weight is proportional to the similarity between the corresponding embedding and the corresponding position vector PV 308 assigned thereto. In some examples, the similarity may include a cosine similarity (e.g., a cosine distance). In the example shown, the weight layer 310 outputs a weighted embedding sequence 312, 312Aa-An, each weighted embedding associated with a corresponding embedding 306, and the weight is proportional to the corresponding position vector PV 308 assigned thereto. In other words, the weighted embedding 312 output by the weight layer 310 for each embedding 306 can correspond to the dot product between the embedding 306 and the corresponding position vector PV 308. The weighted embedding 312 can be interpreted as focusing on the embeddings in proportion to the degree of similarity with which they are positioned to be associated with their corresponding position vector PV 308. To increase computational speed, the prediction network 300 includes a non-recurrent layer, and therefore, the weighted embedding sequence 312Aa-An is not concatenated, but averaged by the weighted average module 316 to generate a weighted average 318A of the weighted embeddings 312Aa-An as the output of the first head 302A, represented as:
[0054]
[0055] In equation (1), h represents the index of head 302, n represents the position in the context, and e represents the embedding dimension. In addition, in equation (6), H, N, and d e Including the size of the corresponding dimension. Position vector PV 308 does not have to be trainable and may include random values. It is worth noting that even if weighted embedding 312 is averaged, position vector PV 308 can potentially preserve position history information, thereby alleviating the need to provide recurrent connections at each layer of prediction network 300.
[0056] The operations described above with respect to the first head 302A are similarly performed by each of the other heads 302B-H of the multi-head attention mechanism 302. Due to the different sets of positioning vectors PV 308 defined by each head 302, the weight layer 310 outputs a weighted embedding sequence 312Ba-Bn, 312Ha-Hn at each of the other heads 302B-H that is different from the weighted embedding sequence 312Aa-Aa at the first head 302A. Thereafter, the weighted average module 316 generates a corresponding weighted average 318B-H of the corresponding weighted embedding 312 of the non-blank symbol sequence as an output of each of the other corresponding heads 302B-H.
[0057] In the example shown, the prediction network 300 includes a head average module 322 that averages the weighted averages 318A-H output from the corresponding heads 302A-H. A projection layer 326 with SWISH may receive as input an output 324 corresponding to the average of the weighted averages 318A-H from the head average module 322 and generate as output a projection output 328. A final layer normalization 330 may normalize the projection output 328 to provide a single embedding vector at a corresponding time step in a plurality of time steps. 350. The prediction network 300 generates only a single embedding vector at each of the multiple time steps after the initial time step 350.
[0058] In some configurations, the prediction network 300 does not implement the multi-head attention mechanism 302, but only performs the operations described above with respect to the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-An is simply passed through the projection layer 326 and the layer normalization 330 to provide a single embedding vector 350.
[0059] In some implementations, in order to further reduce the size of the RNN-T decoder (i.e., the prediction network 300 and the joint network 220), parameter bundling between the prediction network 300 and the joint network 220 is applied. Specifically, for the vocabulary size |V| and the embedding dimension d e , the shared embedding matrix 304 at the prediction network is Meanwhile, at the joint network 220, the last hidden layer includes a dimension size d h , the feed-forward projection weights from the hidden layer to the output logits will be There are additional blank tokens in the vocabulary. Therefore, the feed-forward layer corresponding to the last layer of the joint network 220 includes a weight matrix [d h ,|V]|. By letting the prediction network 300 embed the dimension d e The size of the last hidden layer of the joint network 220 is dh Tied together, the feedforward projection weights of the joint network 220 and the shared embedding matrix 304 of the prediction network 300 can share their weights for all non-blank symbols via a simple transpose transformation. Since the two matrices share all their values, the RNN-T decoder only needs to store the values once in memory instead of storing two separate matrices. By increasing the embedding dimension d e The size of is set equal to the hidden layer dimension d h The size of the RNN-T decoder is reduced to equal the embedding dimension d e The number of parameters is the product of the vocabulary size |V|. This weight tying corresponds to the regularization technique.
[0060] Figure 4 4 is a flow chart of an exemplary operational arrangement of a computer-implemented method 400 for improving rare word recognition. At operation 402, the method 400 includes receiving a sequence of acoustic frames 110 corresponding to an utterance 106. At operation 404, the method 400 includes generating, using the audio encoder 210, at each of a plurality of output steps, a higher-order feature representation 212 of a corresponding acoustic frame 110 in the sequence of received acoustic frames 110.
[0061] The method 400 includes, at operation 406, using the decoder 215 to generate a plurality of speech recognition hypotheses {Y1, ..., Y K}, each speech recognition hypothesis Y k corresponds to a candidate transcription of the utterance 106 and has an associated first likelihood score that includes likelihood scores 222 , 224 .
[0062] At operation 408, method 400 includes using external language model 230 to identify a plurality of speech recognition hypotheses {Y1, ..., Y K Each speech recognition hypothesis Y in k , Generation and Speech Recognition Hypothesis Y k The associated second likelihood score P ELM (Y k )232.
[0063] The method 400 includes, at operation 410, using the LFM 240 to perform a multi-speech recognition analysis on a plurality of speech recognition hypotheses {Y1, ..., Y K Each speech recognition hypothesis Y in k , based on higher-order feature representation 212 and speech recognition hypothesis Y k To determine the fusion weight set λ = {λ1, ..., λ L} and γ={γ1,…,γ LAt operation 412, method 400 includes using LFM 240 to identify a plurality of speech recognition hypotheses {Y1, ..., Y K Each speech recognition hypothesis Y in k , based on the first likelihood scores 222, 224, the second likelihood score 232 and the fusion weight set λ = {λ1, ..., λ L} and γ={γ1,…,γ L} to generate a third likelihood score 242.
[0064] At operation 414, method 400 includes generating a speech recognition hypothesis based on a plurality of speech recognition hypotheses {Y1, ..., Y K}, selects a plurality of speech recognition hypotheses {Y1, ..., Y K Speech recognition hypothesis Y in k Speech recognition result 120 as a sequence of acoustic frames 110 .
[0065] Figure 5 5 is a schematic diagram of an example computing device 500 that can be used to implement the systems and methods described in this document. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit implementations of the inventions described and / or claimed in this document.
[0066] The computing device 500 includes a processor 510 (i.e., data processing hardware) that can be used to implement the data processing hardware 12 and / or 72, a memory 520 (i.e., memory hardware) that can be used to implement the memory hardware 14 and / or 74, a storage device 530 (i.e., memory hardware) that can be used to implement the memory hardware 14 and / or 74, a high-speed interface / controller 540 connected to the memory 520 and the high-speed expansion port 550, and a low-speed interface / controller 560 connected to the low-speed bus 570 and the storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 510 can process instructions for execution within the computing device 500, including instructions stored in the memory 520 or on the storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 580 coupled to the high-speed interface 540). In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories may be used as appropriate. In addition, multiple computing devices 500 may be connected, each of which provides a portion of the necessary operations (e.g., as a server group, a blade server group, or a multi-processor system).
[0067] Memory 520 stores information non-temporarily within computing device 500. Memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporary memory 520 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.
[0068] Storage device 530 is capable of providing mass storage for computing device 500. In some implementations, storage device 530 is a computer-readable medium. In various different implementations, storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.
[0069] The high-speed controller 540 manages bandwidth-intensive operations of the computing device 500, while the low-speed controller 560 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590 (which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet)) can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, through a network adapter.
[0070] As shown, computing device 500 may be implemented in a variety of different forms. For example, computing device may be implemented as a standard server 500a or multiple times in a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.
[0071] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be special purpose or general purpose and can be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.
[0072] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.
[0073] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0074] The processes and logic flows described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CDROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated in special purpose logic circuitry.
[0075] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.
[0076] Unless expressly stated to the contrary, the phrase “at least one of A, B, or C” is intended to refer to any combination or subset of A, B, and C, such as: (1) only at least one A; (2) only at least one B; (3) only at least one C; (4) at least one A and at least one B; (5) at least one A and at least one C; (6) at least one B and at least one C; and (7) at least one A and at least one B and at least one C. Furthermore, unless expressly stated to the contrary, the phrase “at least one of A, B, and C” is intended to refer to any combination or subset of A, B, and C, such as: (1) only at least one A; (2) only at least one B; (3) only at least one C; (4) at least one A and at least one B; (5) at least one A and at least one C; (6) at least one B and at least one C; and (7) at least one A and at least one B and at least one C. Furthermore, unless expressly stated to the contrary, “A or B” is intended to refer to any combination of A and B, such as: (1) only A; (2) only B; and (3) A and B.
[0077] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the appended claims.
Claims
1. A computer-implemented method (400) executed by data processing hardware (510), the computer-implemented method causing the data processing hardware (510) to perform an operation, characterized in that The operations include: Receiving as input a sequence of acoustic frames (110) corresponding to an utterance (106); generating, using an audio encoder (210), at each of a plurality of output steps, a higher-order feature representation (212) of a corresponding acoustic frame (110) in the sequence of acoustic frames (110); generating, using a decoder (215), a plurality of speech recognition hypotheses based on the higher-order feature representation (212) at each of the plurality of output steps, each speech recognition hypothesis corresponding to a candidate transcription of the utterance (106) and having an associated first likelihood score; generating, for each speech recognition hypothesis of the plurality of speech recognition hypotheses, a second likelihood score associated with the speech recognition hypothesis using an external language model (230); determining, using a learnable fusion module (240), for each speech recognition hypothesis in the plurality of speech recognition hypotheses, a set of fusion weights based on the higher-order feature representation (212) and the speech recognition hypothesis; and generating, using the learnable fusion module (240), for each of the plurality of speech recognition hypotheses, a third likelihood score based on the first likelihood score, the second likelihood score, and the set of fusion weights, Wherein in the presence of the external language model (230), the audio encoder (210) is trained using minimum additive error rate training by adjusting the likelihood score according to the minimum additive error rate among the top k speech recognition hypotheses output from the decoder (215).
2. The computer-implemented method (400) of claim 1, wherein: The operations also include selecting a speech recognition hypothesis from the plurality of speech recognition hypotheses as a speech recognition result (120) for the sequence of acoustic frames (110) based on the third likelihood scores of the plurality of speech recognition hypotheses.
3. The computer-implemented method (400) of claim 1 or claim 2, wherein: The first likelihood score includes a conditional likelihood score and an internal language model likelihood score.
4. The computer-implemented method (400) of any one of claims 1-3, characterized in that: The first likelihood score for a particular speech recognition hypothesis comprises a first plurality of language model scores, each language model score in the first plurality of language model scores corresponding to a particular word-gram of the particular speech recognition hypothesis; the second likelihood score for the particular speech recognition hypothesis comprising a second plurality of language model scores, each language model score in the second plurality of language model scores corresponding to a particular word-gram for the particular speech recognition hypothesis; The set of fusion weights for the specific speech recognition hypothesis includes a first plurality of fusion weights and a second plurality of fusion weights, each fusion weight in the first plurality of fusion weights corresponds to a specific word-gram of the specific speech recognition hypothesis, and each fusion weight in the second plurality of fusion weights corresponds to a specific word-gram of the specific speech recognition hypothesis; and Generating the third likelihood score for the particular speech recognition hypothesis using the learnable fusion module (240) includes: determining an internal language model score based on the first plurality of language model scores and the first plurality of fusion weights; determining an external language model score based on the second plurality of language model scores and the second plurality of fusion weights; and The internal language model score and the external language model score are combined.
5. The computer-implemented method (400) of any one of claims 1-4, characterized in that: The audio encoder (210), the decoder (215) and the learnable fusion module (240) are jointly trained.
6. The computer-implemented method (400) of any one of claims 1-5, wherein: The external language model (230) is trained on plain text data.
7. The computer-implemented method (400) of any one of claims 1-6, wherein: The external language model (230) comprises a neural network language model, wherein the neural network language model comprises a stack of conformer layers or transformer layers.
8. The computer-implemented method (400) of any one of claims 1-6, wherein: The external language model (230) comprises a neural network language model comprising a stack of long short-term memory (LSTM) layers.
9. The computer-implemented method (400) of any one of claims 1-8, wherein: The audio encoder (210) comprises: a causal encoder comprising an initial stack of multi-head attention layers; and A non-causal encoder comprising a final multi-head attention layer stack overlying the initial multi-head attention layer stack.
10. The computer-implemented method (400) of any one of claims 1-9, wherein: The learnable fusion module (240) is configured to calculate a per-word fusion weight of an internal language model score of the decoder (215) and an external language model score of the external language model (230).
11. The computer-implemented method (400) of any one of claims 1-10, wherein: The minimum additive error rate includes the minimum word error rate.
12. The computer-implemented method (400) of any one of claims 1 - 5 and 7 - 11, characterized in that: The external language model (230) is trained on plain text data; and The audio encoder (210) and the decoder (215) are trained using Hybrid Autoregressive Transducer (HAT) decomposition to facilitate integration of the external language model (230) trained on the plain text data.
13. An automatic speech recognition (ASR) system (118), characterized in that: include: An audio encoder (210), the audio encoder being configured to: Receiving as input a sequence of acoustic frames (110) corresponding to an utterance (106); as well as generating, at each of a plurality of output steps, a higher-order feature representation (212) of a corresponding acoustic frame (110) in the sequence of acoustic frames (110); A decoder (215), the decoder being configured to: receiving as input the higher-order feature representation (212) generated by the audio encoder (210) at each of the plurality of output steps; as well as generating a plurality of speech recognition hypotheses based on the higher-order feature representations (212) received at each of the plurality of output steps, each speech recognition hypothesis corresponding to a candidate transcription of the utterance (106) and having an associated first likelihood score; An external language model (230), the external language model being configured to: receiving as input the plurality of speech recognition hypotheses generated by the decoder (215); and generating, for each speech recognition hypothesis of the plurality of speech recognition hypotheses, a second likelihood score associated with the speech recognition hypothesis; as well as A learnable fusion module (240) is configured to, for each of the plurality of speech recognition hypotheses: receiving as input the higher-order feature representation (212) generated by the audio encoder (210) at each of the plurality of output steps; determining a set of fusion weights based on the higher-order feature representation (212) and the speech recognition hypothesis; and generating a third likelihood score based on the first likelihood score, the second likelihood score and the fusion weight set, Wherein in the presence of the external language model (230), the audio encoder (210) is trained using minimum additive error rate training by adjusting the likelihood score according to the minimum additive error rate among the top k speech recognition hypotheses output from the decoder (215).
14. The ASR system (118) of claim 13, wherein: The operation also includes selecting a speech recognition hypothesis from the plurality of speech recognition hypotheses as a speech recognition result for the sequence of acoustic frames (110) based on the third likelihood scores of the plurality of speech recognition hypotheses.
15. The ASR system (118) of claim 13 or claim 14, characterized in that The first likelihood score includes a conditional likelihood score and an internal language model likelihood score.
16. The ASR system (118) according to any one of claims 13 to 15, characterized in that: The first likelihood score for a particular speech recognition hypothesis comprises a first plurality of language model scores, each language model score in the first plurality of language model scores corresponding to a particular word-gram of the particular speech recognition hypothesis; the second likelihood score for the particular speech recognition hypothesis comprising a second plurality of language model scores, each language model score in the second plurality of language model scores corresponding to a particular word-gram for the particular speech recognition hypothesis; The set of fusion weights for the specific speech recognition hypothesis includes a first plurality of fusion weights and a second plurality of fusion weights, each fusion weight in the first plurality of fusion weights corresponds to a specific word element of the specific speech recognition hypothesis, and each fusion weight in the second plurality of fusion weights corresponds to a specific word element of the specific speech recognition hypothesis; and The learnable fusion module (240) is configured to generate the third likelihood score for the particular speech recognition hypothesis by: determining an internal language model score based on the first plurality of language model scores and the first plurality of fusion weights; determining an external language model score based on the second plurality of language model scores and the second plurality of fusion weights; and The internal language model score and the external language model score are combined.
17. The ASR system (118) according to any one of claims 13 to 16, characterized in that The audio encoder (210), the decoder (215) and the learnable fusion module (240) are jointly trained.
18. The ASR system (118) according to any one of claims 13 to 17, characterized in that The external language model (230) is trained on plain text data.
19. The ASR system (118) according to any one of claims 13 to 18, characterized in that The external language model (230) comprises a neural network language model, wherein the neural network language model comprises a stack of conformer layers or transformer layers.
20. The ASR system (118) according to any one of claims 13 to 18, characterized in that The external language model (230) comprises a neural network language model comprising a stack of long short-term memory (LSTM) layers.
21. The ASR system (118) according to any one of claims 13 to 20, characterized in that The audio encoder (210) comprises: a causal encoder comprising an initial stack of multi-head attention layers; and A non-causal encoder comprising a final multi-head attention layer stack overlying the initial multi-head attention layer stack.
22. The ASR system (118) according to any one of claims 13 to 21, characterized in that The learnable fusion module (240) is configured to calculate a per-word fusion weight of an internal language model score of the decoder (215) and an external language model score of the external language model (230).
23. The ASR system (118) according to any one of claims 13 to 22, characterized in that The minimum additive error rate includes the minimum word error rate.
24. The ASR system (118) according to any one of claims 13 to 17 and 19 to 23, characterized in that: The external language model (230) is trained on plain text data; and The audio encoder (210) and the decoder (215) are trained using Hybrid Autoregressive Transducer (HAT) decomposition to facilitate integration of the external language model (230) trained on the plain text data.
