Speech recognition method and speech recognition device

Abstract

To propose a customizable attention-based context bias method using an editable term list (called a bias list).SOLUTION: The proposed method combines bias phrase index loss and special tokens to enable efficient learning for detecting bias phrases in input speech data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an attention-based context bias method that can be customized using an editable term list.

Background Art

[0002] End-to-end (E2E) automatic speech recognition (ASR) methods show excellent performance. However, since the performance of such methods is inherently related to the context existing in the training data, E2E-ASR methods do not operate as required for unknown user contexts (e.g., technical terms, personal names, and playlists). For this reason, E2E-ASR methods must be easily contextualized by users or developers.

Summary of the Invention

Problems to be Solved by the Invention

[0003] This document proposes an attention-based context bias method that can be customized using an editable term list (referred to as a bias list).

Means for Solving the Problems

[0004] The proposed method can be efficiently learned to detect bias phrases in the input utterance data by combining a bias phrase exponential loss and special tokens. Also, to further improve the contextualization performance during inference, we propose a bias phrase boost (BPB) beam search algorithm based on the bias phrase exponential probability. Experimental results prove that the proposed method reliably improves the word error rate and character error rate of the target terms in the bias list for each of the Librispeech-960 (English) and our in-house (Japanese) datasets.

Brief Description of the Drawings

[0005] Figure 1: Overall structure including an acoustic encoder, a bias encoder, and a bias decoder in the proposed method. The BPB beam search algorithm is used during inference. Figure 1A: Bias Phrase Boost (BPB) beam search algorithm that utilizes bias phrase probabilities. Figure 1B: Table 1: Preliminary analysis of Librispeech - 100 test - clean. Figure 2: Effect of the bias phrase exponential loss. The horizontal and vertical axes respectively represent the token index s at B and the bias phrase. Figure 2A: Table 2: Main WER results (U - WER / B - WER) obtained on the Librispeech - 960 data. Bold values indicate cases where the proposed method is superior to the baseline, and underlined values indicate the best results. Figure 3: Effect of the decoding weight α of the BPB beam search for Librispeech - 960 bonus . Figure 4: Representative examples. Bold, red, and blue fonts respectively indicate bias phrases, incorrect recognition, and correct recognition. Figure 5: Table 3: Experimental results for our in - house Japanese dataset.

Mode for Carrying Out the Invention

[0006] 1. Introduction The end-to-end (E2E) automatic speech recognition (ASR) [1,2] method directly converts acoustic feature sequences into token sequences without requiring a large number of components such as acoustic models (AMs) and language models (LMs) used in conventional ASR systems. So far, various E2E-ASR methods have been proposed, including the connectionist temporal classification method (CTC) [3], the recurrent neural network transducer (RNN-T) [4], the attention mechanism [5,6], and various hybrid systems thereof [7-9]. Since the effectiveness of E2E-ASR methods is essentially related to the context of the training data, performance prediction may not always be satisfactory for a given user context. For example, personal names and technical terms tend to be important keywords in different contexts, but such terms do not frequently appear in the available training data, resulting in poor recognition accuracy. Since it is not realistic to train the model for all contexts during learning, users or developers should be able to easily contextualize the model without learning.

[0007] A typical approach to this problem is shallow fusion using an external LM [10-14]. For example, [10-12] uses a weighted finite state transducer (WFST) to construct an in-class LM and promote contextualization for target specific entities. Neural LM fusion methods have also been proposed [13,14]. LM fusion techniques attempt to improve accuracy by combining an E2E-ASR model with an external neural LM and then rescoring the hypotheses generated by the E2E-ASR model. However, whether using WFST or neural LM, learning an external LM requires additional learning steps.

[0008] Therefore, several methods that do not require re - learning have been proposed. These methods include knowledge graph modeling

[15] for recognizing out - of - dictionary named entities, context spell - correction

[15] using an editable term list, and an ASR model

[16] considering named entities for recognizing specific named entities based on phoneme similarity. However, these methods have limitations such as requiring a text - to - speech (TTS) model for learning and being unable to handle words other than the named entities of a given target.

[0009] Deep - bias methods [17 - 20] provide an alternative approach to achieve effective contextualization without requiring a re - learning process or a TTS model. In such methods, an end - to - end ASR model can be contextualized using an editable term list, which is called a bias list in this paper. Many deep - bias methods implement a cross - attention layer between the bias list and the input sequence to correctly recognize bias phrases. However, it has been found that simply adding a cross - attention layer to the bias list is not effective

[21] . Therefore, [21,22] introduced an additional branch designed to detect bias phrases that indirectly helps update the parameters of the cross - attention layer by an auxiliary loss. In contrast, [23,24] directly introduced an auxiliary loss function (called the bias - phrase index loss, which is described in Section 3.2) into the cross - attention layer for detecting the bias - phrase index. This approach enables direct parameter update of the cross - attention layer but cannot distinguish whether the output tokens are generated from the bias list or not. Also,

[23] requires a two - stage learning using a time - consuming, pre - trained ASR model.

[0010] This book proposes a deep bias method that uses both a direct auxiliary loss in the cross-attention layer called the bias phrase metric loss and special tokens for bias phrases to achieve more effective bias phrase detection. Different from the conventional indirect methods [21, 22], our method facilitates the effective learning of the cross-attention layer by the bias phrase metric loss. Furthermore, our technique is different from the current method

[23] in that it introduces special tokens for bias phrases. This enables the model to focus more efficiently on bias phrases and eliminates the need for a two-stage learning process. Additionally, we propose a Bias Phrase Boost (BPB) beam search algorithm that integrates the bias phrase metric probability during inference to enhance the performance of bias phrase recognition. The main contributions of this study are as follows: · We propose a deep bias model that uses both the bias phrase metric loss and special tokens for bias phrases. · We propose a Bias Phrase Boost (BPB) beam search algorithm to further improve the performance for target terms. · We prove that the proposed method is effective for both Librispeech-960 and our in-house Japanese dataset.

[0011] 2. Attention-based Encoder-Decoder ASR This section describes an attention-based encoder-decoder system consisting of an acoustic encoder and an attention-based decoder, which is extended to the proposed method.

[0012] 2-1. Acoustic Encoder The acoustic encoder consists of two convolutional layers, a linear projection layer, and a Ma conformable block

[25] . The conformable encoder converts the acoustic feature sequence X into a hidden state vector H = [h1,..., h t ∈ R T×d where d represents the dimension as follows: TIFF2025111300000002.tif9170

[0013] 2.2. Attention-based Decoder The posterior probability is formulated as follows: TIFF2025111300000003.tif18170

[0014] Here, s and S represent the token index and the total number of tokens, respectively. In Equation (1), H generated by the acoustic encoder and the previous token sequence y 0:s-1 in the case of, the attention-based decoder recursively estimates the next token y s as follows: TIFF2025111300000004.tif9170

[0015] The attention-based decoder consists of an embedding layer with a positional encoding layer, M d transformer blocks, and a linear layer. Each transformer block consists of a multi-head self-attention layer, a cross-attention layer (i.e., acoustic attention), and a linear layer with layer normalization (LN) layer and residual connection. Here, the acoustic attention layer including LN is formulated as follows: TIFF2025111300000005.tif17170

[0016] Here, U and U' represent the input and output of the acoustic attention layer, respectively. Also, the hybrid CTC / attention model [7] includes a CTC decoder. The attention-based decoder is extended to the proposed bias decoder in Section 3.2.

[0017] 3. Proposed Deep Bias Method Figure 1 shows the overall structure of the proposed method including an acoustic encoder, a bias encoder, and a bias decoder. These components are described in the following subsections.

[0018] 3.1. Bias Encoder The bias encoder includes an embedding layer with a positional encoding layer, a Me transformer block, an average pooling layer, and a bias list B = {b0, b1, ···, b N}, where n and b n represent the bias phrase index and the token sequence of the nth bias phrase (e.g., "play a song"), respectively. Here, b0 is a dummy phrase meaning "no bias". After applying zero-padding based on the maximum token length L max in the bias list B, the embedding layer and the transformer block extract a set of token-level feature sequences, G ∈ R (N+1)×Lmax×d as follows: TIFF2025111300000006.tif9170

[0019] Next, average pooling is performed, and a phrase-level feature sequence, V = [v0, v1, ···, v N ∈ R (N+1)×d is extracted as follows: TIFF2025111300000007.tif9170

[0020] 3.2. Bias Decoder As shown in Figure 1, the bias decoder is an extension of the attention-based decoder described in Section 2.2, and an additional cross-attention layer (i.e., bias attention) is incorporated into each transformer block. Different from Equation (2), the posterior probability is formulated using the bias list B as follows: TIFF2025111300000008.tif17170

[0021] For H, V, and y 0:s-1 in Equations (1) and (6), the bias decoder recursively estimates the token y s as follows, different from Equation (3): TIFF2025111300000009.tif9170

[0022] In the transformer block of the bias decoder, the bias attention layer including LN is formulated as follows: TIFF2025111300000010.tif18170

[0023] Also, the bias attention layer estimates the bias phrase index sequence ^n = [^n1, ^n2, ···, ^n s as follows: TIFF2025111300000011.tif32170

[0024] Here, u’ s denotes the s-th feature vector of U’ = [u’0, u’1, ···, u’ s . For example, if the bias phrase “play a song” (Figure 1) with bias index 2 is detected within the entire utterance “I play a song today”, the bias phrase index sequence is ^n = [0, 2, 2, 2, 0]. The model parameters are optimized using the cross-entropy loss as follows: TIFF2025111300000012.tif18170

[0025] Here, y gt and ^n gt represent the one-hot vector sequence of the reference transcription and the reference bias phrase index including the bias-free option. Here, we call L bidx the bias phrase index loss respectively.

[0026] 3.3. Learning During the learning process, the bias list B is randomly generated from the corresponding reference transcription for each batch. Specifically, the bias phrases of length from 2 to L max and token length from 0 to N utt are uniformly extracted for each utterance, and a total of N bias phrases are obtained (N utt ×n batch)。After the bias list B is randomly extracted, special tokens are added before and after the extracted phrase in the reference transcription ( <sob> / <eob>) is inserted and it is distinguished whether the output token originated from the bias list. The proposed method is optimized by multi-task learning using the loss as expressed in equations (12) and (13) and the weighted sum with the CTC loss (L ctc ) : TIFF2025111300000013.tif9170 Here, λ ctc , λ batt , and λ bidx represent learning weights.

[0027] 3.4. BPB Beam Search Algorithm Also, we propose a Bias Phrase Boost (BPB) beam search algorithm that utilizes the bias phrase probability as described in Algorithm 1. The bias decoder uses equation (8) (line 5) to obtain a special token <sob> / <eob>Token probability p including new is calculated. Next, we use Equation (11) and the argmax function (line 6) to estimate the bias phrase indicator ^n s . Here, the number N of bias phrases in the bias list B can increase sufficiently during inference and reduce the peak value after applying the softmax function in Equation (9). As a result, Equation (9) is approximated using the following top k score pruning: TIFF2025111300000014.tif18170

[0028] ^n s = 0 (i.e., "no bias"), for the token probabilities for the special tokens p new [sob] and p new [eob], a penalty is imposed based on the weight α pen (lines 8, 9), otherwise the corresponding token probabilities are increased according to the weight α bonus (lines 11 - 13). For example, if the detected bias phrase is "play a song", the token probabilities for "play", "a", and "song" are increased by α bonus . Based on the boost probability p new , top k score pruning such as conventional beam search [7] is performed.

[0029] 4. Experiments 4.1. Experiment Setup The input feature is an 80 - dimensional Mel - filter bank with a window length of 512 samples and a hop length of 160 samples. SpecAugment

[26] is applied. The acoustic encoder consists of two convolutional layers with two strides for downsampling, a 256 - dimensional linear projection layer, and 12 conformable blocks with 1024 linear units. The bias encoder and bias decoder each have 3 transformer blocks with 1024 linear units and 6 transformer layers with 2048 units. The attention layers in the acoustic encoder, bias encoder, and bias decoder are 4 - multi - head attention with a dimension d of 256. During the learning process, as described in Section 3.3, N utt = 2 and L max = 10, and the bias list B is randomly generated for each batch. In this experiment, the bias list B has a total of N = 50 to 200 bias phrases within a batch. The learning weights λ ctc , λ batt , and λ bidx (explained in Equation (14)) are set to 0.3, 0.7, and 1.0 respectively. The proposed model is trained using the Adam optimizer with a warm - up step of 15,000, a learning rate of 0.0015, and 150 epochs. During the decoding process, the hyperparameters k beam , k score , α bonus , and α pen (Section 3.4) are set to 20, 50, 1.0, and 10.0 respectively.

[0030] To evaluate the proposed method using ESPnet like the E2E-ASR toolkit

[28] , the Librispeech corpus (960h, 100h)

[27] is used. The proposed method is evaluated in terms of word error rate (WER), biased phrase WER (B-WER), and unbiased phrase WER (U-WER)

[29] . It should be noted that if there is a phrase inserted in the bias list, insertion errors are counted for B-WER, otherwise, insertion errors are counted for U-WER. The aim of the proposed method is to slightly degrade U-WER and improve B-WER and overall WER.

[0031] 4.2. Preliminary Analysis of the Proposed Technology First, as a preliminary experiment, we verify the effect of the proposed technology on Librispeech-100. Table 1 shows the biased phrase index loss, L explained in Equation (13) bidx、 Special tokens for biased phrases ( <sob> / <eob>) and shows the effect of the BPB beam search on the Librispeech-100 test-clean evaluation set with a bias list size of N = 100. Compared with the baseline (hybrid CTC / attention model [7]), simply introducing the bias attention layer does not improve the performance (A1 vs. B1), but the bias phrase metric loss sufficiently improves the B-WER, thereby improving the overall WER (B1 vs. B2). Figure 2 shows the visualized results of the bias phrase metric probabilities explained in Equation (11). By introducing the bias phrase metric loss into Equation (13), the bias phrase metric probabilities are correctly estimated. Also, special tokens ( <sob> / <eob>Introducing improves B-WER (B2 vs. B3) further. Furthermore, the BPB beam search technique slightly degrades U-WER (B3 vs. B4) while sufficiently improving B-WER.

[0032] 4.3. Main Results Table 2 shows the results obtained by the proposed method for Librispeech-960 data with different bias list sizes N. The baseline is the hybrid CTC / attention model [7]. For the bias list size N = 100, the proposed method improves B-WER, and similarly, sufficiently improves U-WER and WER. Also, the proposed BPB beam search technique further improves B-WER without reducing the overall WER and U-WER. B-WER and U-WER tend to degrade as the number of bias phrases N increases, but the proposed BPB beam search technique is particularly effective in suppressing the degradation of B-WER. As a result, the proposed method is superior to the baseline in terms of both WER and B-WER. The proposed method falls below the baseline when no bias phrases are used (N = 0), but usually, users register important keywords for them, so we do not consider this to be an important issue.

[0033] 4.4. Analysis of the BPB Beam Search Algorithm Figure 3 shows the effect of the decoding weight α bonus of the BPB beam search for Librispeech-960 test-other with the bias list size N = 100. The proposed method improves B-WER as described in Section 4.3 even without using the proposed BPB beam search technique, but the BPB beam search technique further improves B-WER. When the decoding weight α bonus > 1.5, B-WER, U-WER, and the overall WER decrease. B-WER, U-WER, and the overall WER are highest at α bonus = 1.0.

[0034] Figure 4 shows the inference results obtained from three different approaches: the baseline method, our proposed method without using the BPB beam search technique, and our proposed method incorporating the BPB beam search technique. Here, the bold font represents the bias phrases, and the red and blue characters represent the incorrectly and correctly recognized characters, respectively. Even when not using the BPB beam search technique, the proposed method reduces the misrecognition of bias phrases compared to the baseline. However, even if the corrected bias phrase metric is estimated, some bias phrases are not correctly recognized. On the other hand, the proposed BPB beam search technique recognizes bias phrases more accurately.

[0035] 4.5. Verification of Japanese Datasets We also confirm the validity of the proposed method for our in-house dataset, which includes 93 hours of Japanese speech data containing meeting and morning assembly scenarios, a corpus of spontaneous Japanese (581h)

[30] , and 181 hours of Japanese speech in a database developed by the Advanced Telecommunications Research Institute International

[31] using the same experimental setup described in Section 4.1. Table 3 shows the evaluation results obtained in the in-house dataset when N = 203 phrases such as personal names and technical terms are registered in the bias list B. The proposed method slightly degrades the overall CER but sufficiently improves the B-CER. Therefore, the proposed method is effective for both English and Japanese.

[0036] 5. Conclusion This study introduces a deep bias model that incorporates bias phrase metric loss and special tokens for bias phrases. Furthermore, the BPB beam search technique is used, and the bias phrase metric probability is utilized to improve accuracy. The experimental results prove that our model enhances the performance of both WER and B-WER. In particular, the BPB beam search clearly has the least impact on the overall WER and strengthens the B-WER performance in both English and Japanese datasets.

[0037] 6. References [1] Rohit Prabhavalkar, Takaaki Hori, Tara N Sainath, Ralf Schl¨uter, and Shinji Watanabe, "End-to-end speech recognition: A survey,” arXiv preprint arXiv:2303.03329, 2023. [2] Jinyu Li et al., "Recent advances in end-to-end automatic speech recognition,” APSIPA Transactions on Signal and Information Processing, vol. 11, no. 1. [3] Alex Graves, Santiago Fern´andez, Faustino Gomez, and J¨urgen Schmidhuber, "Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in Proc. ICML, 2006, pp. 369-376. [4] Alex Graves, "Sequence transduction with recurrent neural networks,” in Proc. ICML, 2012. [5] Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio, "Attention-based models for speech recognition,” Advances in neural information processing systems, vol. 28, 2015. [6] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals, "Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in Proc. ICASSP, 2016, pp. 4960-4964. [7] Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi, "Hybrid ctc / attention architecture for end-to-end speech recognition,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240-1253, 2017. [8] Tara N Sainath et al., "Two-pass end-to-end speech recognition,” arXiv preprint arXiv:1908.10992, 2019. [9] Yui Sudo, Muhammad Shakeel, Brian Yan, Jiatong Shi, and Shinji Watanabe, "4D ASR: Joint modeling of CTC, attention, transducer, and mask-predict decoders,” in Proc. Interspeech, 2023, pp. 3312-3316.

[10] Rongqing Huang, Ossama Abdel-Hamid, Xinwei Li, and Gunnar Evermann, "Class lm and word mapping for contextual biasing in end-to-end asr,” in Proc. Interspeech, 2020, pp. 4348-4351.

[11] Ian Williams, Anjuli Kannan, Petar Aleksic, David Rybach, and Tara Sainath, "Contextual speech recognition in end-toend neural network systems using beam search,” in Proc. Interspeech, 2018.

[12] Atsushi Kojima, "A study of biasing technical terms in medical speech recognition using weighted finite-state transducer,” Journal of the Acoustical Society of Japan, vol. 43, pp. 66-68, 2022.

[13] Anjuli Kannan, Yonghui Wu, Patrick Nguyen, Tara N Sainath, Zhijeng Chen, and Rohit Prabhavalkar, "An analysis of incorporating an external language model into a sequence-tosequence model,” in Proc. ICASSP, 2018, pp. 5824-5828.

[14] Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates, "Cold fusion: training seq2seq models together with language models,” in Proc. Interspeech, 2018, pp. 387-391.

[15] Xiaoqiang Wang et al., "Towards contextual spelling correction for customization of end-to-end speech recognition systems,” IEEE Trans. Audio, Speech, Lang. Process., vol. 30, pp. 3089-3097, 2022.

[16] Yui Sudo, Kazuya Hata, and Kazuhiro Nakadai, "Retrainingfree customized asr for enharmonic words based on a namedentity-aware model and phoneme similarity estimation,” in Proc. Interspeech, 2023, pp. 3312-3316.

[17] Golan Pundak, Tara N Sainath, Rohit Prabhavalkar, Anjuli Kannan, and Ding Zhao, "Deep context: End-to-end contextual speech recognition,” in Proc. SLT, 2018, pp. 418-425.

[18] Mahaveer Jain, Gil Keren, Jay Mahadeokar, and Yatharth Saraf, "Contextual rnn-t for open domain asr,” in Proc. Interspeech, 2020, pp. 11-15.

[19] Antoine Bruguier, Rohit Prabhavalkar, Golan Pundak, and Tara N Sainath, "Phoebe: Pronunciation-aware contextualization for end-to-end speech recognition,” in Proc. ICASSP, 2019, pp. 6171-6175.

[20] Saket Dingliwal, Monica Sunkara, Srikanth Ronanki, Jeff Farris, Katrin Kirchhoff, and Sravan Bodapati, "Personalization of ctc speech recognition models,” in Proc. SLT, 2023, pp. 302-309.

[21] Kaixun Huang, Ao Zhang, Zhanheng Yang, Pengcheng Guo, Bingshen Mu, Tianyi Xu, and Lei Xie, "Contextualized Endto-End Speech Recognition with Contextual Phrase Prediction Network,” in Proc. Interspeech, 2023, pp. 4933-4937.

[22] Minglun Han, Linhao Dong, Zhenlin Liang, Meng Cai, Shiyu Zhou, Zejun Ma, and Bo Xu, "Improving end-to-end contextual speech recognition with fine-grained contextual knowledge selection,” in Proc. ICASSP, 2022, pp. 491-495.

[23] Christian Huber, Juan Hussain, Sebastian St¨uker, and Alexander Waibel, "Instant one-shot word-learning for contextspecific neural sequence-to-sequence speech recognition,” in Proc. ASRU, 2021, pp. 1-7.

[24] Shilin Zhou, Zhenghua Li, Yu Hong, Min Zhang, Zhefeng Wang, and Baoxing Huai, "Copyne: Better contextual asr by copying named entities,” arXiv preprint arXiv:2305.12839, 2023.

[25] Anmol Gulati et al., "Conformer: Convolution-augmented transformer for speech recognition,” in Proc. Interspeech, 2020, pp. 5036-5040.

[26] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le, "SpecAugment: A simple data augmentation method for automatic speech recognition,” in Proc. Interspeech, 2019, pp. 2613-2617.

[27] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, "Librispeech: an asr corpus based on public domain audio books,” in Proc. ICASSP, 2015, pp. 5206-5210.

[28] ShinjiWatanabe et al., "ESPnet: End-to-end speech processing toolkit,” in Proc. Interspeech, 2018, pp. 2207-2211.

[29] Duc Le, Jain, et al., "Contextualized streaming end-to-end speech recognition with trie-based deep biasing and shallow fusion,” in Proc. Interspeech, 2021, pp. 1772-1776.

[30] Kikuo Maekawa, "Corpus of spontaneous Japanese: Its design and evaluation,” in ISCA & IEEE Workshop on Spontaneous Speech Processing and Recognition, 2003.

[31] Akira Kurematsu et al., "Atr japanese speech database as a tool of speech recognition and synthesis,” Speech Communication, vol. 9, no. 4, pp. 357-363, 1990.< / eob> < / sob> < / eob> < / sob> < / eob> < / sob> < / eob> < / sob>

Claims

【Claim 1】 An attention-based context bias method that can be customized using an editable term list.

Images