Knowledge distillation method, electronic device and storage medium

Abstract

The present invention discloses a knowledge distillation method, an electronic device, and a storage medium, wherein a knowledge distillation method comprises: transferring the knowledge of an AR teacher model to a NAR student model at two distillation levels, wherein the two distillation levels include frame-level distillation and sequence-level distillation, the frame-level distillation of the encoder and the sequence-level distillation of the encoder are completed after the linear layer of the encoder, and the frame-level distillation of the decoder and the sequence-level distillation of the decoder are evolved from an attention-based autoregressive model to a Mask-CTC autoregressive model, wherein the Mask-CTC is an improvement on the CTC result through a conditional mask language model. The results show that this knowledge transfer method narrows the gap between AR and NAR, and the improvement is significantly greater in more difficult evaluation sets (i.e., the test set in AISHELL-1 and test-other in Librispeech). After knowledge transfer and distillation, due to the high prediction accuracy of the AR teacher, the length error problem is greatly alleviated compared to the original NAR model.

Description

Technical Field

[0001] The present invention belongs to the technical field of knowledge distillation, and in particular relates to a knowledge distillation method, an electronic device and a storage medium. Background Art

[0002] In recent years, the performance of Automatic Speech Recognition (ASR) has been greatly improved by sequence-to-sequence modeling, such as Connectionist Temporal Classification (CTC), Recurrent Neural Network Transducer (RNNT), and Attention-based Encoder-Decoder (AED). Many early studies focused on autoregressive (AR) modeling, which uses a left-to-right probabilistic chain rule to generate token sequences. Despite their excellent performance, such AR models require L steps of incremental model computation to generate L tokens, resulting in high inference latency and considerable computational cost.

[0003] On the other hand, non-autoregressive (NAR) modeling generates token sequences in a constant step and eliminates the chain rule assumption. CTC plays an important role in recent NAR research. Modern NAR methods outperform CTC by leveraging alignment (alignment-based) and output token sequences (token-based). Based on a joint CTC / attention architecture, Mask-CTC utilizes a (Conditional) Masked Language Model (C)MLM) decoder to refine the CTC token sequence. Two auxiliary tasks are proposed to address the length prediction problem that arises in Mask-CTC. From another perspective, CTC alignment has demonstrated its advantages in building NAR models in Align-Refine, CASS-NAT, and ALNAT. In addition, the self-supervised pre-trained model wav2vec2.0 has achieved promising results in CTC modeling.

[0004] However, there are still two major challenges in NAR modeling: First, NAR models converge slowly and perform poorly compared to the most advanced (State-Of-The-Art, SOTA) AR models. Second, although NAR models are often favored due to their fast inference speed and high accuracy in resource-constrained situations, the large model size and high computational cost limit the application of NAR modeling. Knowledge distillation (transfer learning) is often used to solve such problems by teaching a smaller student model. Specifically, the student's goal is to imitate the soft target provided by a trained teacher model using Kullback-Leibler Divergence (KLD). However, in the process of realizing this application, the inventors found that when applying knowledge distillation on non-autoregressive ASR, the poor NAR teacher model limits the improvement. Summary of the Invention

[0005] An embodiment of the present invention provides a knowledge distillation method, an electronic device, and a storage medium, which are used to solve at least one of the above-mentioned technical problems.

[0006] In a first aspect, an embodiment of the present invention provides a knowledge distillation method, comprising: transferring the knowledge of an AR teacher model to a NAR student model at two distillation levels, wherein the two distillation levels include frame-level distillation and sequence-level distillation, the frame-level distillation of the encoder and the sequence-level distillation of the encoder are completed after the linear layer of the encoder, and the frame-level distillation of the decoder and the sequence-level distillation of the decoder evolve from an attention-based autoregressive model to a Mask-CTC autoregressive model, wherein the Mask-CTC is an improvement on the CTC result through a conditional mask language model.

[0007] In a second aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the knowledge distillation method of any embodiment of the present invention.

[0008] In a third aspect, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer performs the steps of the knowledge distillation method of any embodiment of the present invention.

[0009] Our method proposes a novel knowledge transfer and distillation architecture to distill knowledge from the AR teacher model to the NAR student model, leveraging the knowledge of the AR model to improve the performance of the NAR while reducing the model size. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 A flowchart of a knowledge distillation method provided by one embodiment of the present invention;

[0012] Figure 2 A diagram of a beam search decoding algorithm provided by one embodiment of the present invention;

[0013] Figure 3 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0015] The knowledge distillation method of the embodiment of the present application can be applied to the conversion of an autoregressive (AR) teacher model to a non-autoregressive (NAR) student model, and can be specifically used for automatic speech recognition.

[0016] Among them, a knowledge distillation method includes: transferring the knowledge of the AR teacher model to the NAR student model at two distillation levels, wherein the two distillation levels include frame-level distillation and sequence-level distillation, the frame-level distillation of the encoder and the sequence-level distillation of the encoder are completed after the linear layer of the encoder, and the frame-level distillation of the decoder and the sequence-level distillation of the decoder are evolved from an attention-based autoregressive model to a Mask-CTC autoregressive model, wherein the Mask-CTC is an improvement on the CTC result through a conditional mask language model.

[0017] In some optional embodiments, a beam search method is used on the Mask-CTC to expand the search space in the inference stage.

[0018] Further optionally, the beam search method includes: during each iteration, retaining a beam of a preset size, and fixing the number of updated tokens, and selecting at least a preset number of candidates in the candidate set based on the log-domain posterior probability.

[0019] In some other optional embodiments, the distillation of the decoder includes: for frame-level distillation, only selecting y mask Position, the objective function is passed <mask>The number of markers is normalized, where the y mask Positions include predicted mask tags obtained by randomly replacing manual annotations with special tags during training; for sequence-level distillation, approximate probabilities from the candidate set are used for calculation.

[0020] In some other optional embodiments, the calculation formula of the final loss L of the knowledge distillation is:

[0021]

[0022] Among them, γ enc is the weight coefficient of encoder knowledge distillation of the AR teacher model, γ dec is the weight coefficient of the decoder knowledge distillation of the AR teacher model, is the loss of the NAR student model, is the loss of encoder knowledge distillation of the AR teacher model, is the loss for decoder knowledge distillation of the AR teacher model.

[0023] In a further optional embodiment, the loss of the student model is a loss with multi-task learning, and the calculation formula is as follows:

[0024] Among them, α∈[0,1] is a hyperparameter, L ctc is the loss of connection temporal classification, L mlm is the loss of the masked language model.

[0025] In other optional embodiments, the NAR student model is used for automatic speech recognition.

[0026] The method of this embodiment proposes a new knowledge transfer and refinement architecture to distill knowledge from the AR teacher model to the NAR student model, utilizing the knowledge of the AR model to improve the performance of the NAR while reducing the size of the model.

[0027] It should be noted that the above method steps are not used to limit the execution order of each step. In fact, some steps may be executed simultaneously or in the opposite order to the step limit. This application has no limitation on this.

[0028] The following describes some problems encountered by the inventor in the process of implementing the present invention and a specific embodiment of the solution finally determined, so that those skilled in the art can better understand the solution of the present application.

[0029] Modern non-autoregressive (NAR) speech recognition systems aim to speed up inference; however, they suffer from performance degradation and large model size compared to autoregressive (AR) models.

[0030] Please refer to Figure 1 , which shows that the embodiment of the present application proposes a new knowledge transfer and refinement architecture, distilling knowledge from autoregressive (AR) to non-autoregressive models (NAR), using the knowledge of the AR model to improve the performance of the NAR while reducing the size of the model. The Chinese and English correspondences are as follows: AR / NAR: autoregressive / non-autoregressive, Teacher / Student: teacher, student, Linear: linear layer, Posterior: posterior probability, KD: knowledge distillation, Encoder / Decoder: encoder, decoder, CTC: connection time series classification, Masking: masking, Mask, obs: mask, observable, MlM: masked language model.

[0031] The frame- and sequence-level objectives are carefully designed for transfer learning.

[0032] To further improve the performance of NAR, a beam search method is developed on Mask-CTC to expand the search space in the inference stage.

[0033] Experiments show that on the AISHELL-1 benchmark, the proposed NAR beam search method reduces the CER by more than 5% with a tolerable real-time factor (RTF) increment.

[0034] Through knowledge transfer, a NAR student with the same size as the AR teacher model achieves 8 / 16% relative CER reduction on the AISHELL-1 development / test sets and more than 25% relative WER reduction on the Librispeech test set.

[0035] Furthermore, a 9x smaller NAR model achieved a 25% relative CER / WER reduction on the AISHELL-1 and Librispeech benchmarks through the knowledge transfer and refinement suggested in the embodiments of this application. The methods of the embodiments of this application can improve the inference speed of automatic speech recognition while maintaining high performance.

[0036] The following introduces the process of implementing the present application, the experiments conducted, and the relevant experimental data so that those skilled in the art can better understand the solution of the present application.

[0037] 1. Introduction

[0038] In recent years, the performance of automatic speech recognition (ASR) has been greatly improved by sequence-to-sequence modeling, such as connectionist temporal classification (CTC), recurrent neural network transducers (RNNT), and attention-based encoder-decoders (AED). Many early studies focused on autoregressive (AR) modeling, which uses a left-to-right probabilistic chain rule to generate token sequences. Despite their excellent performance, such AR models require L steps of incremental model computation to generate L tokens, resulting in high inference latency and considerable computational cost.

[0039] On the other hand, non-autoregressive (NAR) modeling generates token sequences in a constant step and eliminates the chain rule assumption. CTC plays an important role in recent NAR research. Modern NAR methods outperform CTC by leveraging alignment (alignment-based) and outputting token sequences (labeling-based). Based on a joint CTC / attention architecture, Mask-CTC utilizes a (conditional) masked language model ((C)MLM) decoder to refine the CTC token sequence. Related technologies propose two auxiliary tasks to solve the length prediction problem that arises in Mask-CTC. From another perspective, CTC alignment has demonstrated its advantages in building NAR models in Align-Refine, CASS-NAT, and ALNAT. In addition, the self-supervised pre-training model wav2vec2.0 has achieved promising results in CTC modeling.

[0040] However, there are still two major challenges in NAR modeling: First, NAR models converge slowly and perform poorly compared to state-of-the-art (SOTA) AR models. Second, although NAR models are often favored in resource-constrained settings due to their fast inference speed and high accuracy, the large model size and high computational cost limit the application of NAR modeling. Knowledge distillation (transfer learning) is often used to address such problems by teaching a smaller student model. Specifically, the student model aims to imitate the soft target provided by a trained teacher model using Kullback-Leibler divergence (KLD). However, when applying knowledge distillation on non-autoregressive ASR, poor NAR teacher models limit improvements.

[0041] In an embodiment of the present application, a novel architecture is proposed to improve the performance of non-autoregressive modeling by transferring and refining the knowledge of the autoregressive (AR) teacher model to the non-autoregressive (NAR) student model and the beam search decoding method. First, the embodiment of the present application introduces a beam search decoding method to expand the search space of the (conditional) mask language model ((C)MLM) decoder. Then, the embodiment of the present application extends the knowledge distillation technology by transferring the knowledge of the AR teacher model to NAR at two distillation levels, thereby improving the performance of the NAR student model. The encoder distillation is performed according to the previous settings of the embodiment of the present application. For the decoder distillation, the embodiment of the present application develops the frame-level and sequence-level distillation from the attention-based autoregressive model to Mask-CTC. The distillation loss is customized for the token-based NAR model, so that the NAR decoder can benefit from the AR decoder.

[0042] 2. Autoregressive and non-autoregressive ASR

[0043] Basically, the end-to-end ASR model transforms speech features X = [x1, x2, ..., x T ] T , x t ∈R F Mapped to a token sequence y = [y1, y2, ..., y L ] T , y1∈U, where F is the feature dimension and U represents the vocabulary set.

[0044] The traditional attention-based autoregressive (AR) ASR model first encodes the speech feature X into a hidden representation H: H = Encoder(X), which is then combined with the prior token y < 1 to estimate the posterior p(y1|y < 1, X):

[0045] p ar (y l |y<l,H)=Decoder(y<l,H) (1)

[0046] And the entire sequence probability is:

[0047]

[0048] During the inference process, the AR model generates hypotheses one by one

[0049] Connectionist Temporal Classification (CTC) is one of the earliest non-autoregressive (NAR) methods, which starts with a frame-level alignment Z = [z1, z2, ..., z T ] T Introducing many-to-one function η,z t ∈U∪{blank} to the token sequence y by merging the same labels and removing the blanks in Z. The sequence probability is expressed as:

[0050]

[0051] where η is a many-to-one function from Z to y. During inference, greedy CTC predicts the alignment by selecting the label with the highest probability at each step.

[0052] Mask-CTC is a popular example of NAR ASR, which is actually an improvement on CTC results through conditional mask language model (MLM). During training, the manually labeled results (groundtruth) y are specially marked <mask>With random replacement, the MLM decoder generates the observed labels y obs =y\y mask Predicted mask label y mask :

[0053]

[0054] During inference, the output is initialized by CTC greedy decoding, and low confidence tokens are ranked according to a predefined threshold p thr use <mask>Afterwards, the mask is filled using the easy-first algorithm: Fill all masks in iteration, where N represents <mask>The total number of , each iteration predicts the top k tokens with the highest confidence, guided by MLM:

[0055]

[0056] in, C is <mask>The labeled candidate set and is the updated result after mask filling.

[0057] The joint CTC / attention architecture is widely used in modern AR and NAR ASR models with a multi-task learning based loss function:

[0058] L jca =αL ctc +(1-α)L att (6)

[0059] Where α∈[0, 1] is a hyperparameter, for AR ASR, L att =L ar , for NARASR, L att =L mlm .

[0060] 3. Suggested Methods

[0061] In this example, we introduce: (1) the proposed NAR beam search method and (2) the distillation architecture for transferring knowledge from AR to NAR ASR.

[0062] 3.1 Beam Search for NAR ASR

[0063] The embodiment of the present application designs a beam search decoding method to expand the search space of the MLM decoder. The process is as follows Figure 2 As shown in Algorithm 1 in

[15] , Ω1 is the sorted queue to be updated at the beginning of an iteration, and Ω0 stores the final Ω1 after one iteration. During each iteration, a beam of size B is retained, and the number of updated tokens is fixed and determined by k (i.e. ) is calculated. Based on the log-domain posterior probability and formula (5), the Top-B candidates are selected.

[0064] The algorithm is explained as follows:

[0065] Algorithm 1: Beam Search Decoding in Non-Autoregressive Models

[0066] 1 Assign the greedy search result of CTC connection time series classification to y^

[0067] 2 Mask some tokens to mask according to p_thr to generate y^mask, y^obs

[0068] 3. Construct the receiving set Omega_0, which is a set that stores all available hypothesis sequences and is initialized to y^obs

[0069] 4 for loop, the maximum number of iterations is N / K times

[0070] 5. Construct a priority queue Omega_1 to store B pending sequences.

[0071] 6 Calculate k

[0072] 7For all sequences in Omega_0,

[0073] 8According to formula 5, the top B candidates are obtained

[0074] 9 Add these candidates to Omega_1

[0075] 10 will Omega_0=Omega_1

[0076] 11 Returns the maximum a posteriori y^ of z in Omega_0

[0077] 3.2 Knowledge Transfer and Distillation from Autoregressive to Non-Autoregressive ASR

[0078] As mentioned above, due to the poor performance of NAR teachers, the knowledge distillation performance of NAR is limited. This embodiment of the application proposes knowledge transfer and distillation from autoregressive (AR) to non-autoregressive (NAR) ASR, breaking through the limitations of NAR.

[0079] First, the present embodiment introduces two knowledge distillation techniques based on Kullback-Leibler divergence (KLD): KLD(P, Q) = ∑ i P i log(P i / Q i ), where P and Q are the output distributions of the teacher model and the student model respectively.

[0080] The frame-level knowledge distillation formula as the basic distillation standard is as follows:

[0081]

[0082] Among them, P t (c) and Q t (c) is the posterior probability of label c at timestamp t for teacher model P and student model Q. H, y obs and y < t are the conditions for the above probability, but are omitted for simplicity. t (c)logP t (c) is omitted in the calculation of KLD loss due to the teacher model being frozen during training.

[0083] Sequence-level knowledge distillation is another distillation standard:

[0084]

[0085] in, where is an assumption from the teacher model, τ is the set of all possible sequences, and is omitted similarly to Equation (7). Using this sequence-level knowledge distillation is unsustainable because the present embodiment is approaching an exponentially large sequence distribution τ. Similar to MWER training, the N-best candidate set Ω is accessed via beam search, and then It can be approximated as:

[0086]

[0087] Then, the embodiment of the present application can implement knowledge distillation loss in the following way:

[0088] L KD =β F L F-KD +β S L S-KD (10)

[0089] Among them, β F , β S are the hyperparameters for frame-level and sequence-level knowledge distillation, respectively.

[0090] Figure 1 : An overview of knowledge distillation proposed from autoregressive to non-autoregressive ASR.

[0091] like Figure 1 As shown in Figure 2, the proposed knowledge distillation method is divided into two parts: the first part is the distillation after the encoder, and the second part is the distillation after the decoder. This is done after the linear layer of the encoder, which has a similar L layer as in "M. Huang, Y. You, Z. Chen, Y. Qian, and K. Yu, "Knowledge distillation for sequence model," Proc. Interspeech 2018, p. 5, 2018" F-KD and L S-KD The decoder distillation is set up as follows. For frame-level distillation, only y mask Position, so the objective function is <mask>Normalize by the number of markers:

[0092]

[0093] For sequence-level distillation, the approximate probability P′ from N-bestΩ is used:

[0094]

[0095] The final loss is:

[0096]

[0097] Among them, γ enc , γ dec is the weight coefficient of knowledge distillation between encoder and decoder.

[0098] 4. Experiment

[0099] 4.1 Dataset

[0100] The experiments in this application's examples were conducted on the Mandarin AISHELL-1 and English Librispeech corpora. AISHELL-1 includes a 150-hour training set, a development (dev) set, and a test set for evaluation, while Librispeech has a 960-hour training set and a test-clean / other (test c / o) set for testing. This application's examples report the character error rate (CER) on AISHELL-1 and the word error rate (WER) on Librispeech.

[0101] Table 1: Knowledge transfer and distillation performance on the AISHELL1 corpus (CER) (%) and the Librispeech test corpus (WER) (%). 'I+D' is the sum of insertion and deletion errors, and 'A' is the total CER / WER. The brackets in '#Param' contain 'XS', 'S', 'M', and 'L', as shown in Table 2. 'Same size' indicates that NAR has the same model scale as AR, and 'Smaller' indicates that NAR is 9 times smaller than AR.

[0102]

[0103] 4.2 Model Description

[0104] For acoustic feature extraction, 80-dimensional Mel filter bank (Fbank) features are extracted using global level cepstral mean and variance normalization (CMVN). In terms of data augmentation, speed perturbation is only applied to AISHELL-1 and SpecAugment for both datasets. For text modeling, 5000 English byte pair encoding (BPE) subword units are used for English and 4233 characters are used for Mandarin. The baseline follows the recipe of ESPnet v2, which is a 12-layer consistency encoder with four times downsampling and a 6-layer transformer decoder. The weight α of the CTC module is fixed to 0.3.

[0105] For knowledge transfer and distillation, the present embodiment first uses LF-KD to train a new NAR student model from scratch for 80 epochs. The hyperparameters are set to βF = 1.0, βS = 0, γenc = 0.5, and γdec = 0.3. The present embodiment then fine-tunes the distillation process by adding LS-KD, setting the adjustment parameters to βF = 1.0, βS = 1.0, γenc = 0.5, and γdec = 0.5 for a total of 20 epochs. In equations 8 and 9, the present embodiment uses a beam size |Ω| = 10, which is consistent with the decoding hyperparameters in the AR model.

[0106] Different NAR student model sizes are explored in Table 2, labeled as large (L), medium (M), small (S), and extra small (XS). The AR teacher model maintains size L for Librispeech and size M for AISHELL-1.

[0107] During the inference phase, the language model is not used in the following experiments. The model parameters are averaged over the last 5 checkpoints. For the autoregressive model, joint CTC / attention single-pass decoding is used with a beam size of 10 and an interpolated CTC score of 0.3. For non-autoregressive Mask-CTC decoding, this embodiment follows the beam decoding method in Section 3.1, with a beam size of B = 10, a threshold of pthr = 0.99, and K = 2 for the AISHELL-1 and Librispeech corpora.

[0108] 4.3. Results of NAR Beam Decoding

[0109] As described in Section 3.1, the embodiments of the present application first evaluate the beam search performance using the real-time factor (RTF) in Table 3. The RTF is calculated using a single core of an Intel-Xeon E5-2690 CPU on the test set. The NAR(M) model speeds up the AR(M) model by more than 10 times, as the RTF of AR(M) is 0.58 and the RTF of AR(S) is 0.31. Without significantly reducing the inference speed (1.5 times slower than "Beam1"), the beam decoding method achieves better performance on the test set than the greedy (Beam1), with a relative WER reduction of 5% to 9%. The improvement rate decreases as the beam size B increases.

[0110] Table 2: Model hyperparameters for different AR and NAR Conformer scales for L, M, X, and XS.

[0111]

[0112] Table 3: Non-autoregressive Mask-CTC performance (CER) on the AISHELL-1 corpus. Real-time factors (RTF) on the test set are reported.

[0113]

[0114] 4.4. Knowledge Transfer and Distillation Results

[0115] Table 1 compares knowledge transfer distillation with other modern AR and NAR models on the AISHELL-1 and Librispeech datasets to verify the performance.

[0116] AISHELL-1: As shown in Table 1, the teacher AR model achieves a relative CER reduction of over 24% compared to NAR(M) and 40% compared to NAR(XS). After knowledge distillation, NAR(M) with "+LF-KD" achieves an 8% and 16% relative CER reduction on the development and test sets, respectively, while NAR(M) with "+LF-KD" and "++LS-KD" show a further 15% CER reduction on the test set. The student achieves competitive performance (5.0% / 5.4% CER) compared to state-of-the-art NAR models such as CASS-NAT or AL-NAT. For distilled NAR(XS), similar results of 18% / 25% CER reduction are achieved on both evaluation sets.

[0117] Librispeech: Table 1 shows a performance comparison on the large Librispeech corpus. AR(L) was used as the teacher model, while NAR(L,S) was used as the student model. The results are consistent with those of AISHELL1 in Table 1. LF-KD and LS-KD further improve the performance of the NAR Mask-CTC model at the L (3.3 / 7.8% WER) and S (3.7 / 9.2% WER) scales, resulting in a 25% relative WER reduction. However, due to limitations of the AR teacher model, the insertion and deletion error rates on AR(L) are high.

[0118] Results show that this knowledge transfer approach narrows the gap between AR and NAR, with significantly greater improvements on more difficult evaluation sets (i.e., the test set in AISHELL-1 and test-other in Librispeech). After knowledge transfer and distillation, the length error problem is significantly alleviated compared to the original NAR model due to the high prediction accuracy of the AR teacher. Furthermore, both LF-KD and LS-KD, attributed to the reduction of insertion and deletion errors ('I+D'), push the length error problem to the limit of 0.2% CER for 'I+D' in AISHELL-1 and 1.4% for Librispeech test-other. Meanwhile, the NAR student model achieves comparable results to other state-of-the-art NAR methods, including wav2vec2-CTC, improved CASS-NAT, and ALNAT.

[0119] 5. Conclusion

[0120] In this paper, the embodiments of the present application propose a novel knowledge transfer and distillation architecture that leverages knowledge from AR models to improve NAR performance while reducing model size. To further improve NAR performance, the embodiments of the present application propose a beam search method on Mask-CTC that expands the search space during the inference phase. Experiments show that NAR beam search achieves a 5% relative reduction on the AISHELL-1 dataset with a tolerable RTF increment. For knowledge distillation, most results achieve a relative CER / WER reduction of more than 15% on both large and small NAR modeling.

[0121] In other embodiments, embodiments of the present invention further provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions can execute the knowledge distillation method in any of the above method embodiments;

[0122] As an embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are configured as follows:

[0123] The knowledge of the AR teacher model is transferred to the NAR student model at two distillation levels, wherein the two distillation levels include frame-level distillation and sequence-level distillation, the frame-level distillation of the encoder and the sequence-level distillation of the encoder are completed after the linear layer of the encoder, and the frame-level distillation of the decoder and the sequence-level distillation of the decoder are evolved from an attention-based autoregressive model to a Mask-CTC autoregressive model, wherein the Mask-CTC is an improvement on the CTC result through a conditional mask language model.

[0124] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the knowledge distillation device, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the knowledge distillation device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0125] An embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes any one of the above-mentioned knowledge distillation methods.

[0126] Figure 3 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 3 As shown, the device includes: one or more processors 310 and a memory 320, Figure 3 A processor 310 is used as an example. The apparatus of the knowledge distillation method may further include: an input device 330 and an output device 340. The processor 310, the memory 320, the input device 330 and the output device 340 may be connected via a bus or other means. Figure 3 The example of a bus connection is taken. The memory 320 is the non-volatile computer-readable storage medium mentioned above. The processor 310 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 320, that is, implements the knowledge distillation method of the above method embodiment. The input device 330 can receive input digital or character information, and generate key signal input related to user settings and function control of the communication compensation device. The output device 340 may include a display device such as a display screen.

[0127] The above-mentioned product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0128] As an embodiment, the electronic device is applied to a knowledge distillation device and is used for a client, and includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0129] The knowledge of the AR teacher model is transferred to the NAR student model at two distillation levels, wherein the two distillation levels include frame-level distillation and sequence-level distillation, the frame-level distillation of the encoder and the sequence-level distillation of the encoder are completed after the linear layer of the encoder, and the frame-level distillation of the decoder and the sequence-level distillation of the decoder are evolved from an attention-based autoregressive model to a Mask-CTC autoregressive model, wherein the Mask-CTC is an improvement on the CTC result through a conditional mask language model.

[0130] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

[0131] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and are primarily designed to provide voice and data communications. These terminals include smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones.

[0132] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0133] (3) Portable entertainment devices: These devices can display and play multimedia content. These devices include audio and video players (such as iPods), handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0134] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0135] (5) Other electronic devices with data interaction functions.

[0136] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. That is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0137] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or certain parts of the embodiment.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.< / mask> < / mask> < / mask> < / mask> < / mask> < / mask>

Claims

1. A method for training an automatic speech recognition system, based on knowledge distillation from an autoregressive (AR) teacher model to a non-autoregressive (NAR) student model, wherein the non-autoregressive (NAR) student model is a non-autoregressive automatic speech recognition system. The method comprises: Transferring the knowledge of the AR teacher model to the NAR student model at two distillation levels, wherein the two distillation levels include frame-level distillation and sequence-level distillation. The frame-level distillation of the encoder and the sequence-level distillation of the encoder are completed after the linear layer of the encoder, and the frame-level distillation of the decoder and the sequence-level distillation of the decoder are evolved from an attention-based autoregressive model to a Mask-CTC autoregressive model, wherein the Mask-CTC is an improvement on the CTC result through a conditional mask language model. Since the exponential size of the sequence distribution approximation using sequence-level knowledge distillation is unbearable, the N-best candidate set is accessed through beam search; During the training process, all parameters of the autoregressive AR teacher model are frozen, and the manual annotation results of the non-autoregressive automatic speech recognition system are specially marked. <mask>With random replacement, the conditional mask language model decoder predicts the mask token based on the observed token; during inference, the output is initialized by CTC greedy decoding, and low confidence tokens are replaced with the special token according to a predefined threshold. <mask> Instead, the masks are then filled using an easy-first algorithm: all masks are filled in iterations, and each iteration predicts the top k tokens with the highest confidence, guided by the conditional masked language model, where the tokens are the token sequences obtained by mapping the speech features of the non-autoregressive automatic speech recognition system;< / mask> < / mask> Among them, a beam search method is used on the Mask-CTC to expand the search space in the inference stage. The beam search method includes: during each iteration, retaining a beam of a preset size, and the number of updated tokens is fixed, and selecting at least a preset number of candidates in the candidate set based on the log-domain posterior probability.

2. The method according to claim 1, wherein The distillation of the decoder includes: For frame-level distillation, only y mask Position, the objective function is passed <mask>The number of markers is normalized, where the y mask The positions include the predicted mask tags obtained by randomly replacing the manual annotations with special tags during training;< / mask> For sequence-level distillation, approximate probabilities from the candidate set are used for computation.

3. The method according to claim 1, wherein The calculation formula of the final loss L of the knowledge distillation is: Among them, γ enc is the weight coefficient of encoder knowledge distillation of the AR teacher model, γ dec is the weight coefficient of the decoder knowledge distillation of the AR teacher model, is the loss of the NAR student model, is the loss of encoder knowledge distillation of the AR teacher model, is the loss for decoder knowledge distillation of the AR teacher model.

4. The method according to claim 3, wherein: The loss of the student model is the loss of multi-task learning, and the calculation formula is as follows: Among them, α∈[0,1] is a hyperparameter, L ctc is the loss of connection temporal classification, L mlm is the loss of the masked language model.

5. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 4.

6. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images