A cross-distillation-based compression method for end-to-end speech recognition models

Abstract

A cross-distillation-based end-to-end speech recognition model compression method relates to speech recognition. A cross-distillation-based end-to-end speech recognition model compression method is provided. Based on the knowledge distillation method, a cross-distillation method is designed to address the problem of insufficient teacher model learning and improve the accuracy of the student model. The method includes: 1) constructing a teacher model and a student model based on a Conformer model; 2) pre-training the teacher model; 3) the teacher model guiding the student model training through a cross-distillation method; and 4) inferring the student model. Aiming at the difficulties in deploying speech recognition models on devices, an optimized training scheme for a small model is designed. While ensuring model size compression, the student model's decoder can directly obtain feature learning from the teacher model's encoder. The features extracted by the student model's encoder are input into the teacher model's decoder and corrected by the teacher model, thereby improving the accuracy of the small model.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to an end-to-end speech recognition model compression method based on cross distillation. Background Art

[0002] Speech recognition is a technology that allows machines to convert speech signals into corresponding text through recognition. It is widely used in various smart devices. In recent years, with the development of deep learning, end-to-end models have achieved excellent results in speech recognition tasks. These models integrate the different modules of traditional speech recognition systems into a unified architecture, directly converting input speech features into corresponding label sequences for output, greatly simplifying the training process of speech recognition models.

[0003] To achieve high recognition accuracy, end-to-end speech recognition models often require more complex network structures and a larger number of parameters, which significantly increases the difficulty of model deployment. A typical model deployment solution is to deploy the model in the cloud. Mobile devices upload audio to a cloud server via the network. The cloud server then calls the speech recognition model for recognition and transmits the recognition results back to the device in real time. This approach is susceptible to network environment fluctuations. When the network fluctuates significantly, the user experience will be significantly degraded due to network latency. Deploying the speech recognition model directly on the device for local offline recognition can effectively avoid potential network fluctuations and provide users with a better user experience. Device-side storage and computing resources are generally limited. Therefore, the industry uses model compression methods to reduce the model size, thereby deploying the model on the device.

[0004] Common model compression methods include knowledge distillation, model pruning, and model quantization. Pruning methods suffer from unstable performance, while quantization methods are difficult to optimize. Knowledge distillation, on the other hand, allows for flexible design of the compressed model's structure and size, making it easy to migrate and deploy across devices. Therefore, this paper primarily improves knowledge distillation methods. Knowledge distillation methods typically involve two steps: first, training a teacher model; second, using the teacher model to guide the training of smaller models. Currently, several experts and scholars have proposed knowledge distillation methods for model compression. Hinton (Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al., "Distilling the knowledge in a neural network," arXiv preprint arXiv:1503.02531, vol. 2, no. 7, 2015) proposed a classic knowledge distillation method. First, a teacher model is trained. The teacher model's softmax output is then used as a soft label. The soft label is then weighted and combined with the true label to form the learning objective for the student model. This method assumes that the soft labels output by the teacher model contain richer information and provide better guidance for the training of the student model than the true label. Wonpyo (W.Park, D.Kim, Y.Lu and M.Cho, "Relational Knowledge Distillation," 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp.3962-3971, doi:10.1109 / CVPR.2019.00409) proposed that migrating the relationship between samples would be better than migrating the features of a single sample, such as the distance between two samples and the angle between three samples. The FitNet algorithm (Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio, “Fitnets: Hints for thin deep nets,” arXiv preprint arXiv:1412.6550, 2014) uses intermediate layer features for distillation. Its main idea is to use the output features of the intermediate layers of the teacher model to train the intermediate layers of the student model, thereby reducing the difference between the output of the student model and the teacher model. Existing methods do not fully learn from the teacher model, and the student model's performance can be further improved.

[0005] Chinese patent CN202111264520.X discloses a model compression method based on multi-layer feature distillation of attention. First, the teacher model is pre-trained. When training the student model, the network layer of the student model and the output of the network layer of the teacher model are used to calculate the mean square error as the loss function. However, this patent has the following defects: 1. When training the student model, the network layer of the student model needs to calculate the mean square error with the output of the last layer of the teacher model layer by layer, which is computationally expensive. 2. Each layer of the student model network learns the output information of the last layer of the teacher model, resulting in poor generalization ability of the student model network. Summary of the Invention

[0006] The purpose of the present invention is to address the above-mentioned defects in the prior art and provide an end-to-end speech recognition model compression method based on cross distillation. Based on the knowledge distillation method, the present invention addresses the problem that the above-mentioned algorithm does not sufficiently learn the teacher model, and designs a cross distillation method to further improve the accuracy of the student model.

[0007] The present invention provides an end-to-end speech recognition model compression method based on cross distillation, comprising the following steps:

[0008] 1) Construct a teacher model and a student model based on the Conformer model;

[0009] 2) Pre-training the teacher model;

[0010] 3) The teacher model guides the student model to train through cross distillation method;

[0011] 4) Perform inference on the student model.

[0012] In step 1), the Conformer model is used as the base model of the teacher model and the student model, where the student model reduces the number of parameters by reducing the number of network layers and the number of network nodes in each layer, thereby achieving the purpose of model compression.

[0013] In step 2), the teacher model is trained using the training data until the teacher model converges.

[0014] In step 3), the teacher model participates in the training of the student model, and the network parameters of the teacher model are frozen; during the training process, both the teacher model and the student model will perform forward propagation. In the present invention, in addition to the original forward propagation method (i.e., the encoder output features of the teacher model are the input of the decoder of the teacher model, and the encoder output features of the student model are the input of the decoder of the student model), the encoder output features of the teacher model and the student model are crossed before being input into the decoder (i.e., the encoder output features of the teacher model are input into the decoder of the student model, and the encoder output features of the student model are input into the encoder of the teacher model), and then the KL divergence loss of the probability distribution of the two encoder outputs and the true distribution is calculated. The present invention also designs an adaptive distillation weight function, which calculates the accuracy of the teacher model in predicting the current batch of data based on the decoder output obtained by the original forward propagation method of the teacher model, so as to adjust the weight of the teacher model in participating in the guidance. When the teacher model has a high recognition accuracy for the current batch of data, the proportion of the KL divergence loss in the total loss is high; when the teacher model has a low recognition accuracy for the current batch of data, the original cross entropy loss accounts for a high proportion of the total loss. In addition, the present invention also calculates the mean squared error between the encoder output features of the last layer of the student model and the encoder output features of the last layer of the teacher model. During the backpropagation process, the teacher model network parameters do not need to be updated and therefore do not participate in backpropagation. The student model optimizes the above mean squared error, KL divergence loss, and its own cross entropy loss until the student model converges.

[0015] In step 4), only the network parameters of the student model need to be loaded during inference, and the teacher model does not need to be involved.

[0016] Compared with the existing technology, the present invention has the following technical effects: the present invention optimizes the design of the training scheme for the small model to address the difficulties in deploying the speech recognition model on the device; the cross-distillation-based training method proposed in the present invention enables the decoder of the student model to directly obtain the features from the encoder of the teacher model for learning while ensuring the compression of the model size. The features extracted by the encoder of the student model are input into the decoder of the teacher model and corrected by the teacher model, which can further improve the accuracy of the small model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the process of training a student model using cross-distillation for a joint teacher model. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the following embodiments will be further described in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0019] This paper proposes to cross-pollinate the network structures of the student model and the teacher model, so that the student model's network can directly use information from the teacher model for training, while the teacher model can guide and correct the student model. The main steps are as follows:

[0020] 1) Constructing teacher model and student model based on Conformer model

[0021] The teacher model used in the present invention consists of a 12-layer encoder and a 6-layer decoder, where each encoder and decoder layer contains 2048 neuron nodes, 4 attention heads are set, and the dimension of the attention head is 256. The student model consists of a 6-layer encoder and a 3-layer decoder, where each encoder and decoder layer contains 1024 neuron nodes, and contains 4 attention heads like the teacher model, and the dimension is set to 256.

[0022] First, the teacher model is trained using the FBank features of speech, with a learning rate of 0.001, a batch size of 16, and 60 training iterations. The trained teacher model is used to guide the training of the student model.

[0023] 2) Pre-training the teacher model

[0024] The teacher model is trained using the training data until the teacher model converges.

[0025] 3) The teacher model guides the student model to train through cross distillation method

[0026] like Figure 1 The cross-distillation method in this invention includes two cross-distillation schemes. One scheme splices the encoder of the teacher model with the decoder of the student model. During the forward propagation process, the features output by the encoder of the teacher model are input into the decoder of the student model. The other scheme splices the encoder of the student model with the decoder of the teacher model. During the forward propagation process, the features output by the encoder of the student model are input into the decoder of the teacher model. The cross-entropy loss between the final output of the two decoders and the true label is then calculated before backpropagation. During the backpropagation process, both schemes freeze the network portion of the teacher model and only update the parameters of the student model network until the student model converges.

[0027] In order to prevent the errors of the teacher model from being transferred to the student model through the knowledge distillation method, the present invention designs an adaptive distillation weight function. The weight calculation formula is as follows:

[0028]

[0029] Among them, Acc tWith Acc s Represents the accuracy of the teacher model and the student model respectively. Based on this, the weight of the teacher model in the student model training can be adjusted to allow the student model to receive more guidance from the teacher model in the initial stage of training. When the accuracy of the student model and the teacher model are close, the student model is more inclined to directly learn the true label. The specific implementation formula is as follows:

[0030]

[0031]

[0032] in, and They represent the loss functions of the two methods: teacher model encoder splicing student model decoder and student model encoder splicing teacher model decoder, Represents the cross entropy loss calculated directly from the decoder output of this method and the true label, Represents the KL loss calculated between the decoder output of this method and the output of the teacher model.

[0033] In addition, in order to make the output features of the student model encoder and the teacher model encoder as similar as possible, the present invention also uses the mean square error loss function (MSE Loss) to measure the difference in the output features of the two encoders. The loss is defined as follows:

[0034]

[0035] Among them, δ T and δ S They represent the network parameters of the teacher model and the student model respectively, Enc represents the output of the last layer of the encoder, and D represents the dimension of the output feature of the last layer of the encoder.

[0036] Finally, the calculated and After weighting, back propagation is performed to update the network parameters of the student model, and training is stopped when the student model converges.

[0037] When training the student model, the FBank feature is also used, the learning rate is set to 0.001, the batch size is set to 16, and the training iteration is 60 rounds.

[0038] 4) Use the student model for inference

[0039] During inference, only the network parameters of the student model need to be loaded, and the teacher model does not need to be involved.

[0040] In the present invention, the network structures of the teacher model and the student model are configured according to Table 1.

[0041] Table 1 shows the structural configuration of the teacher model and the student model

[0042] Model Number of encoder layers Number of decoder layers Number of network nodes Attention Dimension Number of attention heads Parameter quantity Teacher Model 12 6 2048 256 4 49.3M Student Model 6 3 1024 256 4 19.3M

[0043] Table 2 shows the results on the Aishell1 dataset

[0044] Exp id Model Methods Development set CER (%) Test set CER (%) T0 - 4.58％ 4.75％ S0 Baseline 5.28％ 5.92％ S1 Cross KD 4.76％ 5.24％

[0045] The comparative experiments in Table 2 are described as follows:

[0046] T0: teacher model trained using the Aishell1 dataset;

[0047] S0: Student model trained using the Aishell1 dataset without using distillation techniques;

[0048] S1: Student model trained on the Aishell1 dataset using the Cross KD algorithm;

[0049] It can be found from Table 2 that the cross distillation algorithm proposed in this invention can effectively improve the accuracy of the student model.

Claims

1. A cross-distillation-based end-to-end speech recognition model compression method, characterized by The following steps are involved: 1) Construct a teacher model and a student model based on the Conformer model; 2) Pre-training the teacher model; 3) The teacher model guides the student model to train through the cross-distillation method. The specific steps include: (1) During the training process, both the teacher model and the student model perform forward propagation. In addition to the forward propagation method, the encoder output features of the teacher model and the student model are crossed before being input into the decoder, and then the KL divergence loss between the probability distribution of the two encoder outputs and the true distribution is calculated; the forward propagation means that the encoder output features of the teacher model are the input of the decoder of the teacher model, and the encoder output features of the student model are the input of the decoder of the student model, and the crossing means that the encoder output features of the teacher model are input into the decoder of the student model, and the encoder output features of the student model are input into the encoder of the teacher model; (2) Design an adaptive distillation weight function. According to the decoder output obtained by the original forward propagation method of the teacher model, calculate the accuracy of the teacher model in predicting the current batch of data, and adjust the weight of the teacher model in participating in the guidance. When the teacher model has a high recognition accuracy for the current batch of data, the proportion of KL divergence loss in the total loss is high; when the teacher model has a low recognition accuracy for the current batch of data, the proportion of original cross entropy loss in the total loss is high. (3) Calculate the mean square error between the encoder’s last layer output features of the student model and the encoder’s last layer output features of the teacher model; (4) During the back propagation process, the teacher model network parameters do not need to be updated and do not participate in back propagation. The student model optimizes the above mean square error, KL divergence loss and its own cross entropy loss until the student model converges; 4) Perform inference on the student model.

2. The end-to-end speech recognition model compression method based on cross distillation as claimed in claim 1, characterized in that In step 1), the teacher model and student model based on the Conformer model are constructed, and the Conformer model is used as the basic model of the teacher model and the student model, wherein the student model reduces the number of parameters by reducing the number of network layers and the number of network nodes in each layer, thereby achieving the purpose of model compression.

3. The end-to-end speech recognition model compression method based on cross distillation as claimed in claim 1, characterized in that In step 2), the teacher model is pre-trained and the teacher model is trained using the training data until the teacher model converges.

4. The end-to-end speech recognition model compression method based on cross distillation as claimed in claim 1, characterized in that In step 4), the student model is inferred. During inference, only the network parameters of the student model need to be loaded, and the teacher model does not need to be involved.

Images