Transform network model training method and device, electronic equipment and medium
Through the pruning and distillation technology of the Transformer network model, the operation efficiency of the model on low-computing terminal equipment is solved, the model accuracy is maintained, and the parameter quantity and calculation complexity are reduced, so that the Transformer network model can be low Run on computing equipment.
Patent Information
- Application Number
- CN202310637322.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-03
AI Technical Summary
With the increase of the number of layers and parameters of the Transformer network model, there are problems such as complex network structure, large computing volume and slow speed, and it is difficult to run on terminal devices with low computing power. How to reduce the number of parameters and calculation complexity while reducing it. Model accuracy loss is an urgent problem.
By obtaining the original Transformer network model, training to obtain the teacher model, and pruning and distilling it, the student model is obtained, pruning reduces the number of layers of the encoder and decoder, and the distillation method is used to determine the losses of the intermediate layer and the output layer based on the losses of the intermediate layer and the output layer by using the distillation method. The loss used during the training process of the student model achieves a reduction in the amount of parameters and calculation complexity.
Through the distillation method, the student model can better learn the knowledge of the teacher model, maintain the high accuracy of the student model, reduce the number of parameters and calculation complexity, and enable the Transformer network model to run on terminal devices with low computing power.
Smart Images

Figure CN119067169A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a training method, device, electronic device and medium for a Transformer network model. Background Art
[0002] Transformer is a sequence model based on the attention mechanism. Different from traditional recurrent neural networks (RNN) and convolutional neural networks (CNN), Transformer only uses the self-attention mechanism to process input sequences and output sequences, so it can be calculated in parallel, greatly improving the computational efficiency.
[0003] As the number of layers of the Transformer network model increases, the number of parameters increases, and the network structure becomes complex, the amount of computation is large, and the speed is slow, making it difficult to port to low-computing terminals (such as embedded devices). How to reduce the accuracy loss of the model while reducing the number of parameters and computational complexity of the Transformer model is an urgent problem to be solved. Summary of the invention
[0004] The embodiments of the present invention provide a training method, device, electronic device and medium for a Transformer network model, which are used to solve the problem of how to reduce the accuracy loss of the model while reducing the number of Transformer model parameters and computational complexity.
[0005] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:
[0006] In a first aspect, an embodiment of the present invention provides a training method for a Transformer network model, comprising:
[0007] Get the original Transformer network model;
[0008] Using the first training data set to train the original Transformer network model to obtain a teacher model;
[0009] Pruning the teacher model to obtain a student model;
[0010] The student model is trained using the teacher model and the second training data set, wherein the training of the student model using the teacher model and the second training data set includes: distilling the outputs of the intermediate layers and the final output layer of the teacher model, and determining the loss used in the process of training the student model based on the distillation loss of the intermediate layers and the final output layer.
[0011] Optionally, pruning the teacher model to obtain a student model includes:
[0012] The number of layers of the encoder and / or the number of layers of the decoder of the teacher model is reduced to obtain a student model.
[0013] Optionally, distilling the outputs of the intermediate layer and the final output layer of the teacher model, and determining the loss used in the process of training the student model according to the distillation loss of the intermediate layer and the final output layer, comprises:
[0014] Distilling the outputs of the intermediate layers and the final output layer of the teacher model using a two-stage distillation method, and determining the loss used in training the student model according to the distillation losses of the intermediate layers and the final output layer;
[0015] Wherein, the two-stage distillation method includes:
[0016] In the first stage, the output of the intermediate layer of the teacher model is distilled to obtain the distillation loss of the intermediate layer of the teacher model, and the loss used in the process of training the student model in the first stage is determined according to the distillation loss of the intermediate layer of the teacher model and the hard label loss of the teacher model;
[0017] In the second stage, the final output layer of the teacher model is distilled to obtain the distillation loss of the final output layer of the teacher model, and the distillation loss of the final output layer of the teacher model is used as the loss used in the process of training the student model in the second stage.
[0018] Optionally, distilling the outputs of the intermediate layer and the final output layer of the teacher model, and determining the loss used in the process of training the student model according to the distillation loss of the intermediate layer and the final output layer, comprises:
[0019] Distilling the outputs of the intermediate layer and the final output layer of the teacher model by joint learning distillation, and determining the loss used in the process of training the student model according to the distillation loss of the intermediate layer and the final output layer;
[0020] The joint learning distillation method includes:
[0021] Distilling the output of the middle layer of the teacher model to obtain a distillation loss of the middle layer of the teacher model;
[0022] Distilling the output of the last output layer of the teacher model to obtain the distillation loss of the last output layer of the teacher model;
[0023] The sum of the distillation loss of the intermediate layers of the teacher model and the distillation loss of the final output layer of the teacher model is used as the loss used in the process of training the student model.
[0024] Optionally, the adopting the teacher model and the second training data set to train the student model includes:
[0025] Inputting the second training data set into the teacher model to obtain a first output, dividing the first output by T, and calculating a normalized exponential function of the result after the division to obtain a soft target, wherein T is a temperature used in the distillation process;
[0026] Inputting the second training data set into the student model to obtain a second output, dividing the second output by T, and calculating a normalized exponential function of the result after the division, and determining a soft loss according to the calculated result and the soft target, wherein the soft loss includes a loss of a soft label of a CTC branch of the teacher model and a loss of a soft label of an attention branch of the teacher model;
[0027] Calculate a normalized exponential function of the second output, and determine a hard loss according to the calculated result and the hard target, wherein the hard loss includes a loss of a hard label of a CTC branch of the teacher model and a loss of a hard label of an attention branch of the teacher model;
[0028] A distillation loss of a final output layer of the teacher model is determined according to the soft loss and the hard loss.
[0029] Optionally, the distillation loss of the last output layer of the teacher model is determined by at least one of the following:
[0030] The hard label loss of the CTC branch of the teacher model;
[0031] The hard label loss of the attention branch of the teacher model;
[0032] The soft label loss of the CTC branch of the teacher model;
[0033] The soft label loss of the attention branch of the teacher model.
[0034] Optionally, the distillation loss of the last output layer of the teacher model is calculated as follows:
[0035] loss tail =λ*(1-α ctc_distill )*loss ctc_hard +(1-λ)*(1-α att_distill )*loss att_hard +λ*α ctc_distill *loss ctc_distill +(1-λ)*αatt_distill *loss att_distill
[0036] Among them, loss tail is the distillation loss of the last output layer of the teacher model, loss ctc_hard is the hard label loss of the CTC branch of the teacher model, loss att_hard is the hard label loss of the attention branch of the teacher model, and λ is the loss ctc_hard The weight, loss ctc_distill is the soft label loss of the CTC branch of the teacher model, loss att_distill is the soft label loss of the attention branch of the teacher model, α ctc_distill loss ctc_distill The weight of .
[0037] Optionally, distilling the output of the intermediate layer of the teacher model includes:
[0038] Selecting multiple target intermediate layers in the encoder or decoder of the teacher model, wherein the number of layers of the target intermediate layers is equal to the number of layers of the intermediate layers of the encoder or decoder of the student model, and the target intermediate layers correspond one to one;
[0039] The final output of the target intermediate layer and the output of the attention layer in the target intermediate layer are used as labels to perform distillation training on the corresponding intermediate layer of the encoder or decoder of the student model.
[0040] Optionally, the selecting of multiple target intermediate layers in the encoder or decoder of the teacher model includes:
[0041] For the encoder or decoder of the teacher model, a target intermediate layer is selected every K layers; wherein K=M / N, M is the total number of intermediate layers of the encoder or decoder of the teacher model, and N is the total number of intermediate layers of the encoder or decoder of the student model.
[0042] Optionally, the distillation loss of the intermediate layer of the teacher model is determined by at least one of the following:
[0043] The total number of intermediate layers of the encoder or decoder of the teacher model;
[0044] Parameters of the hidden layers in the intermediate layers;
[0045] Parameters of the attention layer in the intermediate layer.
[0046] Optionally, the calculation formula of the distillation loss of the intermediate layer of the teacher model is as follows:
[0047]
[0048] Among them, loss middle is the distillation loss of the intermediate layer of the encoder or decoder of the teacher model, M is the total number of intermediate layers of the encoder or decoder of the teacher model, i is the layer number of the intermediate layer of the encoder or decoder of the teacher model, H T is the parameter of the hidden layer in the middle layer of the encoder or decoder of the teacher model, H S is the parameter of the hidden layer in the middle layer of the encoder or decoder of the student model, Att T is the parameter of the attention layer in the middle layer of the encoder or decoder of the teacher model, Att S is the parameter of the attention layer in the intermediate layer of the encoder or decoder of the student model, K is the ratio of the total number of intermediate layers of the encoder or decoder of the teacher model to the total number of intermediate layers of the encoder or decoder of the student model, and MES is the mean square loss function.
[0049] Optionally, the method further includes:
[0050] The trained student model is used to infer the inference data to obtain an inference result.
[0051] Optionally, the Transformer network model is an end-to-end speech recognition model, and the trained student model is applied to at least one of the following scenarios:
[0052] Offline environment;
[0053] Private deployment environment.
[0054] In a second aspect, an embodiment of the present invention provides a training system for a Transformer network model, including:
[0055] The first acquisition module is used to obtain the original Transformer network model;
[0056] A first training module, used to train the original Transformer network model using a first training data set to obtain a teacher model;
[0057] A pruning module, used for pruning the teacher model to obtain a student model;
[0058] A second training module is used to train the student model using the teacher model and the second training data set, wherein the training of the student model using the teacher model and the second training data set includes: distilling the outputs of the intermediate layers and the final output layers of the teacher model, and determining the loss used in the process of training the student model based on the distillation losses of the intermediate layers and the final output layers.
[0059] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the training method of the Transformer network model as described in the first aspect above are implemented.
[0060] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the training method of the Transformer network model as described in the first aspect above are implemented.
[0061] In an embodiment of the present invention, the Transformer network model is trained to obtain a teacher model, and the teacher model is pruned and distilled to obtain a student model with smaller parameters and lower computational complexity. Through distillation, the student model can better learn the knowledge of the teacher model, and the accuracy of the student model is comparable to that of the teacher model, making it possible for the Transformer network model to run on low-computing-power terminals (such as embedded devices). BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0063] Figure 1 A flowchart of a training method for a Transformer network model according to an embodiment of the present invention;
[0064] Figure 2 A schematic diagram of the structure of a Transformer network model according to an embodiment of the present invention;
[0065] Figure 3 A schematic diagram of the structure of the intermediate layer of the encoder and decoder according to an embodiment of the present invention;
[0066] Figure 4A schematic diagram of the loss of the hidden layer and the attention layer in the middle layer of the encoder of the student model of an embodiment of the present invention;
[0067] Figure 5 A schematic diagram of selecting multiple target intermediate layers in an encoder or decoder of a teacher model according to an embodiment of the present invention;
[0068] Figure 6 A schematic diagram of the structure of a training system for a Transformer network model according to an embodiment of the present invention;
[0069] Figure 7 FIG. 4 is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0071] Please refer to Figure 1 , an embodiment of the present invention provides a training method for a Transformer network model, comprising:
[0072] Step 11: Get the original Transformer network model;
[0073] Step 12: Using the first training data set to train the original Transformer network model to obtain a teacher model;
[0074] In the embodiment of the present invention, the first training data set has hard labels, and the original Transformer network model is trained by the hard labels.
[0075] Step 13: Prune the teacher model to obtain a student model;
[0076] Step 14: Use the teacher model and the second training data set to train the student model, wherein the use of the teacher model and the second training data set to train the student model includes: distilling the outputs of the intermediate layers and the final output layers of the teacher model, and determining the loss used in the process of training the student model based on the distillation loss of the intermediate layers and the final output layers.
[0077] The intermediate layer includes: a Transformer block or a conformer block of the encoder of the teacher model, and / or a Transformer block or a conformer block of the decoder of the teacher model.
[0078] In an embodiment of the present invention, the Transformer network model is trained to obtain a teacher model, and the teacher model is pruned and distilled to obtain a student model with smaller parameters and lower computational complexity. Through distillation, the student model can better learn the knowledge of the teacher model, and the accuracy of the student model is comparable to that of the teacher model, making it possible for the Transformer network model to run on low-computing-power terminals (such as embedded devices).
[0079] The structure of the Transformer network model of the example of the present invention is described below.
[0080] The Transformer network model in the example of the present invention can be an end-to-end speech recognition model, or a text recognition model, etc.
[0081] Traditional speech recognition models are generally divided into acoustic models and language models. The acoustic model is responsible for converting audio sequences into phoneme sequences, such as Chinese pinyin and English phonetic symbols, while the language model is responsible for converting these phoneme sequences into text sequences. The acoustic model and language model do not need to be coupled during training and can be trained independently. The disadvantage of traditional speech recognition models is that they require a pronunciation dictionary and phoneme annotations. The end-to-end speech recognition model overcomes this problem and directly converts audio sequences into text sequences, greatly simplifying the architecture of the speech recognition model.
[0082] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of a Transformer network model of an embodiment of the present invention. In this embodiment, the Transformer network model is taken as an end-to-end speech recognition model. In addition to the end-to-end speech recognition model, the structure of other Transformer network models is similar. The Transformer network model includes a front-end feature extractor, a Transformer encoder (encoder), a Transformer decoder (decoder) (also called an attention branch) and a CTC (Convolutional Turbo Code) branch (also called a CTC decoder), wherein:
[0083] The front-end feature extractor generally consists of several convolutional layers. Its purpose is to downsample in the time dimension, thereby reducing computational overhead;
[0084] The Transformer encoder part is composed of N Transformer encoder blocks (also called Transformer blocks or Transformer blocks), and can also be composed of various variant structures based on Transformer, such as conformer blocks (also called conformer blocks); in the description of the following embodiments, only the conformer block is used as an example. It should be noted that the conformer block can also be replaced by a conformer block.
[0085] The Transformer decoder is composed of M Transformer decoder blocks (also called Transformer blocks or Transformer blocks), and can also be composed of various Transformer variants, such as the Conformer block. Figure 3 The difference between the Transformer decoder and the Transformer encoder is that the Transformer decoder adds a multi-head cross attention layer to obtain the feature encoding obtained by the Transformer encoder, masks the position to be predicted in the multi-head self-attention, and uses autoregressive decoding. The output of the Transformer decoder at the previous time point will be used as the input at the next time point.
[0086] CTC branch: Its structure is usually a fully connected layer, which converts the features of the Transformer encoder into the probability of each word and constrains it with CTC loss.
[0087] The modules of the Transformer network model are not all necessary, such as the front-end feature extractor, but they can reduce computational overhead. Secondly, the CTC branch and the Transformer decoder (also called the attention branch) can actually achieve the basic function of speech recognition with only one of them. However, experiments have shown that training the model with the two losses of these two branches together can achieve better results. For example, the weighted sum of the two is usually used during training:
[0088] loss hard =λ*loss ctc +(1-λ)*loss att
[0089] Among them, loss hard The loss used to train the Transformer network model. λ represents the weight value, which ranges from (0 to 1). ctc is the loss of the CTC branch (also called CTC loss), loss att is the loss of the Transformer decoder part (such as Figure 2 It should be noted that although both branches are used when training the model, only one of the branches may be used during inference (i.e. when the model is used), or the results of the two branches may be combined for weighted output.
[0090] Pruning is one of the methods to improve model efficiency. It can efficiently generate models with smaller models, higher memory utilization, and faster forward speed. In convolutional neural networks, pruning can specifically include two types: pruning based on amplitude and pruning based on channel contribution. In the method of pruning based on amplitude, a certain threshold is taken. If the amplitude of a certain weight is lower than the threshold, the weight corresponding to the amplitude is pruned. If you want to prune more, just set the threshold larger. Pruning based on channel contribution is to achieve the purpose of trimming the model size and speeding up the model inference speed by trimming the number of channels in the convolutional neural network. With the application of Transformers networks in natural language processing (NLP), computer vision (CV), and speech fields, people began to consider pruning different components in the Transformer structure. In an embodiment of the present invention, optionally, the pruning of the teacher model to obtain the student model includes: reducing the number of layers of the encoder (i.e., Transformer encoder) and / or the number of layers of the decoder (i.e., Transformer decoder, all decoders mentioned below are Transformer decoders) of the teacher model to obtain the student model. In an embodiment of the present invention, the number of layers of the encoder refers to the number of Transformer encoder blocks (also referred to as Transformer blocks, or Transformer blocks) in the encoder, and the number of layers of the decoder refers to the number of Transformer decoder blocks (also referred to as Transformer blocks, or Transformer blocks) in the decoder. In an embodiment of the present invention, the number of parameters and computational complexity of the model can be quickly reduced by a coarse-grained layer reduction method, and then the accuracy of the pruned model can be improved as much as possible by using model distillation, so that the accuracy loss of the pruned model is minimized. At the same time, the coarse-grained layer pruning scheme has greater flexibility and convenience, so that the lightweight layer of the model can be quickly adjusted according to the performance of the terminal hardware.
[0091] In the above embodiment, pruning the teacher model refers to removing one or more Transformer blocks (including MHA (multi-head attention) and FFN (feed-forward network) layers) of the encoder and / or decoder. In some other embodiments of the present invention, it may also be removing certain heads of the multi-head attention in the Transformer block, removing certain layers of the feed-forward (FFN) layer, etc., and the present invention is not limited to this.
[0092] In the following embodiments, an end-to-end speech recognition model with a conformer as the backbone network is taken as an example. We call the model obtained after training the original Transformer network model a teacher model, and the model after layer pruning is a student model. The specific configurations of the teacher model and the student model are shown in Table 1:
[0093]
[0094]
[0095] Table 1
[0096] That is, train the original Transformer network model to obtain the teacher model, then reduce the number of encoder layers of the teacher model from 12 to 3, and the number of decoder layers from 6 to 3 or 2, and then use the distillation method to let the student model fit the knowledge of the teacher model to improve the model effect.
[0097] The four technologies for model compression and acceleration are designing efficient small networks, pruning, quantization, and distillation. Distillation is the knowledge distillation, which transfers the knowledge of the teacher model (also called the teacher network) to the student model (also called the student network), so that the performance of the student model is the same as that of the teacher model, so that the lightweight student model can be deployed on low-computing terminals such as embedded devices.
[0098] Model distillation is generally divided into the following steps: the first step is to train a teacher model using large-scale training data; the second step is to use large-scale training data and the trained teacher model to train a student model. The loss function L for training the student model consists of two parts: one is to calculate the distillation loss / KL divergence (KL divergence is a metric used to measure the similarity of two probability distributions) using the output logits of the teacher and student networks (the output of the final fully connected layer), and the other is to directly calculate the cross entropy loss using the output of the student model and the labels of the training data:
[0099] L=αL soft +(1-α)*L hard
[0100] Among them, α is the weight, L soft is the distillation loss / KL divergence calculated using the output logits of the teacher and student networks. It can also be the relative entropy KL Dvi Loss function, MES Loss function, or Cosine Embedding Loss function. L hardis a cross entropy loss function calculated directly using the output of the student model and the labels of the training data.
[0101] It should be noted that when calculating the loss of the logits output by the teacher model and the student model, the softmax (normalized exponential function) result output by the model is divided by T:
[0102]
[0103]
[0104] in, is the value of the softmax output of the teacher model at temperature = T on the i-th class, is the value of the softmax output of the student model on the i-th class at temperature = T. The exp function is an exponential function with the natural constant e as the base. Here T is a hyperparameter in knowledge distillation, called temperature. z i is the logits of the student model, v i It is the logits of the teacher model. In order to make the probabilities of each category output by logits smoother and more evenly distributed, the student model can learn some information about categories with smaller probabilities during training.
[0105] The above L soft The calculation formula is as follows:
[0106]
[0107] Among them, p is the output result of the teacher model, and then the output result p of the teacher model is used as the target of the student model, so that the output result q of the student model is as close to p as possible, and N is the total number of labels.
[0108] In an embodiment of the present invention, two methods can be used to distill the outputs of the intermediate layer and the final output layer of the teacher model, wherein one method is a two-stage distillation method, and the other method is a joint learning distillation method, which are introduced below respectively.
[0109] It should be noted that the above two distillation methods involve three types of distillation:
[0110] 1) Directly distill the output of the last output layer of the teacher model, that is, the knowledge of the last output layer;
[0111] 2) Distill the knowledge of hidden layers in the middle layers of the encoding layer and / or decoding layer;
[0112] 3) Distill the knowledge of the attention layers in the intermediate layers of the encoding and / or decoding layers.
[0113] In some embodiments of the present invention, optionally, distilling the outputs of the intermediate layer and the final output layer of the teacher model, and determining the loss used in the process of training the student model according to the distillation loss of the intermediate layer and the final output layer, includes:
[0114] Distilling the outputs of the intermediate layers and the final output layer of the teacher model using a two-stage distillation method, and determining the loss used in training the student model according to the distillation losses of the intermediate layers and the final output layer;
[0115] Wherein, the two-stage distillation method includes:
[0116] In the first stage (Stage one), the output of the intermediate layer of the teacher model is distilled to obtain the distillation loss of the intermediate layer of the teacher model. middle ), and according to the distillation loss of the intermediate layer of the teacher model and the hard label loss of the teacher model (loss hard ), determine the loss used in the first stage of training the student model; that is, using loss middle and loss hard To make the student model fit the knowledge of the intermediate layer of the teacher model;
[0117] Optionally, the loss used in the first stage of training the student model is loss stage1 Use the following formula to determine: loss stage1 =loss middle +loss hard .
[0118] In the second stage (Stage two), the final output layer (i.e., the final classification layer) of the teacher model is distilled to obtain the distillation loss of the final output layer of the teacher model. tail ), the distillation loss of the final output layer of the teacher model is used as the loss used in the second stage of training the student model. tail To fine-tune the output of the final classification layer of the student model.
[0119] That is, the loss loss used in the second stage of training the student model stage2 Use the following formula to determine: loss stage2 =loss tail .
[0120] In some other embodiments of the present invention, optionally, distilling the outputs of the intermediate layer and the final output layer of the teacher model, and determining the loss used in the process of training the student model according to the distillation loss of the intermediate layer and the final output layer, includes:
[0121] Distilling the outputs of the intermediate layer and the final output layer of the teacher model by joint learning distillation, and determining the loss used in the process of training the student model according to the distillation loss of the intermediate layer and the final output layer;
[0122] The joint learning distillation method includes: distilling the output of the intermediate layer of the teacher model to obtain the distillation loss of the intermediate layer of the teacher model. middle ), distill the output of the final output layer of the teacher model to obtain the distillation loss of the final output layer of the teacher model tail ), the sum of the distillation loss of the intermediate layer of the teacher model and the distillation loss of the final output layer of the teacher model is used as the loss used in the process of training the student model. middle and loss tail The student model is made to fit the knowledge of the intermediate layers and the final output layers of the teacher model at the same time, and the teacher network is also corrected and adjusted according to the hard target of the output.
[0123] Optionally, the loss used in the process of training the student model is determined by the following formula: loss = loss middle +loss tail .
[0124] The following describes a method for determining the distillation loss of the intermediate layer of the teacher model and the distillation loss of the final output layer of the teacher model.
[0125] (1) Distill the final output layer of the teacher model
[0126] The loss function for training the student model consists of two parts: one is to calculate the distillation loss / KL divergence using the output logits of the teacher model and the student model, and the other is to directly calculate the cross entropy loss using the output of the student model and the labels of the training data.
[0127] In some embodiments, the Transformer network model adopts a joint learning method based on ctc loss and attention loss (i.e., Transformer decoder loss). Optionally, the distillation loss of the last output layer of the teacher model is determined by at least one of the following:
[0128] The hard label loss of the CTC branch of the teacher model;
[0129] The hard label loss of the attention branch of the teacher model;
[0130] The soft label loss of the CTC branch of the teacher model;
[0131] The soft label loss of the attention branch of the teacher model.
[0132] In the embodiment of the present invention, optionally, the calculation formula of the distillation loss of the last output layer of the teacher model is as follows:
[0133] loss tail =λ*(1-α ctc_distill )*loss ctc_hard +(1-λ)*(1-α att_distill )*loss att_hard +λ*α ctc_distill *loss ctc_distill +(1-λ)*α att_distill *loss att_distill
[0134] Among them, loss tail is the distillation loss of the last output layer of the teacher model, loss ctc_hard is the hard label loss of the CTC branch of the teacher model, loss att_hard is the hard label loss of the attention branch of the teacher model, and λ is the loss ctc_hard The weight, loss ctc_distill is the soft label loss of the CTC branch of the teacher model, loss att_distill is the soft label loss of the attention branch of the teacher model, α ctc_distill loss ctc_distill The weight of .
[0135] In the embodiment of the present invention, optionally, the adopting the teacher model and the second training data set to train the student model includes:
[0136] 1) inputting the second training data set into the teacher model to obtain a first output (logits), dividing the first output by T (i.e., softening), and calculating a normalized exponential function (softmax) of the result after the division to obtain a soft target (soft target), wherein T is the temperature used in the distillation process;
[0137] In an embodiment of the present invention, the second training data set may be the same as or different from the first training data set used to train the teacher model.
[0138] In the embodiment of the present invention, the teacher model only participates in the forward transfer.
[0139] 2) Inputting the second training data set into the student model to obtain a second output (logits), dividing the second output by T (i.e., softening), and calculating the normalized exponential function (softmax) of the result after the division, and determining the soft loss (distillation loss) according to the calculated result and the soft target (soft target), wherein the soft loss includes the loss of the soft label of the CTC branch of the teacher model (loss ctc_distill ) and the loss of the soft label of the attention branch of the teacher model (loss att_distill );
[0140] Optionally, determining the soft loss (distillation loss) according to the calculated result and the soft target (soft target) includes: performing KL divergence calculation on the calculated result and the soft target to obtain the soft loss (distillation loss).
[0141] 3) Calculate the normalized exponential function (softmax) of the second output (logits), and determine the hard loss based on the calculated result and the hard target (hard target), wherein the hard loss includes the loss of the hard label of the CTC branch of the teacher model (loss ctc_hard ) and the loss of the hard label of the attention branch of the teacher model (loss att_hard );
[0142] Optionally, determining the hard loss according to the calculated result and a hard target includes: performing cross entropy calculation on the calculated result and the hard target to obtain the hard loss.
[0143] 4) determining a distillation loss of a final output layer of the teacher model according to the soft loss and the hard loss.
[0144] Optionally, the distillation loss of the last output layer of the teacher model is calculated as follows:
[0145]
[0146] Generally speaking, the soft loss (distillation loss) has a larger weight, and the hard loss (hard loss) has a smaller weight, so that a better distillation effect can be achieved. Distillation loss means that the student model should try to learn the generalization ability of the teacher model; hard loss means that the student model also needs to learn to generate correct labels. In other words, the student model must learn the generalization ability of the teacher model and also make error correction adjustments to the teacher model based on the output hard target.
[0147] (2) Distilling the intermediate layers of the teacher model
[0148] Based on the architecture of the Transformer network model, the encoder and decoder parts are both composed of multiple layers of Transformer blocks (or conformer blocks). Therefore, in the embodiment of the present invention, it is proposed to use layered distillation to distill the knowledge of the middle layer (Transformer block or conformer block) of the teacher model, so that the middle layer network of the student model can better fit the middle layer network of the teacher model, that is, to extract knowledge from the middle layer of the teacher model, and avoid the phenomenon of fitting too fast when distilling the final output layer (with the risk of overfitting), so as to achieve better results. In the embodiment of the present invention, optionally, the output of the hidden layer (hidden layer) and / or attention layer (attention layer) in the middle layer of the encoder and / or decoder of the teacher model is distilled.
[0149] Please refer to Figure 4 , Figure 4 Schematic diagram of the loss of the hidden layer and attention layer of the middle layer (conformer block in this embodiment) of the encoder of the student model. The loss of the hidden layer refers to the loss of the final output of the middle layer.
[0150] In the embodiment of the present invention, optionally, the distilling the output of the intermediate layer of the teacher model includes:
[0151] Selecting multiple target intermediate layers in the encoder or decoder of the teacher model, wherein the number of layers of the target intermediate layers is equal to the number of layers of the intermediate layers of the encoder or decoder of the student model, and the target intermediate layers correspond one to one;
[0152] The final output of the target intermediate layer and the output of the attention layer in the target intermediate layer are used as labels to perform distillation training on the corresponding intermediate layer of the encoder or decoder of the student model.
[0153] It should be noted that, because the number of layers of the Transformer block of the teacher model is generally more than that of the student model, a layered corresponding distillation method can be adopted. In the correspondence between the Transformer block of the teacher model and the Transformerblock of the student model, a K-skip method is adopted, that is, the multiple target intermediate layers in the encoder or decoder of the teacher model are selected, including: for the encoder or decoder of the teacher model, one target intermediate layer is selected every K layers; wherein, K=M / N, M is the total number of intermediate layers of the encoder or decoder of the teacher model, and N is the total number of intermediate layers of the encoder or decoder of the student model.
[0154] For example, see Figure 5 , the output of Multi-Head Attention in multiple conformer blocks of the encoder of the teacher model (i.e., the output of the attention layer) and the final output of the conformer block (i.e., the output of the hidden layer) can be used as labels. For example, the output of Multi-Head Attention in the 3rd, 7th, and 11th layers of the conformer block of the teacher model and the final output of the conformer block are used as labels to perform distillation training on the corresponding 1st, 2nd, and 3rd layers of the conformer block in the student model, so that the conformer block of the student model can fit the conformer block of the teacher model as much as possible.
[0155] In the embodiment of the present invention, optionally, the distillation loss of the intermediate layer of the teacher model is determined by at least one of the following:
[0156] The total number of intermediate layers of the encoder or decoder of the teacher model;
[0157] Parameters of the hidden layers in the intermediate layers;
[0158] Parameters of the attention layer in the intermediate layer.
[0159] In the embodiment of the present invention, optionally, the calculation formula of the distillation loss of the intermediate layer of the teacher model is as follows:
[0160]
[0161] Among them, loss middle is the distillation loss of the intermediate layer of the encoder or decoder of the teacher model, M is the total number of intermediate layers of the encoder or decoder of the teacher model, i is the layer number of the intermediate layer of the encoder or decoder of the teacher model, H Tis the parameter of the hidden layer in the middle layer of the encoder or decoder of the teacher model, H S is the parameter of the hidden layer in the middle layer of the encoder or decoder of the student model, Att T is the parameter of the attention layer in the middle layer of the encoder or decoder of the teacher model, Att S is the parameter of the attention layer in the intermediate layer of the encoder or decoder of the student model, K is the ratio of the total number of intermediate layers of the encoder or decoder of the teacher model to the total number of intermediate layers of the encoder or decoder of the student model, and MES is the mean square loss function.
[0162] Please refer to Figure 4 That is, the adopting the teacher model and the second training data set to train the student model includes:
[0163] For the encoder or decoder of the teacher model, an intermediate layer is selected every K layers ( Figure 4 In the embodiment shown, K=4);
[0164] For the encoder or decoder of the student model, selecting the intermediate layers in order;
[0165] Determining a distillation loss of a selected intermediate layer of the teacher model and a selected intermediate layer of the student model;
[0166] Among them, K=M / N, M is the total number of intermediate layers of the encoder or decoder of the teacher model, and N is the total number of intermediate layers of the encoder or decoder of the student model.
[0167] The teacher model and student model trained by the training method of the Transformer network model in the embodiment of the present invention and the teacher model and student model trained by the existing Transformer network model training method are compared to obtain the comparison results shown in Tables 2 to 5 below.
[0168] Table 2 Effects of the teacher model and pruned student model on the test set
[0169]
[0170]
[0171] Table 3 Effect of student model under the soft label method of direct distillation of teacher model output in the prior art
[0172]
[0173] Table 4 Effect of the student model in the joint learning distillation method of the embodiment of the present invention
[0174] Model T <![CDATA[α ctc_distill ]]> <![CDATA[α att_distill ]]> CER(%) Student model 1 1 0.3 0.3 5.43 Student model 2 1 0.3 0.3 5.51
[0175] Table 5 Effect of the student model under the two-stage distillation method of the embodiment of the present invention
[0176]
[0177] Among them, Table 2 gives the test results of the teacher model and the pruned student model under the same data set. The CER (char error rate) in the table is the word error rate. The lower the index, the higher the recognition rate of the model.
[0178] Table 3 shows the existing distillation scheme, that is, the word error rate of the student model when the soft label of the teacher model is directly distilled. The temperature coefficient T=5 selected during the experiment, the loss weight coefficient of the soft label part of the CTC branch is 0.5, and the loss weight coefficient of the soft label part of the attention branch is 0.5. The results show that the word error rate of student model1 is reduced from the original 5.87% to 5.55%, and the word error rate of studentmodel2 is reduced from the original 5.98% to 5.76%.
[0179] Table 4 shows the word error rates of the student model under two configurations under the joint learning distillation scheme of the embodiment of the present invention. The temperature coefficient T=1 selected during the experiment, the loss weight coefficient of the soft label part of the CTC branch is 0.3, and the loss weight coefficient of the soft label part of the attention branch is 0.3. The results show that the word error rate of student model1 is reduced from the original 5.87% to 5.43%, and the word error rate of student model2 is reduced from the original 5.98% to 5.51%, which proves the effectiveness of the joint learning distillation scheme proposed in the embodiment of the present invention, and the joint learning distillation scheme proposed in the embodiment of the present invention has a certain improvement over the existing distillation scheme.
[0180] Table 5 shows the word error rates of the student model under two configurations under the two-stage distillation scheme proposed in the embodiment of the present invention. The temperature coefficient T=1, the loss weight coefficient of the soft label part of the CTC branch, and the loss weight coefficient of the soft label part of the attention branch are 0.3. The results show that the word error rate of student model 1 is reduced from the original 5.87% to 5.24%, and the word error rate of student model 2 is reduced from the original 5.98% to 5.43%, which proves the effectiveness of the two-stage distillation scheme proposed in the embodiment of the present invention, and the two-stage distillation scheme proposed in the embodiment of the present invention has great advantages over the existing distillation scheme.
[0181] In the embodiment of the present invention, optionally, the method further comprises: using the trained student model to perform reasoning on the data to be reasoned to obtain a reasoning result. Optionally, during reasoning, the temperature T is set to 1.
[0182] In the embodiment of the present invention, optionally, the Transformer network model is an end-to-end speech recognition model, and the trained student model is applied to at least one of the following scenarios:
[0183] (1) Offline environment;
[0184] The ultimate goal of lightweight speech recognition is to facilitate the operation of the model on offline terminals. The reason why offline speech recognition modules are widely used is that this offline speech device system can perform speech recognition without a network, and the information extraction sensitivity will be higher. Without network control, there will be no interruption of the recognition system. Therefore, this product can be used in any environment. It is most widely used in the field of smart homes. Basically, offline speech recognition systems are used in smart home products, such as smart TVs, refrigerators and lighting systems used in people's daily lives.
[0185] (2) Private deployment environment
[0186] For example, the private deployment of voice recognition algorithms on various terminal devices at bank branches can prevent the spread of user privacy information on the Internet. The more common ones are voice quality inspection, etc.
[0187] Please refer to Figure 6 The embodiment of the present invention further provides a training system 60 for a Transformer network model, comprising:
[0188] A first acquisition module 61 is used to acquire an original Transformer network model;
[0189] A first training module 62 is used to train the original Transformer network model using a first training data set to obtain a teacher model;
[0190] A pruning module 63, used for pruning the teacher model to obtain a student model;
[0191] The second training module 64 is used to train the student model using the teacher model and the second training data set, wherein the training of the student model using the teacher model and the second training data set includes: distilling the outputs of the intermediate layer and the final output layer of the teacher model, and determining the loss used in the process of training the student model based on the distillation loss of the intermediate layer and the final output layer.
[0192] In an embodiment of the present invention, the Transformer network model is trained to obtain a teacher model, and the teacher model is pruned and distilled to obtain a student model with smaller parameters and lower computational complexity. Through distillation, the student model can better learn the knowledge of the teacher model, and the accuracy of the student model is comparable to that of the teacher model, making it possible for the Transformer network model to run on low-computing-power terminals (such as embedded devices).
[0193] Optionally, the pruning module 63 is used to reduce the number of layers of the encoder and / or the number of layers of the decoder of the teacher model to obtain a student model.
[0194] Optionally, the second training module 64 is used to distill the outputs of the intermediate layer and the final output layer of the teacher model using a two-stage distillation method, and determine the loss used in the process of training the student model according to the distillation loss of the intermediate layer and the final output layer;
[0195] Wherein, the two-stage distillation method includes:
[0196] In the first stage, the output of the intermediate layer of the teacher model is distilled to obtain the distillation loss of the intermediate layer of the teacher model, and the loss used in the process of training the student model in the first stage is determined according to the distillation loss of the intermediate layer of the teacher model and the hard label loss of the teacher model;
[0197] In the second stage, the final output layer of the teacher model is distilled to obtain the distillation loss of the final output layer of the teacher model, and the distillation loss of the final output layer of the teacher model is used as the loss used in the process of training the student model in the second stage.
[0198] Optionally, the second training module 64 is used to distill the outputs of the intermediate layer and the final output layer of the teacher model by using a joint learning distillation method, and determine the loss used in the process of training the student model according to the distillation loss of the intermediate layer and the final output layer;
[0199] The joint learning distillation method includes:
[0200] Distilling the output of the middle layer of the teacher model to obtain a distillation loss of the middle layer of the teacher model;
[0201] Distilling the output of the last output layer of the teacher model to obtain the distillation loss of the last output layer of the teacher model;
[0202] The sum of the distillation loss of the intermediate layers of the teacher model and the distillation loss of the final output layer of the teacher model is used as the loss used in the process of training the student model.
[0203] Optionally, the second training module 64 is used to input the second training data set into the teacher model to obtain a first output, divide the first output by T, and calculate the normalized exponential function of the result after the division to obtain a soft target, wherein T is the temperature used in the distillation process; input the second training data set into the student model to obtain a second output, divide the second output by T, and calculate the normalized exponential function of the result after the division, and determine the soft loss based on the calculated result and the soft target, wherein the soft loss includes the loss of the soft label of the CTC branch of the teacher model and the loss of the soft label of the attention branch of the teacher model; calculate the normalized exponential function of the second output, and determine the hard loss based on the calculated result and the hard target, wherein the hard loss includes the loss of the hard label of the CTC branch of the teacher model and the loss of the hard label of the attention branch of the teacher model; determine the distillation loss of the last output layer of the teacher model based on the soft loss and the hard loss.
[0204] Optionally, the distillation loss of the last output layer of the teacher model is determined by at least one of the following:
[0205] The hard label loss of the CTC branch of the teacher model;
[0206] The hard label loss of the attention branch of the teacher model;
[0207] The soft label loss of the CTC branch of the teacher model;
[0208] The soft label loss of the attention branch of the teacher model.
[0209] Optionally, the distillation loss of the last output layer of the teacher model is calculated as follows:
[0210] loss tail =λ*(1-α ctc_distill )*loss ctc_hard +(1-λ)*(1-α att_distill )*loss att_hard +λ*α ctc_distill *loss ctc_distill +(1-λ)*α att_distill *loss att_distill
[0211] Among them, loss tail is the distillation loss of the last output layer of the teacher model, loss ctc_hard is the hard label loss of the CTC branch of the teacher model, loss att_hard is the hard label loss of the attention branch of the teacher model, and λ is the loss ctc_hard The weight, loss ctc_distill is the soft label loss of the CTC branch of the teacher model, loss att_distill is the soft label loss of the attention branch of the teacher model, α ctc_distill loss ctc_distill The weight of .
[0212] Optionally, the second training module 64 is used to select multiple target intermediate layers in the encoder or decoder of the teacher model, and the number of layers of the target intermediate layers is equal to the number of layers of the intermediate layers of the encoder or decoder of the student model, and corresponds one to one; the final output of the target intermediate layer and the output of the attention layer in the target intermediate layer are used as labels to perform distillation training on the corresponding intermediate layers of the encoder or decoder of the student model.
[0213] Optionally, the selecting of multiple target intermediate layers in the encoder or decoder of the teacher model includes: for the encoder or decoder of the teacher model, selecting a target intermediate layer every K layers; wherein K=M / N, M is the total number of intermediate layers of the encoder or decoder of the teacher model, and N is the total number of intermediate layers of the encoder or decoder of the student model.
[0214] Optionally, the distillation loss of the intermediate layer of the teacher model is determined by at least one of the following:
[0215] The total number of intermediate layers of the encoder or decoder of the teacher model;
[0216] Parameters of the hidden layers in the intermediate layers;
[0217] Parameters of the attention layer in the intermediate layer.
[0218] Optionally, the calculation formula of the distillation loss of the intermediate layer of the teacher model is as follows:
[0219]
[0220] Among them, loss middle is the distillation loss of the intermediate layer of the encoder or decoder of the teacher model, M is the total number of intermediate layers of the encoder or decoder of the teacher model, i is the layer number of the intermediate layer of the encoder or decoder of the teacher model, H T is the parameter of the hidden layer in the middle layer of the encoder or decoder of the teacher model, H S is the parameter of the hidden layer in the middle layer of the encoder or decoder of the student model, Att T is the parameter of the attention layer in the middle layer of the encoder or decoder of the teacher model, Att S is the parameter of the attention layer in the intermediate layer of the encoder or decoder of the student model, K is the ratio of the total number of intermediate layers of the encoder or decoder of the teacher model to the total number of intermediate layers of the encoder or decoder of the student model, and MES is the mean square loss function.
[0221] Optionally, the training system 60 of the Transformer network model further includes:
[0222] The reasoning module is used to use the trained student model to reason on the reasoning data to obtain the reasoning result.
[0223] Optionally, the Transformer network model is an end-to-end speech recognition model, and the trained student model is applied to at least one of the following scenarios:
[0224] Offline environment;
[0225] Private deployment environment.
[0226] Please refer to Figure 7 An embodiment of the present invention further provides an electronic device 70, including a processor 71, a memory 72, and a computer program stored in the memory 72 and executable on the processor 71. When the computer program is executed by the processor 71, the various processes of the training method embodiment of the above-mentioned Transformer network model are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0227] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the training method embodiment of the above-mentioned Transformer network model is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0228] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0229] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0230] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.
Claims
1. A training method for a Transformer network model, It is characterized in that include: Get the original Transformer network model; Using the first training data set to train the original Transformer network model to obtain a teacher model; Pruning the teacher model to obtain a student model; The student model is trained using the teacher model and the second training data set, wherein the training of the student model using the teacher model and the second training data set includes: distilling the outputs of the intermediate layers and the final output layer of the teacher model, and determining the loss used in the process of training the student model based on the distillation loss of the intermediate layers and the final output layer.
2. The method according to claim 1, It is characterized in that The step of pruning the teacher model to obtain a student model includes: The number of layers of the encoder and / or the number of layers of the decoder of the teacher model is reduced to obtain a student model.
3. The method according to claim 1, It is characterized in that The step of distilling the outputs of the intermediate layer and the final output layer of the teacher model, and determining the loss used in the process of training the student model according to the distillation loss of the intermediate layer and the final output layer, comprises: Distilling the outputs of the intermediate layers and the final output layer of the teacher model using a two-stage distillation method, and determining the loss used in training the student model according to the distillation losses of the intermediate layers and the final output layer; Wherein, the two-stage distillation method includes: In the first stage, the output of the intermediate layer of the teacher model is distilled to obtain the distillation loss of the intermediate layer of the teacher model, and the loss used in the process of training the student model in the first stage is determined according to the distillation loss of the intermediate layer of the teacher model and the hard label loss of the teacher model; In the second stage, the final output layer of the teacher model is distilled to obtain the distillation loss of the final output layer of the teacher model, and the distillation loss of the final output layer of the teacher model is used as the loss used in the process of training the student model in the second stage.
4. The method according to claim 1, It is characterized in that The step of distilling the outputs of the intermediate layer and the final output layer of the teacher model, and determining the loss used in the process of training the student model according to the distillation loss of the intermediate layer and the final output layer, comprises: Distilling the outputs of the intermediate layer and the final output layer of the teacher model by joint learning distillation, and determining the loss used in the process of training the student model according to the distillation loss of the intermediate layer and the final output layer; The joint learning distillation method includes: Distilling the output of the middle layer of the teacher model to obtain a distillation loss of the middle layer of the teacher model; Distilling the output of the last output layer of the teacher model to obtain the distillation loss of the last output layer of the teacher model; The sum of the distillation loss of the intermediate layers of the teacher model and the distillation loss of the final output layer of the teacher model is used as the loss used in the process of training the student model.
5. The method according to claim 3 or 4, It is characterized in that The adopting the teacher model and the second training data set to train the student model comprises: Inputting the second training data set into the teacher model to obtain a first output, dividing the first output by T, and calculating a normalized exponential function of the result after the division to obtain a soft target, wherein T is a temperature used in the distillation process; Inputting the second training data set into the student model to obtain a second output, dividing the second output by T, and calculating a normalized exponential function of the result after the division, and determining a soft loss according to the calculated result and the soft target, wherein the soft loss includes a loss of a soft label of a CTC branch of the teacher model and a loss of a soft label of an attention branch of the teacher model; Calculate a normalized exponential function of the second output, and determine a hard loss according to the calculated result and the hard target, wherein the hard loss includes a loss of a hard label of a CTC branch of the teacher model and a loss of a hard label of an attention branch of the teacher model; A distillation loss of a final output layer of the teacher model is determined according to the soft loss and the hard loss.
6. The method according to claim 3 or 4, It is characterized in that The distillation loss of the last output layer of the teacher model is determined by at least one of the following: The hard label loss of the CTC branch of the teacher model; The hard label loss of the attention branch of the teacher model; The soft label loss of the CTC branch of the teacher model; The soft label loss of the attention branch of the teacher model.
7. The method according to claim 6, It is characterized in that The calculation formula for the distillation loss of the last output layer of the teacher model is as follows: loss tail =λ*(1-a ctc_distill )*loss ctc_hard +(1-λ)*(1-α att_distill )*loss att_hard +l*a ctc_distill *loss ctc_distill +(1-λ)*a att_distill *loss att_distill Among them, loss tail is the distillation loss of the last output layer of the teacher model, loss ctc_hard is the hard label loss of the CTC branch of the teacher model, loss att_hard is the hard label loss of the attention branch of the teacher model, and λ is the loss ctc_hard The weight, loss ctc_distill is the soft label loss of the CTC branch of the teacher model, loss att_distill is the soft label loss of the attention branch of the teacher model, α ctc_distill Loss ctc_distill The weight of .
8. The method according to claim 3 or 4, It is characterized in that The step of distilling the output of the intermediate layer of the teacher model comprises: Selecting multiple target intermediate layers in the encoder or decoder of the teacher model, wherein the number of layers of the target intermediate layers is equal to the number of layers of the intermediate layers of the encoder or decoder of the student model, and the target intermediate layers correspond one to one; The final output of the target intermediate layer and the output of the attention layer in the target intermediate layer are used as labels to perform distillation training on the corresponding intermediate layer of the encoder or decoder of the student model.
9. The method according to claim 8, It is characterized in that The selecting of multiple target intermediate layers in the encoder or decoder of the teacher model includes: For the encoder or decoder of the teacher model, a target intermediate layer is selected every K layers; wherein K=M / N, M is the total number of intermediate layers of the encoder or decoder of the teacher model, and N is the total number of intermediate layers of the encoder or decoder of the student model.
10. The method according to claim 3 or 4, It is characterized in that The distillation loss of the intermediate layers of the teacher model is determined by at least one of the following: The total number of intermediate layers of the encoder or decoder of the teacher model; Parameters of the hidden layers in the intermediate layers; Parameters of the attention layer in the intermediate layer.
11. The method according to claim 10, It is characterized in that The calculation formula of the distillation loss of the intermediate layer of the teacher model is as follows: Among them, loss middle is the distillation loss of the intermediate layer of the encoder or decoder of the teacher model, M is the total number of intermediate layers of the encoder or decoder of the teacher model, i is the layer number of the intermediate layer of the encoder or decoder of the teacher model, H T is the parameter of the hidden layer in the middle layer of the encoder or decoder of the teacher model, H S is the parameter of the hidden layer in the middle layer of the encoder or decoder of the student model, Att T is the parameter of the attention layer in the middle layer of the encoder or decoder of the teacher model, Att S is the parameter of the attention layer in the intermediate layer of the encoder or decoder of the student model, K is the ratio of the total number of intermediate layers of the encoder or decoder of the teacher model to the total number of intermediate layers of the encoder or decoder of the student model, and MES is the mean square loss function.
12. The method of claim 1, It is characterized in that Also includes: The trained student model is used to infer the inference data to obtain an inference result.
13. The method according to claim 12, It is characterized in that The Transformer network model is an end-to-end speech recognition model, and the trained student model is applied to at least one of the following scenarios: Offline environment; Private deployment environment.
14. A training system for a Transformer network model, It is characterized in that include: The first acquisition module is used to obtain the original Transformer network model; A first training module, used to train the original Transformer network model using a first training data set to obtain a teacher model; A pruning module, used for pruning the teacher model to obtain a student model; A second training module is used to train the student model using the teacher model and the second training data set, wherein the training of the student model using the teacher model and the second training data set includes: distilling the outputs of the intermediate layers and the final output layers of the teacher model, and determining the loss used in the process of training the student model based on the distillation losses of the intermediate layers and the final output layers.
15. An electronic device, It is characterized in that include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the training method of the Transformer network model as described in any one of claims 1 to 13 are implemented.
16. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the training method of the Transformer network model as described in any one of claims 1 to 13 are implemented.
Citation Information
Patent Citations
Model compression method for multilayer characteristic distillation based on attention
CN113988292A
Model training method based on knowledge distillation, related equipment and readable storage medium
CN114861885A
Automatic compression method and platform for multilevel knowledge distillation-based pre-trained language model
WO2022126797A1
