Knowledge distillation method based on parameter-efficient modules and multi-teacher knowledge distillation

Abstract

The present invention discloses a knowledge distillation method based on parameter efficient module and multi-teacher knowledge distillation, the method comprising: obtaining a downstream task data set, generating multiple sub-teacher models according to a pre-constructed parameter efficient multi-teacher model, fine-tuning each sub-teacher model to obtain multiple fine-tuned sub-teacher models; training a student model to be trained according to the downstream task data set, generating a supervisory signal for the training according to the downstream task data set and multiple fine-tuned sub-teacher models, updating parameters according to the supervisory signal and the training result obtained by the student model during the training process, and obtaining and outputting the trained student model when the preset training conditions are met; obtaining target processing data, inputting the target processing data into the trained student model, and outputting the target result. The present invention can take into account both efficiency and accuracy when training the student model, so as to generate accurate target results through the trained student model.

Description

Technical Field

[0001] The present invention relates to the field of model compression technology, and in particular to a knowledge distillation method based on parameter efficient modules and multi-teacher knowledge distillation. Background Art

[0002] Offline knowledge distillation refers to a knowledge distillation method in which the teacher model is not updated during the training process. Offline knowledge distillation can use more refined supervisory feature signals and improved loss functions to improve the effect of knowledge distillation. Offline knowledge distillation can be divided into single-teacher distillation and multi-teacher distillation according to the number of teacher models.

[0003] Among them, multi-teacher distillation is a special type of offline distillation method. This method uses multiple teacher models to provide different supervision signals to a single student model for learning, and integrates the knowledge of different teacher models to improve the performance and generalization of the student model.

[0004] However, although multi-teacher distillation can provide more diverse supervision signals, it requires more computing and storage resources, resulting in low training efficiency. Therefore, the current multi-teacher distillation method cannot take into account both efficiency and accuracy. When using multi-teacher distillation to train student models and apply them to reasoning, it is impossible to take into account both efficiency and accuracy.

[0005] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0006] The main purpose of the present invention is to provide a knowledge distillation method, system, terminal and computer-readable storage medium based on a parameter-efficient module and multi-teacher knowledge distillation, aiming to solve the problem that multi-teacher distillation in the prior art can provide more diverse supervision signals but requires more computing resources and storage resources, resulting in low training efficiency. Therefore, the current multi-teacher distillation method cannot take into account both efficiency and accuracy. When multi-teacher distillation is used to train student models and applied to reasoning, the problem of not being able to take into account both efficiency and accuracy.

[0007] To achieve the above object, the present invention provides a knowledge distillation method based on parameter efficient module and multi-teacher knowledge distillation, and the knowledge distillation method based on parameter efficient module and multi-teacher knowledge distillation comprises the following steps:

[0008] Obtain a downstream task data set, generate multiple sub-teacher models according to a pre-built parameter efficiency multi-teacher model, and fine-tune each sub-teacher model according to the downstream task data set to obtain multiple corresponding fine-tuned sub-teacher models;

[0009] The student model to be trained is trained according to the downstream task data set, and in each training process, a supervision signal for the training is generated according to the downstream task data set and multiple fine-tuning sub-teacher models, and the parameters of the student model are updated according to the supervision signal and the training results obtained by the student model in the training process, and the training is iterated. When the preset training conditions are met, the trained student model is obtained and output;

[0010] Obtain target processing data, input the target processing data into the trained student model, and output the target result.

[0011] Optionally, the parameter efficiency multi-teacher model comprises a large-scale pre-trained model and a parameter efficiency module, wherein the parameter efficiency module comprises a plurality of prompt modules and a plurality of prefix modules;

[0012] The generating of multiple sub-teacher models according to the pre-built parameter efficiency multi-teacher model specifically includes:

[0013] Based on the gate control unit, different numbers of prompt modules and different numbers of prefix modules in the parameter efficiency multi-teacher model are selected to take effect, thereby generating multiple sub-teacher models.

[0014] Optionally, fine-tuning each sub-teacher model according to the downstream task dataset to obtain a corresponding plurality of fine-tuned sub-teacher models specifically includes:

[0015] Optimizing parameters of a parameter-efficient module of each sub-teacher model according to the downstream task dataset;

[0016] According to the optimization results, multiple fine-tuned sub-teacher models are obtained.

[0017] Optionally, the optimizing the parameters of the parameter efficiency module of each sub-teacher model according to the downstream task data set specifically includes:

[0018] Using the downstream task data set to train the parameters of the parameter efficient module of each sub-teacher model, wherein in each process of training the parameters of the parameter efficient module of each sub-teacher model, an uncertainty learning mechanism is adopted to train each sub-teacher model according to the total loss of the current training and the loss of each sub-teacher model;

[0019] When all the downstream task data sets are used to train the parameters of the parameter-efficient module of each sub-teacher model, the optimization of the parameters of the parameter-efficient module of each sub-teacher model is completed to obtain multiple trained sub-teacher models.

[0020] Optionally, the student model to be trained is trained according to the downstream task data set, and in each training process, a supervision signal for the training is generated according to the downstream task data set and multiple fine-tuning sub-teacher models, and the parameters of the student model are updated according to the supervision signal and the training results obtained by the student model in the training process, and the training is iterated. When the preset training conditions are met, the trained student model is obtained and output, which specifically includes:

[0021] Iteratively train the student model to be trained using the training data in the downstream task dataset, and generate a supervision signal for the training according to the downstream task dataset and multiple fine-tuning sub-teacher models in each training, calculate a weighted loss function according to the supervision signal and the training result obtained by the student model in the training process, and update the parameters of the student model according to the weighted loss function using the gradient back propagation method;

[0022] After each training session, the student model is iteratively trained using the untrained data in the downstream task dataset. When all the data in the downstream task dataset is used to train the student model, the student model obtained from the last training is used as the trained student model.

[0023] Optionally, the method uses the training data in the downstream task data set to iteratively train the student model to be trained, generates a supervision signal for the training according to the downstream task data set and multiple fine-tuning sub-teacher models in each training, calculates a weighted loss function according to the supervision signal and the training result obtained by the student model in the training process, and updates the parameters of the student model according to the weighted loss function by using the gradient back propagation method, specifically including:

[0024] A set of training data in the downstream task dataset is used as current training data, and the current training data is input into the student model to be trained and each fine-tuning sub-teacher model respectively;

[0025] Obtain the prediction results output by each fine-tuning sub-teacher model, and generate a supervisory signal for the training according to the prediction results;

[0026] Obtain the prediction results of the student model to be trained on the training data, and calculate the weighted loss function of the training according to the prediction results of the student model to be trained on the training data, the supervisory signal and the label of the current training data, and update the parameters of the student model using the gradient back propagation method according to the weighted loss function of the training.

[0027] Optionally, obtaining the prediction results output by each fine-tuning sub-teacher model and generating a supervisory signal for the training according to the prediction results specifically includes:

[0028] Get the predictions output by each fine-tuned sub-teacher model;

[0029] Based on the pre-trained selector, calculate the correlation parameters between the current training data and the prediction results output by each fine-tuned sub-teacher model;

[0030] According to the multiple correlation parameters, the prediction result with the greatest correlation with the current training data is selected from the prediction results output by the multiple fine-tuning sub-teacher models as the supervisory signal for this training.

[0031] In addition, to achieve the above-mentioned purpose, the present invention also provides a knowledge distillation system based on parameter efficient module and multi-teacher knowledge distillation, wherein the knowledge distillation system based on parameter efficient module and multi-teacher knowledge distillation includes:

[0032] A fine-tuning module is used to obtain a downstream task data set, generate multiple sub-teacher models according to a pre-built parameter efficiency multi-teacher model, and fine-tune each sub-teacher model according to the downstream task data set to obtain a corresponding multiple fine-tuned sub-teacher models;

[0033] The student model training module is used to train the student model to be trained according to the downstream task data set, and in each training process, generate a supervision signal for the training according to the downstream task data set and multiple fine-tuning sub-teacher models, update the parameters of the student model according to the supervision signal and the training results obtained by the student model in the training process, iterate the training, and when the preset training conditions are met, obtain and output the trained student model;

[0034] The result output module is used to obtain target processing data, input the target processing data into the trained student model, and output the target result.

[0035] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a knowledge distillation program based on a parameter-efficient module and multi-teacher knowledge distillation stored on the memory and run on the processor, wherein the knowledge distillation program based on a parameter-efficient module and multi-teacher knowledge distillation implements the steps of the knowledge distillation method based on a parameter-efficient module and multi-teacher knowledge distillation as described above when executed by the processor.

[0036] In addition, to achieve the above-mentioned objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a knowledge distillation program based on parameter-efficient modules and multi-teacher knowledge distillation, and when the knowledge distillation program based on parameter-efficient modules and multi-teacher knowledge distillation is executed by a processor, the steps of the knowledge distillation method based on parameter-efficient modules and multi-teacher knowledge distillation as described above are implemented.

[0037] In the present invention, a downstream task data set is obtained, and multiple sub-teacher models are generated according to a pre-constructed parameter efficiency multi-teacher model. Each sub-teacher model is fine-tuned according to the downstream task data set to obtain corresponding multiple fine-tuned sub-teacher models; the student model to be trained is trained according to the downstream task data set, and in each training process, a supervision signal for the training is generated according to the downstream task data set and the multiple fine-tuned sub-teacher models, and the parameters of the student model are updated according to the supervision signal and the training results obtained by the student model in the training process, and the training is iterated. When the preset training conditions are met, the trained student model is obtained and output; the target processing data is obtained, the target processing data is input into the trained student model, and the target result is output. The present invention combines a parameter-efficient module with a large-scale pre-trained model to construct a parameter-efficient multi-teacher model. While maintaining the performance of the multimodal pre-trained model, different fine-tuning sub-teacher models can be obtained, thereby reducing the computing resource consumption of the traditional multi-teacher model. In addition, an efficient fine-tuning method is designed for the proposed parameter-efficient multi-teacher model. By using an uncertainty method to improve the robustness of parallel training of different teacher models, multiple trained sub-teacher models can be obtained. In addition, a selector method is used to dynamically select a suitable teacher model to provide a supervision signal, thereby achieving the purpose of model compression and ensuring the effect of multi-teacher distillation. Through the above settings, the present invention can achieve the rich supervision signals of multi-teacher distillation while maintaining the computing and storage resource consumption of a single teacher distillation, and while compressing the model scale, maintain the performance of the model, and train an accurate student model for reasoning, and obtain accurate results. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of a preferred embodiment of the knowledge distillation method based on parameter efficient module and multi-teacher knowledge distillation of the present invention;

[0039] Figure 2 It is a schematic diagram of a multi-teacher knowledge distillation process based on a parameter efficient module in the knowledge distillation method based on a parameter efficient module and multi-teacher knowledge distillation of the present invention;

[0040] Figure 3 is a schematic diagram of the steps of constructing parameter-efficient multi-model teachers in the knowledge distillation method based on parameter-efficient modules and multi-teacher knowledge distillation of the present invention;

[0041] Figure 4 It is a schematic diagram of combining prompts and large-scale pre-training models in the knowledge distillation method based on parameter-efficient modules and multi-teacher knowledge distillation of the present invention;

[0042] Figure 5It is a schematic diagram of combining prefixes with a large-scale pre-training model in the knowledge distillation method based on parameter-efficient modules and multi-teacher knowledge distillation of the present invention;

[0043] Figure 6 is a schematic diagram of a parameter-efficient multi-teacher model in a knowledge distillation method based on a parameter-efficient module and multi-teacher knowledge distillation of the present invention;

[0044] Figure 7 It is a schematic diagram of a fine-tuning process in the knowledge distillation method based on parameter efficient module and multi-teacher knowledge distillation of the present invention;

[0045] Figure 8 It is a schematic diagram of a selector-based multi-teacher distillation process in the knowledge distillation method based on parameter efficient module and multi-teacher knowledge distillation of the present invention;

[0046] Fig. 9 It is a structural diagram of a preferred embodiment of the knowledge distillation system based on parameter efficient module and multi-teacher knowledge distillation of the present invention;

[0047] Fig.10 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0049] Offline knowledge distillation refers to a knowledge distillation method in which the teacher model is not updated during the training process. It focuses on using more refined supervisory feature signals and improved loss functions to improve the effect of knowledge distillation. According to the number of teacher models, it can be divided into single-teacher distillation and multi-teacher distillation.

[0050] The common method in single teacher distillation is the naive distillation method, which can be divided into two stages: the pre-training stage of pre-training the teacher model and the distillation stage of training the student model. In the distillation stage, under a given input, the output of the teacher model is used as supervision information to try to minimize the difference between the output distributions of the teacher model and the student model, so that the student model learns the knowledge from the teacher model to achieve the purpose of maintaining accuracy and reducing computational complexity.

[0051] Multi-teacher distillation is a special type of offline distillation method that uses multiple teacher models to provide different supervision signals to a single student model for learning, and integrates the knowledge of different teacher models to improve the performance and generalization of the student model.

[0052] However, the current offline knowledge distillation has inherent defects, that is, it cannot balance efficiency and accuracy improvement: the accuracy improvement of single-teacher distillation is limited by the richness of the supervision signal, that is, the information provided by a single teacher model is limited; and although multi-teacher distillation can provide more diverse supervision signals, it requires more computing resources and storage resources, resulting in low training efficiency. This has become an obstacle to the lightweight deployment of multimodal pre-trained models on downstream tasks. Therefore, although multi-teacher distillation can provide more diverse supervision signals, it requires more computing resources and storage resources, resulting in low training efficiency. Therefore, the current multi-teacher distillation method cannot balance efficiency and accuracy. When using multi-teacher distillation to train student models and apply them to reasoning, it is impossible to balance efficiency and accuracy.

[0053] In response to one or more of the above problems, the present application obtains a downstream task dataset, generates multiple sub-teacher models based on a pre-built parameter efficiency multi-teacher model, fine-tunes each sub-teacher model according to the downstream task dataset, and obtains corresponding multiple fine-tuned sub-teacher models; trains the student model to be trained according to the downstream task dataset, and in each training process, generates a supervision signal for the training according to the downstream task dataset and the multiple fine-tuned sub-teacher models, updates the parameters of the student model according to the supervision signal and the training results obtained by the student model in the training process, iterates the training, and when the preset training conditions are met, obtains and outputs the trained student model; obtains target processing data, inputs the target processing data into the trained student model, and outputs the target result.

[0054] Specific as Figure 2 As shown, it is a flow chart of an embodiment of multi-teacher knowledge distillation based on parameter efficient module in the knowledge distillation method based on parameter efficient module and multi-teacher knowledge distillation of the present invention. By building a parameter efficient multi-teacher model and fine-tuning the corresponding model, a plurality of fine-tuned sub-teacher models are obtained, and the student model is trained by corresponding a selector to the plurality of fine-tuned sub-teacher models.

[0055] The knowledge distillation method based on parameter efficient module and multi-teacher knowledge distillation described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the knowledge distillation method based on parameter efficient module and multi-teacher knowledge distillation includes the following steps:

[0056] Step S10, obtain a downstream task data set, generate multiple sub-teacher models according to a pre-constructed parameter efficiency multi-teacher model, fine-tune each sub-teacher model according to the downstream task data set, and obtain corresponding multiple fine-tuned sub-teacher models.

[0057] It should be noted that for the selection of parameter-efficient multi-teacher models, the existing large-scale pre-trained models include large-scale language models and large-scale visual models, as well as a series of models aggregated on top of them (such as expert group models based on uncertainty, expert group models based on mutual learning methods, etc.), which can provide multi-angle feature vectors for supervising the training process of student models and improving the effect of distillation. However, these models are still limited by computing resources. In order to solve this problem, the present invention adopts parameter-efficient modules to build an aggregation model, which mainly includes the following three steps, namely, adding prompts and embedding prefixes, to design parameter-efficient multi-teacher models. As follows Figure 3 As shown in the figure, first, a suitable prompt tuning method is selected to be combined with the input embedding of the large-scale pre-trained model, that is, the prompt module is combined with the large-scale pre-trained model; secondly, a suitable module is selected from the prefix tuning method and embedded into the multimodal pre-trained model, that is, the prefix module is combined with the large-scale pre-trained model; finally, a selection method is designed to dynamically use prompts and prefixes to combine with the model, so as to construct a multi-teacher model and obtain multiple sub-teacher models.

[0058] Furthermore, the parameter efficiency multi-teacher model includes a large-scale pre-trained model and a parameter efficiency module, wherein the parameter efficiency module includes a plurality of prompt modules and a plurality of prefix modules;

[0059] The generating of multiple sub-teacher models according to the pre-built parameter efficiency multi-teacher model specifically includes:

[0060] Based on the gate control unit, different numbers of prompt modules and different numbers of prefix modules in the parameter efficiency multi-teacher model are selected to take effect, thereby generating multiple sub-teacher models.

[0061] Specifically, the parameter-efficient multi-teacher model consists of a large-scale pre-trained model and a parameter-efficient module. The parameter-efficient module includes multiple prompt modules and multiple prefix modules. The large-scale pre-trained model includes a large-scale language model and a large-scale visual model, and a series of models aggregated on top of it. The corresponding prompt module can modify the input features of the model through a prompt adjustment method to obtain features that have both input corpus and external knowledge.

[0062] The prompt adjustment method used in the present invention is a soft prompt. Soft prompts use trainable embedding vectors as prompt modules, which are parameter-efficient modules like the prefix module used below. In the present invention, soft prompts do not need to be trained in advance, and the soft prompt vectors are initialized using the word element embedding in the downstream task, and are updated when each sub-teacher model is fine-tuned according to the downstream task dataset.

[0063] The feature data of the prompt module is combined with the feature data encoded by the original model as the basis for the input features of other parts of the model. The process of modifying the embedding x of the original input by the prompt module can be described by the following formula:

[0064] E'(x)=Concat(h0,h1,...,h i , E(x));

[0065] Where E(x) is the embedded representation of the input data x, which is obtained by encoding the data x using an encoder based on the Transformer model (such as BERT). The data x can be image data or text data; h i is a soft hint, which is initialized with the word embedding in the downstream task and trained in the teacher fine-tuning phase; Concat(·) is a vector concatenation operation; the soft hint h i It is directly concatenated with the embedding E(x) of the original data to obtain a new embedding E′(x). This new feature E′(x) contains the prompt adjustment {h0, …, h i The external information given by} has more knowledge than the original feature E(x). Figure 4 As shown in Figure 2, the prompt module is located before the encoder in the parameter-efficient multi-teacher model. The data flows through the prompt adjustment module and then enters the encoder. Figure 4 In the present application, the parameter efficiency multi-teacher model can also include multiple prompt modules. In a preferred embodiment of the present application, the parameter efficiency multi-teacher model includes 3 prompt modules.

[0066] In addition to the prompt module, the present invention also applies the prefix module to construct a parameter-efficient multi-teacher model. The prefix module is a trainable embedding across multiple layers, which has an impact on the features of multiple layers, and the prefixes between different layers are independent of each other and are not derived from the Transformer of the previous layer. In this way, the prefix module has more adjustable task parameters to provide more task capacity, but the number of parameters is still much smaller than that of large-scale pre-trained models. The processing of the prefix module can be described by the following formula:

[0067] h′←Concat(prefix, h).

[0068] Where h is the feature of the middle layer before adding the prefix, prefix is ​​the prefix module, and h′ is the feature after adding the prefix. Specifically, the prefix module adds an adjustable prefix vector P before the key K and value V in the multi-head attention mechanism of each layer of Transformer. k , P v ∈R l×d, R represents the real number space, l and d are the number and dimension of prefix vectors respectively, and the two sets of prefix vectors P k , P v Connect with the original key K and value V, calculate the multi-head attention on the new key and value and pass it forward. The multi-head attention mechanism in the prefix module can be expressed as the following formula:

[0069]

[0070] Among them, head i represents the i-th attention head, x is the input of the attention layer, is the query transformation matrix, is the key transformation matrix, is the value transformation matrix. In order to adapt to multi-head attention, P k , P v Divided into N sub-prefixes on the header like Figure 5 As shown, the position of the prefix module in the framework is generally each layer of the model, and multiple prefixes are used in the figure. In the present application, multiple prefix modules can also be included in the parameter efficiency multi-teacher model. In a preferred embodiment of the present application, each layer of the model in the parameter efficiency multi-teacher model includes 2 prompt modules.

[0071] Based on the prompt module and prefix module, a parameter-efficient multi-teacher model is constructed in combination with a large-scale pre-trained model. The prompt module, prefix module and large-scale pre-trained model are independent of each other. The prompt module and prefix module can be combined with the large-scale pre-trained model separately and fine-tuned on downstream tasks to achieve downstream task migration of the model. For the parameter-efficient multi-teacher model, gates can be used to control the activation and deactivation of different prompt modules and prefix modules of the unit, so that different sub-teacher models can be generated according to user needs.

[0072] The main idea of ​​constructing a parameter-efficient multi-teacher model based on parameter-efficient modules is to apply the parameter-efficient modules to downstream tasks, store the knowledge in the corresponding modules while updating the parameters, and then generate the corresponding downstream task models by combining the modules of the corresponding tasks with the encoder and decoder of the pre-trained large model. These models not only have the generalized knowledge of large-scale pre-trained models, but also have the knowledge acquired by the parameter-efficient modules from downstream tasks. On this basis, multiple sub-teacher models can be constructed through the gate control unit, and these models have their own strengths in knowledge. Figure 6 As shown in the figure, a parameter-efficient multi-teacher model is demonstrated. The constructed parameter-efficient multi-teacher model is shown at the top of the figure. Different prompt modules and prefix modules are selected through gated radio buttons to obtain the corresponding sub-teacher models, namely teacher model 1 and teacher model 2 in the figure.

[0073] For the student model learning on downstream tasks, data enhancement can improve the effect of the student model by increasing the learning samples of the student model. Multi-teacher distillation provides multiple different aspects of representation for the same batch of data, which is also a kind of data enhancement and can increase the knowledge acquired by the student model. In this way, each batch of data can be converted into a representation that depends on the proprietary knowledge of all teacher models, and combined with the correlation between knowledge from different angles, the features generated by the student can better maintain the semantic structure of the training data.

[0074] Further, fine-tuning each sub-teacher model according to the downstream task data set to obtain a corresponding plurality of fine-tuned sub-teacher models specifically includes:

[0075] Optimizing parameters of a parameter-efficient module of each sub-teacher model according to the downstream task dataset;

[0076] According to the optimization results, multiple fine-tuned sub-teacher models are obtained.

[0077] Specifically, there are multiple combinations of different parameter-efficient modules included in the parameter-efficient multi-teacher model, that is, the number of prompt modules and prefix modules in each combination is different, and each combination can constitute a sub-teacher model alone, so that multiple sub-teacher models can provide multi-source heterogeneous expressions of the same batch of data, that is, the output of data on different models. However, if each combination is trained separately, the amount of computing resources required is large, and the modules that are repeatedly trained may overfit, damaging the expressive power of the model, while the training of another part of the teacher model may have underfitting problems. To this end, the present invention further proposes a multi-model fine-tuning method based on an uncertainty learning mechanism, which is applied to multi-teacher distillation tasks.

[0078] like Figure 7 As shown in the figure, the multi-model fine-tuning method based on uncertainty reasoning is divided into two stages, the uncertainty reasoning stage and the back-propagation stage, wherein the uncertainty reasoning aims to calculate the corresponding uncertainty through the distribution of the model output on the input sample to reflect the prediction differences of different teacher models for the same data, so as to train different teacher models in a targeted manner; for teachers with higher uncertainty, larger weights are given to promote learning to balance the performance differences between different teacher models, and uncertainty is used as the weight to balance the loss functions of different teacher models. The loss function is calculated and back-propagated in the training stage to update the parameters of the corresponding modules.

[0079] Through the above two parts, namely uncertainty reasoning and model training, multiple sub-teacher models can be efficiently trained to obtain multiple fine-tuned sub-teacher models, improve the overfitting and underfitting problems of model training, and enhance the robustness of the model.

[0080] Further, the optimization of the parameters of the parameter efficiency module of each sub-teacher model according to the downstream task data set specifically includes:

[0081] Using the downstream task data set to train the parameters of the parameter efficient module of each sub-teacher model, wherein in each process of training the parameters of the parameter efficient module of each sub-teacher model, an uncertainty learning mechanism is adopted to train each sub-teacher model according to the total loss of the current training and the loss of each sub-teacher model;

[0082] When all the downstream task data sets are used to train the parameters of the parameter-efficient module of each sub-teacher model, the optimization of the parameters of the parameter-efficient module of each sub-teacher model is completed to obtain multiple trained sub-teacher models.

[0083] Specifically, before providing supervision to the student model, multiple sub-teacher models M T Fine-tuning is required on the downstream task dataset D to optimize the prefix parameter θ Prefix (i.e., the parameters of the hint module and prefix module), extracting knowledge from downstream tasks while keeping the parameters θ of the large-scale pre-trained model PLM unchanged, where θ PLM >>θ Prefix . Among them, for a set of parameters {prefix1, prefix2, ..., prefix k}, which can be expressed as θ Prefix ={θ Prefix1 ,θ Prefix2 , ..., θ Prefixk}, each prefix i Both can be used independently as a sub-teacher model MT with PLM (Pre-trained Language Model) i .

[0084] When training the parameters of the parameter-efficient module of each sub-teacher model, the task-related cross entropy loss L can be calculated through the prediction and label y obtained by each sub-teacher model for the input instance x task , where p(·) is the prediction vector of the teacher model:

[0085]

[0086] In order to improve the diversity of different prefixes, this application introduces an L1 loss for each prefix module and prompt module. L1 calculates the sum of the absolute values ​​of each element in the prefix. This loss can prevent the model from overfitting due to excessive values ​​of some elements in the prefix. taskCombined with the L1 loss, we can get the loss L of each sub-teacher model Ti :

[0087]

[0088] Then, an uncertainty-based learning mechanism is introduced to fine-tune the teacher model. In this process, multiple sub-teacher models can be trained in parallel for the input instance x. Ti The uncertainty u of the input x i is the entropy of the probability distribution, which is used to balance the losses of different teachers. The formula is as follows, where y j is the jth category in the category set:

[0089]

[0090] On this basis, the loss L of each sub-teacher is Ti Multiply by u i The relevant weights are added together to get the total loss L of all sub-teacher models. teacher , where K is the number of all sub-teacher models:

[0091]

[0092] Through the uncertainty-based loss, the child teacher model can learn to make more correct predictions on downstream tasks more confidently, thereby providing higher quality supervision signals to the student model.

[0093] Step S20: train the student model to be trained according to the downstream task data set, and generate a supervision signal for this training according to the downstream task data set and multiple fine-tuning sub-teacher models during each training process, update the parameters of the student model according to the supervision signal and the training results obtained by the student model during this training process, iterate the training, and when the preset training conditions are met, obtain and output the trained student model.

[0094] Specifically, the student model of the downstream task is subjected to selector-based multi-teacher distillation. In each training or reasoning process of the student model, feature extraction is first performed, that is, the input data (such as a frame of image, a piece of speech or a word) is mapped to the corresponding encoding vector through the encoder of the neural network; among them, the pre-trained model encoder (ResNet152, BE RT, etc.) used in the present invention realizes the feature extraction of downstream task data, mainly including image feature extraction and text feature extraction.

[0095] Further, the student model to be trained is trained according to the downstream task data set, and in each training process, a supervision signal for the training is generated according to the downstream task data set and multiple fine-tuning sub-teacher models, and the parameters of the student model are updated according to the supervision signal and the training results obtained by the student model in the training process, and the training is iterated. When the preset training conditions are met, the trained student model is obtained and output, which specifically includes:

[0096] Iteratively train the student model to be trained using the training data in the downstream task dataset, and generate a supervision signal for the training according to the downstream task dataset and multiple fine-tuning sub-teacher models in each training, calculate a weighted loss function according to the supervision signal and the training result obtained by the student model in the training process, and update the parameters of the student model according to the weighted loss function using the gradient back propagation method;

[0097] After each training session, the student model is iteratively trained using the untrained data in the downstream task dataset. When all the data in the downstream task dataset is used to train the student model, the student model obtained from the last training is used as the trained student model.

[0098] Specifically, after fine-tuning multiple sub-teacher models with the downstream task dataset D, the student model is trained with this downstream task dataset D and the output of the sub-teacher model on D. When training the student model, the supervision signal includes two parts: the label in D and the output of the sub-teacher model on D. In order to obtain the output of the multi-teacher model on D, a selector is needed to calculate the correlation between the input data and multiple teacher models, so as to select the output of the teacher model with the highest correlation with the input data as the supervision signal to supervise the student model.

[0099] The method uses the training data in the downstream task data set to iteratively train the student model to be trained, generates a supervision signal for the training according to the downstream task data set and multiple fine-tuning sub-teacher models in each training, calculates a weighted loss function according to the supervision signal and the training result obtained by the student model in the training process, and updates the parameters of the student model according to the weighted loss function by using the gradient back propagation method, specifically including:

[0100] A set of training data in the downstream task dataset is used as current training data, and the current training data is input into the student model to be trained and each fine-tuning sub-teacher model respectively;

[0101] Obtain the prediction results output by each fine-tuning sub-teacher model, and generate a supervisory signal for the training according to the prediction results;

[0102] Obtain the prediction results of the student model to be trained on the training data, and calculate the weighted loss function of the training according to the prediction results of the student model to be trained on the training data, the supervisory signal and the label of the current training data, and update the parameters of the student model using the gradient back propagation method according to the weighted loss function of the training.

[0103] The training data is input into multiple fine-tuned sub-teacher models, and the output is the prediction of the fine-tuned sub-teacher model for the data. The student model uses the prediction of the fine-tuned sub-teacher model as a soft label different from the data label, that is, the supervision signal, participates in the calculation of the loss function and back-propagates to update the parameters, thereby supervising its own learning process. In this process, the prediction form of the sub-teacher model is used as implicit knowledge to guide the student model. The loss of the teacher model and the student model is called the knowledge distillation loss, which is calculated by the Kullback-Leibler divergence between the predictions of the two models; specifically, formally, given a downstream task dataset D, the total loss L can be written as S and knowledge distillation loss L KD as follows:

[0104] L S =αL task +(1-α)L KD (p S , p T )

[0105]

[0106] Among them, α is a hyperparameter used to control the weight of the two losses, p T is the prediction of the input instance x selected by multiple fine-tuned sub-teacher models, i.e., the supervisory signal, p S is the student model’s prediction for the input instance x, is the i-th component in the model prediction output.

[0107] Among them, according to the pre-trained selector, the correlation parameter between the current training data and the prediction results output by each fine-tuned sub-teacher model is calculated, and according to multiple of the correlation parameters, the prediction result with the greatest correlation with the current training data is selected from the prediction results output by multiple fine-tuned sub-teacher models as the prediction p of the sub-teacher model for the input instance x at this time T .

[0108] Iterative training of the student model to be trained can be done by Figure 8It is further stated that in the distillation process of downstream tasks, for the input training data, multiple fine-tuned sub-teacher models and student models simultaneously receive the data features encoded by the multimodal encoder and input them into the model. Multiple fine-tuned sub-teacher models select one of them as the supervision signal by outputting the corresponding prediction vector; the student model also predicts the input features; at the same time, the label corresponding to the data feature is used to calculate the student model label loss. The supervision signal is used to calculate the teacher supervision loss of the student model. The two loss functions are weighted and summed to calculate the gradient back propagation and update the parameters of the student model.

[0109] Furthermore, the step of obtaining the prediction results output by each fine-tuning sub-teacher model and generating the supervisory signal for the training according to the prediction results specifically includes:

[0110] Get the predictions output by each fine-tuned sub-teacher model;

[0111] Based on the pre-trained selector, calculate the correlation parameters between the current training data and the prediction results output by each fine-tuned sub-teacher model;

[0112] According to the multiple correlation parameters, the prediction result with the greatest correlation with the current training data is selected from the prediction results output by the multiple fine-tuning sub-teacher models as the supervisory signal for this training.

[0113] Specifically, expert guidance, i.e., selector, plays an important role in the quality of multi-teacher distillation. For the student model, expert guidance determines which fine-tuned sub-teacher models will be used to guide the student model, i.e., what knowledge the student model can learn. In one embodiment of the present invention, a variety of different forms of expert guidance methods are used, and different guidance methods are selected according to the data of different tasks to ensure the effect of multi-teacher distillation. For general data, a variety of methods can be used: the completely random selection method is expected to have the same selection probability for different fine-tuned sub-teacher models. It can be considered that in this case, the difference between different fine-tuned sub-teacher models is small, and there is no need to design additional mechanisms or indicators to maintain the expert guidance module; the aggregation method is a method of aggregating the outputs of multiple models into one model, such as weighted summation or averaging to aggregate the prediction vectors of multiple models into one vector, and use this vector to guide the training of the student model. Its advantage is that when combining the prediction vectors of multiple models, it is less affected by a single model. At this time, the poor performance of a single teacher model has less effect on knowledge distillation.

[0114] In this application, preferably, a multi-layer perceptron (MLP) is selected for expert guidance. At this time, the multi-layer perceptron is also called a selector, which is used to select the teacher model with the highest correlation with the current data from multiple fine-tuned teacher models. Given a batch of data and a sub-teacher model, the selector will calculate the degree of matching between the data and the teacher model and output the correlation parameter. Among them, the selector is also optimized, and its optimization process is carried out in the process of training the student model to be trained and obtaining the trained student model.

[0115] Specifically, for each fine-tuned sub-teacher model, the prediction result is obtained, and the supervisory signal for this training is generated according to the prediction result, where x is the input instance, prefix i is the parameter of the parameter efficient module in the i-th sub-teacher module, then the single sub-teacher model M Ti Output E i for:

[0116]

[0117] where θ PLM are the parameters of the large-scale pre-trained model, θ prefixi is the parameter efficient module prefix used i The corresponding parameters. By integrating the output of different fine-tuned sub-teacher models through the output of the selector, the output p of the entire framework T for:

[0118] p T =G i (x)·E i , i = argmax i G i (x);

[0119] Among them G i (x) is the correlation parameter of the i-th fine-tuning sub-teacher, from which it is indicated that the prediction result with the greatest correlation with the current training data is selected as the supervisory signal for this training.

[0120] Step S30: Obtain target processing data, input the target processing data into the trained student model, and output the target result.

[0121] Specifically, in this application, after the student model is trained, the target processing data can be input into the student model, and the target result can be output accordingly. The target processing data can be a frame of image, a segment of speech, or a word.

[0122] Furthermore, this application verifies the knowledge distillation method based on parameter efficient modules and multi-teacher knowledge distillation through experiments. In order to reflect the effectiveness of the method, Vanilla-KD and various multi-teacher distillation methods are used as comparison methods, and several effective single-teacher distillation methods (CRD and DKD) are also included. Multi-teacher distillation methods include: random switching method (Rand-Switch), randomly selected teachers; average ensemble method (A-Ensemble), averaging the outputs of different teachers; interpolation ensemble method (I-Ensemble), assigning posterior weights to teachers; LR-Ensemble method (LR-Ensemble), using logistic regression (LR) model to output weights for teachers; best model method (Best-Single), selecting the teacher with the lowest loss. The above comparison methods can be divided into two groups: V-KD uses the signal of a single teacher to supervise the student, including Vanilla-KD, CRD and DKD; Ensemble method (Ensemble) integrates the signals from multiple teachers to generate supervision for the student, including Rand-Switch, Best-Single, A-Ensemble, I-Ensemble and LR-Ensemble. In addition to the above comparison methods, the experiment also includes multi-teacher knowledge distillation (MoE-KD) based on multi-parameter efficient modules and a comparison method (Selector) using a selector module combined with multiple teacher models.

[0123] In the verification, in the multi-teacher distillation framework, the present invention uses fully fine-tuned RoBERTa-large, BERT-large and DeBERTa-large as teacher models, and uses RoBERTa-base and BERT-base as student models for training. In the V-KD-type distillation method, the teacher model uses a larger model that is isomorphic to the student model.

[0124] In the ensemble method, the teacher model uses three teacher models (RoBERTa-large, BERTlarge, and DeBERTa-large) for knowledge distillation. For the CB dataset, the metrics reported in the experiment are the average of accuracy and F1 value. For other datasets, the metric reported in the experiment is accuracy.

[0125] Through experiments, we obtained the corresponding experimental results. The following are the verification experimental results on some tasks of the natural language understanding dataset SuperGLUE and the named entity recognition dataset CoNLL03. The experimental results include accuracy, teacher model parameters, training strategies, and student training time.

[0126] The following Tables 1 and 2 show the performance of student models in natural language understanding tasks using different knowledge distillation methods. MoE-KD (a knowledge distillation method based on parameter-efficient modules and multi-teacher knowledge distillation) performs better than the comparison methods in multiple natural language understanding tasks and maintains the same fast training speed as Vanilla-KD.

[0127]

[0128] Table 1. Comparison of distillation effects of different distillation methods on the RoBERTa-base student model on the SuperGLUE dataset

[0129]

[0130] Table 2. Comparison of distillation effects of different distillation methods on the BERT-base student model on the SuperGLUE dataset

[0131] Table 3 shows the effects of various knowledge distillation methods on the named entity recognition task. All methods use RoBERTa-base as the student model. The named entity recognition task involves the classification of a series of labels, which is related to information extraction. In the named entity recognition task, the relevant indicators of MoEKD are also better than other comparison methods, while maintaining a low training time, which confirms the effectiveness of the proposed method.

[0132]

[0133] Table 3. Comparison of distillation effects of different distillation methods on the RoBERTa-base student model on the coNLL03 dataset

[0134] The above experiments demonstrate the effectiveness of the model compression method based on parameter-efficient modules and multi-teacher knowledge distillation proposed in the present invention, which can accelerate the model training phase and improve the performance of the student model, thereby achieving the goal of model compression that takes both efficiency and performance into consideration.

[0135] In summary, the present invention obtains a downstream task data set, generates multiple sub-teacher models according to a pre-constructed parameter efficiency multi-teacher model, and fine-tunes each sub-teacher model according to the downstream task data set to obtain corresponding multiple fine-tuned sub-teacher models; trains the student model to be trained according to the downstream task data set, and in each training process, generates a supervision signal for this training according to the downstream task data set and the multiple fine-tuned sub-teacher models, updates the parameters of the student model according to the supervision signal and the training results obtained by the student model in this training process, iterates the training, and when the preset training conditions are met, obtains and outputs the trained student model; obtains target processing data, inputs the target processing data into the trained student model, and outputs the target result. The present invention combines a parameter-efficient module with a large-scale pre-trained model to construct a parameter-efficient multi-teacher model. While maintaining the performance of the multimodal pre-trained model, different fine-tuning sub-teacher models can be obtained, thereby reducing the computing resource consumption of the traditional multi-teacher model. In addition, an efficient fine-tuning method is designed for the proposed parameter-efficient multi-teacher model. By using an uncertainty method to improve the robustness of parallel training of different teacher models, multiple trained sub-teacher models can be obtained. In addition, a selector method is used to dynamically select a suitable teacher model to provide a supervision signal, thereby achieving the purpose of model compression and ensuring the effect of multi-teacher distillation. Through the above settings, the present invention can achieve the rich supervision signals of multi-teacher distillation while maintaining the computing and storage resource consumption of a single teacher distillation, and while compressing the model scale, maintain the performance of the model, and train an accurate student model for reasoning, and obtain accurate results.

[0136] Furthermore, if Fig. 9 As shown, based on the above-mentioned knowledge distillation method based on parameter efficient module and multi-teacher knowledge distillation, the present invention also provides a knowledge distillation system based on parameter efficient module and multi-teacher knowledge distillation, wherein the knowledge distillation system based on parameter efficient module and multi-teacher knowledge distillation includes:

[0137] A fine-tuning module 91 is used to obtain a downstream task data set, generate multiple sub-teacher models according to a pre-built parameter efficiency multi-teacher model, and fine-tune each sub-teacher model according to the downstream task data set to obtain corresponding multiple fine-tuned sub-teacher models;

[0138] The student model training module 92 is used to train the student model to be trained according to the downstream task data set, and in each training process, generate a supervision signal for the training according to the downstream task data set and multiple fine-tuning sub-teacher models, update the parameters of the student model according to the supervision signal and the training results obtained by the student model in the training process, iterate the training, and when the preset training conditions are met, obtain and output the trained student model;

[0139] The result output module 93 is used to obtain target processing data, input the target processing data into the trained student model, and output the target result.

[0140] Furthermore, if Fig.10 As shown, based on the above-mentioned knowledge distillation method and system based on parameter efficient module and multi-teacher knowledge distillation, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Fig.10 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0141] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a knowledge distillation program 40 based on a parameter efficient module and multi-teacher knowledge distillation is stored on the memory 20, and the knowledge distillation program 40 based on a parameter efficient module and multi-teacher knowledge distillation can be executed by the processor 10, thereby realizing the knowledge distillation method based on a parameter efficient module and multi-teacher knowledge distillation in the present application.

[0142] In some embodiments, the processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the knowledge distillation method based on parameter efficient modules and multi-teacher knowledge distillation.

[0143] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0144] In one embodiment, the above steps are implemented when the processor 10 executes the knowledge distillation program 40 based on the parameter efficient module and multi-teacher knowledge distillation in the memory 20.

[0145] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a knowledge distillation program based on parameter-efficient modules and multi-teacher knowledge distillation, and when the knowledge distillation program based on parameter-efficient modules and multi-teacher knowledge distillation is executed by a processor, the steps of the knowledge distillation method based on parameter-efficient modules and multi-teacher knowledge distillation as described above are implemented.

[0146] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or terminal including the element.

[0147] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0148] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A knowledge distillation method based on parameter-efficient modules and multi-teacher knowledge distillation, characterized in that: The knowledge distillation method based on parameter efficient module and multi-teacher knowledge distillation includes: Obtain a downstream task data set, generate multiple sub-teacher models according to a pre-built parameter efficiency multi-teacher model, and fine-tune each sub-teacher model according to the downstream task data set to obtain multiple corresponding fine-tuned sub-teacher models; The student model to be trained is trained according to the downstream task data set, and in each training process, a supervision signal for the training is generated according to the downstream task data set and multiple fine-tuning sub-teacher models, and the parameters of the student model are updated according to the supervision signal and the training results obtained by the student model in the training process, and the training is iterated. When the preset training conditions are met, the trained student model is obtained and output; Obtain target processing data, input the target processing data into the trained student model, and output the target result; The parameter efficiency multi-teacher model includes a large-scale pre-training model and a parameter efficiency module, wherein the parameter efficiency module includes a plurality of prompt modules and a plurality of prefix modules; The generating of multiple sub-teacher models according to the pre-built parameter efficiency multi-teacher model specifically includes: Based on the gate control unit, different numbers of prompt modules and different numbers of prefix modules in the parameter efficiency multi-teacher model are selected to take effect, thereby generating multiple sub-teacher models; The fine-tuning of each sub-teacher model according to the downstream task data set to obtain a corresponding plurality of fine-tuned sub-teacher models specifically includes: Optimizing parameters of a parameter-efficient module of each sub-teacher model according to the downstream task dataset; According to the optimization results, multiple fine-tuned sub-teacher models are obtained; The optimizing of the parameters of the parameter efficient module of each sub-teacher model according to the downstream task data set specifically includes: Using the downstream task data set to train the parameters of the parameter efficient module of each sub-teacher model, wherein in each process of training the parameters of the parameter efficient module of each sub-teacher model, an uncertainty learning mechanism is adopted to train each sub-teacher model according to the total loss of the current training and the loss of each sub-teacher model; When all the downstream task data sets are used to train the parameters of the parameter-efficient module of each sub-teacher model, the optimization of the parameters of the parameter-efficient module of each sub-teacher model is completed to obtain multiple trained sub-teacher models.

2. The knowledge distillation method based on parameter efficient module and multi-teacher knowledge distillation according to claim 1 is characterized in that: The method trains the student model to be trained according to the downstream task data set, generates a supervisory signal for the training according to the downstream task data set and multiple fine-tuning sub-teacher models in each training process, updates the parameters of the student model according to the supervisory signal and the training result obtained by the student model in the training process, iterates the training, and obtains and outputs the trained student model when the preset training conditions are met, specifically including: Iteratively train the student model to be trained using the training data in the downstream task dataset, and generate a supervision signal for the training according to the downstream task dataset and multiple fine-tuning sub-teacher models in each training, calculate a weighted loss function according to the supervision signal and the training result obtained by the student model in the training process, and update the parameters of the student model according to the weighted loss function using the gradient back propagation method; After each training session, the student model is iteratively trained using the untrained data in the downstream task dataset. When all the data in the downstream task dataset is used to train the student model, the student model obtained from the last training is used as the trained student model.

3. The knowledge distillation method based on parameter efficient module and multi-teacher knowledge distillation according to claim 2 is characterized in that: The method uses the training data in the downstream task data set to iteratively train the student model to be trained, generates a supervision signal for the training according to the downstream task data set and multiple fine-tuning sub-teacher models in each training, calculates a weighted loss function according to the supervision signal and the training result obtained by the student model in the training process, and updates the parameters of the student model according to the weighted loss function by using the gradient back propagation method, specifically including: A set of training data in the downstream task dataset is used as current training data, and the current training data is input into the student model to be trained and each fine-tuning sub-teacher model respectively; Obtain the prediction results output by each fine-tuning sub-teacher model, and generate a supervisory signal for the training according to the prediction results; Obtain the prediction results of the student model to be trained on the training data, and calculate the weighted loss function of the training according to the prediction results of the student model to be trained on the training data, the supervisory signal and the label of the current training data, and update the parameters of the student model using the gradient back propagation method according to the weighted loss function of the training.

4. The knowledge distillation method based on parameter efficient module and multi-teacher knowledge distillation according to claim 3 is characterized in that: The obtaining of the prediction results output by each fine-tuning sub-teacher model and generating the supervisory signal for the training according to the prediction results specifically includes: Get the predictions output by each fine-tuned sub-teacher model; Based on the pre-trained selector, calculate the correlation parameters between the current training data and the prediction results output by each fine-tuned sub-teacher model; According to the multiple correlation parameters, the prediction result with the greatest correlation with the current training data is selected from the prediction results output by the multiple fine-tuning sub-teacher models as the supervisory signal for this training.

5. A knowledge distillation system based on parameter-efficient modules and multi-teacher knowledge distillation, characterized in that: The knowledge distillation system based on parameter efficient module and multi-teacher knowledge distillation includes: A fine-tuning module is used to obtain a downstream task data set, generate multiple sub-teacher models according to a pre-built parameter efficiency multi-teacher model, and fine-tune each sub-teacher model according to the downstream task data set to obtain a corresponding multiple fine-tuned sub-teacher models; The student model training module is used to train the student model to be trained according to the downstream task data set, and in each training process, generate a supervision signal for the training according to the downstream task data set and multiple fine-tuning sub-teacher models, update the parameters of the student model according to the supervision signal and the training results obtained by the student model in the training process, iterate the training, and when the preset training conditions are met, obtain and output the trained student model; A result output module is used to obtain target processing data, input the target processing data into the trained student model, and output the target result; The parameter efficiency multi-teacher model includes a large-scale pre-training model and a parameter efficiency module, wherein the parameter efficiency module includes a plurality of prompt modules and a plurality of prefix modules; The generating of multiple sub-teacher models according to the pre-built parameter efficiency multi-teacher model specifically includes: Based on the gate control unit, different numbers of prompt modules and different numbers of prefix modules in the parameter efficiency multi-teacher model are selected to take effect, thereby generating multiple sub-teacher models; The fine-tuning of each sub-teacher model according to the downstream task data set to obtain a corresponding plurality of fine-tuned sub-teacher models specifically includes: Optimizing parameters of a parameter-efficient module of each sub-teacher model according to the downstream task dataset; According to the optimization results, multiple fine-tuned sub-teacher models are obtained; The optimizing of the parameters of the parameter efficient module of each sub-teacher model according to the downstream task data set specifically includes: Using the downstream task data set to train the parameters of the parameter efficient module of each sub-teacher model, wherein in each process of training the parameters of the parameter efficient module of each sub-teacher model, an uncertainty learning mechanism is adopted to train each sub-teacher model according to the total loss of the current training and the loss of each sub-teacher model; When all the downstream task data sets are used to train the parameters of the parameter-efficient module of each sub-teacher model, the optimization of the parameters of the parameter-efficient module of each sub-teacher model is completed to obtain multiple trained sub-teacher models.

6. A terminal, characterized in that: The terminal includes: a memory, a processor, and a knowledge distillation program based on a parameter-efficient module and multi-teacher knowledge distillation stored in the memory and executable on the processor. When the knowledge distillation program based on a parameter-efficient module and multi-teacher knowledge distillation is executed by the processor, the steps of the knowledge distillation method based on a parameter-efficient module and multi-teacher knowledge distillation are implemented as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a knowledge distillation program based on a parameter-efficient module and multi-teacher knowledge distillation. When the knowledge distillation program based on a parameter-efficient module and multi-teacher knowledge distillation is executed by a processor, the steps of the knowledge distillation method based on a parameter-efficient module and multi-teacher knowledge distillation are implemented as described in any one of claims 1 to 4.

Images