Model compression method, system and equipment based on dynamic adaptive distillation

Abstract

The invention discloses a model compression method, system and equipment based on dynamic adaptive distillation, and the method comprises the steps: obtaining general public data, and screening to obtain a training data set; inputting the training data set into an existing large model according to an open source framework for fine tuning training to obtain a teacher model; determining a student model, and determining a training state of the student model by adopting a partial weight freezing strategy; and respectively inputting training data into the student model and the teacher model for forward propagation, calculating dynamic adaptive distillation loss and updating and optimizing weight according to a dynamic adaptive distillation loss function combining SKL and SRKL, completing distillation training of the student model, and realizing compression from the teacher model to the student model. Through the technical scheme of the invention, the computing resource and time cost required by training are greatly reduced, the training efficiency is improved, the model compression effect is better, the universal generalization ability is stronger, and the problem of invalid or excessive error information is effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of large model technology, and in particular to a model compression method based on dynamic adaptive distillation, a model compression system based on dynamic adaptive distillation, and an electronic device. Background Art

[0002] In recent years, large language models (LLMs) have made great breakthroughs in the field of natural language processing (NLP), especially in machine translation, text summarization, and intelligent question answering. However, these large language models usually contain billions to hundreds of billions of parameters. This leads to extremely high computational costs for training and inference, making them difficult to deploy in resource-limited environments such as mobile devices and edge computing. In order to solve the resource problem, it is very important to compress the model. There are two common model compression methods.

[0003] Knowledge distillation: This is a model compression technique that transfers the knowledge learned from a large model (called the teacher model) to a smaller model (called the student model). It does this by minimizing the difference between the output of the student model and the teacher model. This difference is generally measured by the Kubek-Leibler divergence. It makes the student model pay more attention to the categories with higher probability of the teacher model, which improves the convergence speed and performance of the student model to a certain extent.

[0004] Parameter pruning: Reduce the size and computational complexity of the model by removing redundant parameters in the neural network. Pruning can be done gradually during training or all at once after training. Moreover, it can be retrained after pruning, and pruning and retraining can be performed multiple times.

[0005] However, both of these model compression methods have certain defects, as follows:

[0006] Knowledge distillation: The output probability distribution of current text generation tasks is highly complex, which may cause the student model to be unable to fully capture the output distribution of the teacher model, or to be too concentrated in the high probability distribution area, which will cause the student model to fail to achieve optimal performance. Secondly, conventional knowledge distillation methods generally use fixed data sets for training, which may lead to inconsistent distributions between input sequences for training and reasoning, that is, there is a large difference in the distribution of training data and reasoning data, which will lead to poor performance.

[0007] Parameter pruning: This method is very critical and difficult to select appropriate pruning criteria and thresholds. Different criteria may lead to different effects. In addition, the structure of the pruned model may become irregular, which may bring challenges to parallel computing of hardware. Therefore, it performs poorly in terms of stability. It is too dependent on the pruning criteria and threshold parameters, making it difficult to guarantee the performance of the pruned model and difficult to apply to a variety of text generation tasks. Summary of the invention

[0008] In view of the above problems, the present invention provides a model compression method, system and device based on dynamic adaptive distillation. By adopting a model freezing strategy for training, not only can the memory, video memory and other resource requirements of model training be greatly reduced, but also the training efficiency and actual performance of the model can be improved. While training in a knowledge distillation manner, a dynamic adaptive distillation loss function is introduced. While achieving model compression, the knowledge of the teacher model can be efficiently transferred to the student model. The dynamic adaptive distillation loss function combines the number of iterations and the dynamic gradient of the cosine function, and can adaptively adjust the optimization target of the student model itself. It can not only efficiently learn the output preference of the teacher model, but also prevent it from overestimating the low probability area of ​​the teacher model distribution.

[0009] To achieve the above object, the present invention provides a model compression method based on dynamic adaptive distillation, comprising:

[0010] Obtain common public data related to the large model and filter it to obtain a training data set;

[0011] According to the open source framework, the training data set is input into the existing large model for fine-tuning training, and the fine-tuned model is used as the teacher model;

[0012] Determine a student model, and use a partial weight freezing strategy to determine a training state of the student model;

[0013] The training data are respectively input into the student model and the teacher model for forward propagation, and the dynamic adaptive distillation loss function combining SKL and SRKL is used to calculate the dynamic adaptive distillation loss and update the optimization weights to complete the distillation training of the student model and realize the compression from the teacher model to the student model.

[0014] In the above technical solution, preferably, the model compression method based on dynamic adaptive distillation also includes:

[0015] The trained student model is tested and verified to determine whether the student model can achieve the expected prediction results.

[0016] In the above technical solution, preferably, the process of obtaining common public data related to the large model and screening to obtain a training data set includes:

[0017] Collecting common public data related to the large model, and filtering out high-quality data from the common public data, wherein the common public data includes data on reading comprehension, information extraction, text summarization, machine translation, and intelligent question answering;

[0018] The high-quality data is saved as a json file in the form of three key-value pairs of "instruction", "input" and "output" as the training data set, where "instruction" represents the task instructions given to the model, "input" represents the text content input into the model, and "output" represents the standard response result of the model.

[0019] In the above technical solution, preferably, the training data set is input into the existing large model for fine-tuning training according to the open source framework, and the fine-tuned model is used as the teacher model. The specific process includes:

[0020] The training data set is input into Internlm2-7B, and fine-tuning training is performed according to the open source framework Huggingface. The fine-tuning training includes word segmentation processing of the original text, parallelization of data and model, forward propagation and calculation of loss value, reverse gradient calculation and update, and model weight preservation;

[0021] The fine-tuned Internlm2-7B model is used as the teacher model.

[0022] In the above technical solution, preferably, the student model is determined, and a partial weight freezing strategy is adopted to determine the training state of the student model, and the specific process includes:

[0023] The Internlm2-1.8B model is used as the student model;

[0024] The training state of the student model is determined by adopting a partial weight freezing strategy in which the first and last ends and some layers in the middle are released from training and the remaining layers are frozen.

[0025] In the above technical solution, preferably, the training data is respectively input into the student model and the teacher model for forward propagation, and the dynamic adaptive distillation loss function combining SKL and SRKL is used to calculate the dynamic adaptive distillation loss and update the optimization weight. The specific process includes:

[0026] Initialize the hyperparameters of the distillation training, load the training data and the word segmenter of the student model, and segment the training data;

[0027] Loading the student model, and training the student model according to the training state;

[0028] The teacher model is loaded and set to the model verification state, and each batch of the training data is respectively input into the student model and the teacher model for forward propagation to obtain the corresponding output probability;

[0029] Performing conventional loss function calculation and distillation loss calculation on the output probabilities of the student model and the teacher model respectively, wherein the conventional loss calculation represents the calculation of the cross entropy loss function on the prediction result and the label, and the distillation loss calculation is obtained based on the skewed KL divergence loss SKL and the skewed reverse KL divergence loss SRKL, the SKL loss uses parameters to control the mixing ratio of the distribution of the student model and the teacher model, and the SRKL loss is calculated by exchanging the output probabilities of the student model and the teacher model;

[0030] The SKL loss and the SRKL loss are combined using a cosine function and the number of iterations to obtain a dynamic adaptive distillation loss.

[0031] The conventional loss function and the dynamic adaptive distillation loss are weighted and calculated, and the model weights are updated and optimized as the number of iterations increases, so that the output probability of the student model is close to the output probability distribution of the teacher model to imitate the prediction method of the teacher model.

[0032] In the above technical solution, preferably, the specific process of testing and verifying the trained student model includes:

[0033] The transformer library is used to record the trained student model and word segmenter, and the word segmenter is used to segment the actual test data;

[0034] The obtained word segmentation ID is input into the student model for model reasoning, the prediction result is compared with the expected label, and the prediction result of the student model is evaluated and verified.

[0035] The present invention also proposes a model compression system based on dynamic adaptive distillation, which uses the model compression method based on dynamic adaptive distillation disclosed in any one of the above technical solutions, including:

[0036] The training data acquisition module is used to obtain common public data related to the large model and filter it to obtain the training data set;

[0037] A teacher model training module is used to input the training data set into the existing large model for fine-tuning training according to the open source framework, and use the fine-tuned model as the teacher model;

[0038] A student model training module, used to determine a student model and determine a training state of the student model using a partial weight freezing strategy;

[0039] The distillation loss training module is used to input the training data into the student model and the teacher model respectively for forward propagation, calculate the dynamic adaptive distillation loss and update the optimization weights using the dynamic adaptive distillation loss function combining SKL and SRKL, complete the distillation training of the student model, and realize the compression from the teacher model to the student model.

[0040] In the above technical solution, preferably, the model compression system based on dynamic adaptive distillation also includes a model testing and verification module, which is used to test and verify the trained student model to determine whether the student model can achieve the expected prediction result.

[0041] The present invention also proposes an electronic device, which includes a memory and a processor, wherein the memory is used to store at least one instruction, and the processor is used to execute the at least one instruction to implement the model compression method based on dynamic adaptive distillation as disclosed in any one of the above technical solutions.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows: by adopting the model freezing strategy for training, not only can the memory, video memory and other resource requirements of the model training be greatly reduced, but also the training efficiency and actual performance of the model can be improved. While adopting the knowledge distillation method for training, a dynamic adaptive distillation loss function is introduced. While achieving model compression, the knowledge of the teacher model can be efficiently transferred to the student model. The dynamic adaptive distillation loss function combines the number of iterations and the dynamic gradient of the cosine function, which can adaptively adjust the optimization target of the student model itself, and can not only efficiently learn the output preference of the teacher model, but also prevent it from overestimating the low probability area of ​​the teacher model distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic flow chart of a model compression method based on dynamic adaptive distillation disclosed in an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a module of a model compression system based on dynamic adaptive distillation disclosed in an embodiment of the present invention.

[0045] In the figure, the corresponding relationship between each component and the reference numeral is as follows:

[0046] 1. Training data acquisition module, 2. Teacher model training module, 3. Student model training module, 4. Distillation loss training module, 5. Model testing and verification module. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0049] like Figure 1 As shown, a model compression method based on dynamic adaptive distillation provided by the present invention includes:

[0050] Obtain common public data related to the large model and filter it to obtain a training data set;

[0051] According to the open source framework, the training data set is input into the existing large model for fine-tuning training, and the fine-tuned model is used as the teacher model;

[0052] Determine the student model and use a partial weight freezing strategy to determine the training status of the student model;

[0053] The training data is input into the student model and the teacher model for forward propagation respectively. The dynamic adaptive distillation loss function combining SKL and SRKL is used to calculate the dynamic adaptive distillation loss and update the optimization weights to complete the distillation training of the student model and realize the compression from the teacher model to the student model.

[0054] In this implementation, by adopting the model freezing strategy for training, not only can the memory, video memory and other resource requirements of the model training be greatly reduced, but also the training efficiency and actual performance of the model can be improved. While adopting the knowledge distillation method for training, a dynamic adaptive distillation loss function is introduced. While achieving model compression, the knowledge of the teacher model can be efficiently transferred to the student model. The dynamic adaptive distillation loss function combines the number of iterations and the dynamic gradient of the cosine function, which can adaptively adjust the optimization goal of the student model itself, and can not only efficiently learn the output preference of the teacher model, but also prevent it from overestimating the low probability area of ​​the teacher model distribution.

[0055] Specifically, on the public general test data, the model compression method of the present invention has higher accuracy, lower perplexity, better model compression effect, and stronger general generalization ability compared with direct fine-tuning training. In terms of reading comprehension and information extraction capabilities, the output text of the fine-tuning training model is not stable enough and it is difficult to meet the needs of practical applications. Specifically, the output text is a single or a small number of phrases that are repeatedly cycled, too redundant, invalid or have too much error information; while the method of the present invention, the stability of the model output has been greatly improved, and there is basically no output of repeated cycles, and the length of the text is controlled within a certain range, which effectively solves the problem of too much invalid or error information. The present invention adopts a model freezing strategy, which can greatly reduce the amount of trainable parameters, thereby greatly reducing the computing resources required for training, reducing the time cost of training, and by releasing the training of the weights of important parts of the model, it also improves the training efficiency.

[0056] In the above implementation, preferably, the general public data related to the large model is obtained and the training data set is obtained by screening. The specific process includes:

[0057] Collect common public data related to the big model and filter out high-quality data from the common public data. Common public data includes data on reading comprehension, information extraction, text summarization, machine translation, and intelligent question answering.

[0058] For high-quality data, save it in the form of three key-value pairs of "instruction", "input" and "output" as a json file as a training data set, where "instruction" represents the task instructions given to the model, such as "extract specific information from the input text" and "generate a suitable response to the input content"; "input" represents the text content input to the model, and "output" represents the standard response result of the model. The training data set can be obtained through the above processing.

[0059] In the above implementation, preferably, the training data set is input into the existing large model for fine-tuning training according to the open source framework, and the fine-tuned model is used as the teacher model. The specific process includes:

[0060] The training data set is input into Internlm2-7B, and fine-tuning training is performed based on the open source framework Huggingface. The fine-tuning training includes word segmentation of the original text, parallelization of data and model, forward propagation and calculation of loss value, reverse gradient calculation and update, and model weight preservation.

[0061] The fine-tuned Internlm2-7B model is used as the teacher model.

[0062] In the above implementation, preferably, the student model is determined, and a partial weight freezing strategy is adopted to determine the training state of the student model. The specific process includes:

[0063] The Internlm2-1.8B model is used as the student model;

[0064] The training status of the student model is determined by adopting a partial weight freezing strategy in which the first and last ends and some layers in the middle are released for training and the remaining layers are frozen.

[0065] Specifically, the 12 layers at the head and tail of the Internlm2-1.8B model are released for training, and every 8 layers in the middle are released for training, and the remaining layers are frozen.

[0066] In the above implementation, preferably, the training data is respectively input into the student model and the teacher model for forward propagation, and the dynamic adaptive distillation loss function combining SKL and SRKL is used to calculate the dynamic adaptive distillation loss and update the optimization weight. The specific process includes:

[0067] Initialize the hyperparameters of distillation training, load the training data and the tokenizer of the student model, tokenize the training data, and convert the training data into token ids;

[0068] Load the student model, and train the student model by releasing the training for the 12 layers at the head and tail of the student model, releasing the training for every 8 layers in the middle, and freezing the remaining layers;

[0069] 1.1. Load the teacher model and set it to model validation state. For each batch of training data, input it into the student model and teacher model for forward propagation to obtain the corresponding output probabilities student_probs and teacher_probs. The specific calculation process is as follows: x i represents the input data of the i-th batch, f S (.) represents the forward propagation function of the student model, f T (.) represents the forward propagation function of the teacher model.

[0070] student_probs(x i )=Softmax(f S (x i ))

[0071] teacher_probs(x i )=Softmax(f T (x i ))

[0072] The output probabilities of the student model and the teacher model are calculated by conventional loss function and distillation loss respectively. Conventional loss calculation means calculating the cross entropy loss function between the prediction results and the labels. The specific calculation formula is as follows:

[0073]

[0074] The distillation loss calculation is based on two variants of KL divergence loss (KLD), namely the tilted KL divergence loss SKL and the tilted reverse KL divergence loss SRKL. The SKL loss uses parameter a to control the mixing ratio of the student model and the teacher model distribution, where parameter a is empirically taken as 0.1. Based on SKL, the SRKL loss is calculated by exchanging the output probabilities of the student model and the teacher model. The specific calculation formula is as follows:

[0075] L (α) SKL (p,q θ )=L KL (p,αp+(1-α)q θ )

[0076] L (α) SRKL (p,q θ )=L KL (q θ ,αq θ +(1-α)p)

[0077] Among them, p represents the output probability distribution of the teacher model, q θ represents the output probability distribution of the student model, which is controlled by the model weight θ. As the training progresses, the model weight θ is continuously updated and optimized so that it gradually approaches p to imitate the teacher model;

[0078] The SKL loss and SRKL loss are combined using the cosine function and the number of iterations to obtain the dynamic adaptive distillation loss DAKL.

[0079] The conventional loss function and the dynamic adaptive distillation loss are weighted and calculated, and the model weights are updated and optimized as the number of iterations increases, so that the output probability of the student model is close to the output probability distribution of the teacher model to imitate the prediction method of the teacher model.

[0080] Specifically, only SKL is used in the early stage of training, so that the student distribution quickly aligns with the teacher distribution. In the later stage of training, a combination of SKL and SRKL is used, so that the student model focuses on adjusting its own distribution preference. The combination is implemented based on the cosine function and the number of iterations. The specific calculation formula is as follows:

[0081]

[0082] L DAKL (p,q θ )=L SKL (p,q θ ), (r_current≤r)

[0083] L DAKL (p,q θ )=βL SKL (p,q θ )+(1-β)L SRKL (p,q θ ), (r_current>r)

[0084] Among them, i represents the current iteration number, I represents the total iteration number, and β is the weight parameter. r refers to the threshold for distinguishing the early and late stages of training, which is empirically set to 0.3. r_current represents the current training progress, which ranges from 0 to 1. In this way, the two distillation loss functions of SKL and SRKL can be flexibly combined to give full play to their respective advantages.

[0085] The above conventional cross entropy loss and dynamic adaptive distillation loss are weighted and calculated to get the final loss result. The obtained loss value is sent back to the optimizer, and the gradient calculation of back propagation is performed to update and optimize the model weights. After all training data are iterated, the final model weights are saved to get the student model trained by distillation.

[0086] In the above implementation, preferably, the model compression method based on dynamic adaptive distillation also includes:

[0087] The trained student model is tested and verified to ensure that the student model can achieve the expected prediction results. The specific process includes:

[0088] The transformer library is used to record the trained student model and word segmenter, and the word segmenter is used to segment the actual test data to obtain the input word segmentation ID of the student model;

[0089] The obtained word segmentation ID is input into the student model for model reasoning, and the prediction results are compared with the expected labels. The actual effect of the model is evaluated based on the prediction results of all test samples and the expected labels, and the prediction results of the student model are evaluated and verified.

[0090] like Figure 2 As shown, the present invention further proposes a model compression system based on dynamic adaptive distillation, which applies the model compression method based on dynamic adaptive distillation disclosed in any one of the above embodiments, including:

[0091] The training data acquisition module 1 is used to acquire common public data related to the large model and filter out the training data set;

[0092] Teacher model training module 2 is used to input the training data set into the existing large model for fine-tuning training according to the open source framework, and use the fine-tuned model as the teacher model;

[0093] The student model training module 3 is used to determine the student model and determine the training state of the student model by adopting a partial weight freezing strategy;

[0094] The distillation loss training module 4 is used to input the training data into the student model and the teacher model for forward propagation respectively, and calculate the dynamic adaptive distillation loss and update the optimization weights with the dynamic adaptive distillation loss function combining SKL and SRKL to complete the distillation training of the student model and realize the compression from the teacher model to the student model.

[0095] In the above-mentioned embodiment, preferably, the model compression system based on dynamic adaptive distillation also includes a model testing and verification module 5, which is used to test and verify the trained student model to determine whether the student model can achieve the expected prediction result.

[0096] The present invention also proposes an electronic device, which includes a memory and a processor, wherein the memory is used to store at least one instruction, and the processor is used to execute at least one instruction to implement the model compression method based on dynamic adaptive distillation as disclosed in any one of the above embodiments.

[0097] According to the model compression system and electronic device based on dynamic adaptive distillation disclosed in the above-mentioned embodiments, the functions to be implemented correspond to the various steps of the model compression method based on dynamic adaptive distillation disclosed in the above-mentioned embodiments. During the implementation process, operations are performed with reference to the above-mentioned embodiments, which will not be repeated here.

[0098] According to the model compression method, system and device based on dynamic adaptive distillation disclosed in the above-mentioned embodiments, the compression process of the model is explained by taking an information extraction scenario as an example.

[0099] 1) Construct training data, including three types of content: task instructions, input and output. The task instruction is "generate appropriate answers based on given information and questions". The input consists of relevant information and questions, and the output is the corresponding standard answer.

[0100] 2) Send the above data to the Internlm2-7B model for fine-tuning training to obtain the teacher model.

[0101] 3) For the Internlm2-1.8B student model, define the training status of its weights, that is, the first and last 8 layers are free for training, every 8 layers in the middle are free for training, and the remaining layers are frozen. ;

[0102] 4) The training data is sent to the student and teacher models for forward propagation, and the dynamic adaptive distillation loss is calculated, and the weights are updated and optimized based on it.

[0103] 5) After completing the training of all the data, save the model and perform actual test evaluation on the model.

[0104] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A model compression method based on dynamic adaptive distillation, characterized in that: include: Obtain common public data related to the large model and filter it to obtain a training data set; According to the open source framework, the training data set is input into the existing large model for fine-tuning training, and the fine-tuned model is used as the teacher model; Determine a student model, and use a partial weight freezing strategy to determine a training state of the student model; The training data are respectively input into the student model and the teacher model for forward propagation, and the dynamic adaptive distillation loss function combining SKL and SRKL is used to calculate the dynamic adaptive distillation loss and update the optimization weights to complete the distillation training of the student model and realize the compression from the teacher model to the student model.

2. The model compression method based on dynamic adaptive distillation according to claim 1, characterized in that: Also includes: The trained student model is tested and verified to determine whether the student model can achieve the expected prediction results.

3. The model compression method based on dynamic adaptive distillation according to claim 1 or 2, characterized in that: The process of obtaining common public data related to the large model and screening to obtain a training data set includes: Collecting common public data related to the large model, and filtering out high-quality data from the common public data, wherein the common public data includes data on reading comprehension, information extraction, text summarization, machine translation, and intelligent question answering; The high-quality data is saved as a json file in the form of three key-value pairs of "instruction", "input" and "output" as the training data set, where "instruction" represents the task instructions given to the model, "input" represents the text content input into the model, and "output" represents the standard response result of the model.

4. The model compression method based on dynamic adaptive distillation according to claim 3, characterized in that: The training data set is input into the existing large model for fine-tuning training according to the open source framework, and the fine-tuned model is used as the teacher model. The specific process includes: The training data set is input into Internlm2-7B, and fine-tuning training is performed according to the open source framework Huggingface. The fine-tuning training includes word segmentation processing of the original text, parallelization of data and model, forward propagation and calculation of loss value, reverse gradient calculation and update, and model weight preservation; The fine-tuned Internlm2-7B model is used as the teacher model.

5. The model compression method based on dynamic adaptive distillation according to claim 4, characterized in that: The student model is determined, and the training state of the student model is determined by using a partial weight freezing strategy. The specific process includes: The Internlm2-1.8B model is used as the student model; The training state of the student model is determined by adopting a partial weight freezing strategy in which the first and last ends and some layers in the middle are released from training and the remaining layers are frozen.

6. The model compression method based on dynamic adaptive distillation according to claim 5, characterized in that: The training data is respectively input into the student model and the teacher model for forward propagation, and the dynamic adaptive distillation loss function combining SKL and SRKL is used to calculate the dynamic adaptive distillation loss and update the optimization weight. The specific process includes: Initialize the hyperparameters of the distillation training, load the training data and the word segmenter of the student model, and segment the training data; Loading the student model, and training the student model according to the training state; The teacher model is loaded and set to the model verification state, and each batch of the training data is respectively input into the student model and the teacher model for forward propagation to obtain the corresponding output probability; Performing conventional loss function calculation and distillation loss calculation on the output probabilities of the student model and the teacher model respectively, wherein the conventional loss calculation represents the calculation of the cross entropy loss function on the prediction result and the label, and the distillation loss calculation is obtained based on the skewed KL divergence loss SKL and the skewed reverse KL divergence loss SRKL, the SKL loss uses parameters to control the mixing ratio of the distribution of the student model and the teacher model, and the SRKL loss is calculated by exchanging the output probabilities of the student model and the teacher model; The SKL loss and the SRKL loss are combined using a cosine function and the number of iterations to obtain a dynamic adaptive distillation loss. The conventional loss function and the dynamic adaptive distillation loss are weighted and calculated, and the model weights are updated and optimized as the number of iterations increases, so that the output probability of the student model is close to the output probability distribution of the teacher model to imitate the prediction method of the teacher model.

7. The model compression method based on dynamic adaptive distillation according to claim 2, characterized in that: The specific process of testing and verifying the trained student model includes: The transformer library is used to record the trained student model and word segmenter, and the word segmenter is used to segment the actual test data; The obtained word segmentation ID is input into the student model for model reasoning, the prediction result is compared with the expected label, and the prediction result of the student model is evaluated and verified.

8. A model compression system based on dynamic adaptive distillation, characterized in that: Applying the model compression method based on dynamic adaptive distillation as described in any one of claims 1 to 7, comprising: The training data acquisition module is used to obtain common public data related to the large model and filter it to obtain the training data set; A teacher model training module is used to input the training data set into the existing large model for fine-tuning training according to the open source framework, and use the fine-tuned model as the teacher model; A student model training module, used to determine a student model and determine a training state of the student model using a partial weight freezing strategy; The distillation loss training module is used to input the training data into the student model and the teacher model respectively for forward propagation, calculate the dynamic adaptive distillation loss and update the optimization weights using the dynamic adaptive distillation loss function combining SKL and SRKL, complete the distillation training of the student model, and realize the compression from the teacher model to the student model.

9. The model compression system based on dynamic adaptive distillation according to claim 8, characterized in that: It also includes a model testing and verification module, which is used to test and verify the trained student model to determine whether the student model can achieve the expected prediction results.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory is used to store at least one instruction, and the processor is used to execute the at least one instruction to implement the model compression method based on dynamic adaptive distillation as described in any one of claims 1 to 7.

Images