Model distillation method, apparatus, device, and storage medium

Abstract

The present disclosure provides a model distillation method, device, equipment and storage medium, relates to the technical field of computers, in particular to the technical field of model training, knowledge distillation and large language model. The specific implementation scheme is as follows: inputting the same batch of training data in the first training data set into a first model, a second model, a third model and a student model respectively; determining a target training result according to the prediction results output by the first model, the second model and the third model; and training the student model according to the loss value between the prediction result of the student model and the target training result to obtain the student model used for executing a target natural language processing task. According to the scheme of the present disclosure, the training cost and the computing resource consumption of knowledge distillation can be effectively reduced, the distillation training efficiency is significantly improved, and the occupation of server computing resources is reduced.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of model training, knowledge distillation, and large language model technology. Background Technology

[0002] The training and inference processes of a model require significant computational, memory, and energy costs. To reduce the computational cost and deployment resources of models, the industry has researched a series of model compression algorithms, including quantization, pruning, and knowledge distillation. Among these, knowledge distillation is a notable method, widely used to compress large, computationally intensive models into smaller, more manageable models, thereby reducing inference costs and memory usage while maintaining model capabilities.

[0003] In knowledge distillation, large and small models need to be paired for training. Typically, the model with more parameters is considered the teacher network, and the model with fewer parameters is considered the student network. The prerequisite for using the knowledge distillation algorithm is that a larger, higher-performing model is needed as the teacher network to guide the student models in distillation training. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, and storage medium for model distillation.

[0005] According to one aspect of this disclosure, a model distillation method is provided, comprising:

[0006] Input the same batch of training data from the first training dataset into the first model, the second model, the third model, and the student model respectively;

[0007] Based on the prediction results output by the first, second, and third models, the target training result is determined; and

[0008] The student model is trained based on the loss value between the prediction results of the student model and the target training results to obtain a student model for performing the target natural language processing task.

[0009] According to another aspect of this disclosure, a model distillation apparatus is provided, comprising:

[0010] The input module is used to input the same batch of training data from the first training dataset into the first model, the second model, the third model, and the student model respectively;

[0011] The determination module is used to determine the target training result based on the prediction results output by the first model, the second model, and the third model.

[0012] The training module is used to train the student model based on the loss value between the student model's prediction results and the target training results, so as to obtain a student model for performing the target natural language processing task.

[0013] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0014] At least one processor; and

[0015] The memory is communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0017] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0018] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0019] The model distillation method proposed in this disclosure can effectively reduce the training cost and computational resource consumption of knowledge distillation, improve distillation training efficiency, and reduce the occupation of server computing resources.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0022] Figure 1 This is a schematic flowchart of a model distillation method provided according to an embodiment of the present disclosure;

[0023] Figure 2 This is a flowchart illustrating the implementation of a training method for a model distillation method according to an embodiment of the present disclosure.

[0024] Figure 3 This is a schematic diagram of the composition of a model distillation apparatus according to an embodiment of the present disclosure;

[0025] Figure 4This is a block diagram of an electronic device used to implement the model distillation method of the embodiments of this disclosure. Detailed Implementation

[0026] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0027] In related technologies, knowledge distillation algorithms can be broadly categorized into three types: first, data distillation, which uses a large model to generate a chain-like solution process to supplement training data and train the model; second, feature distillation, which uses the internal parameters or output results of a large model to guide the training of a smaller model; and third, distillation based on feedback from the large model, which uses the large model to provide positive or negative feedback to the generated results of the smaller model to improve the quality of the smaller model's output. In practical business scenarios, there is often a lack of a teacher network that performs well on downstream tasks, and training a teacher network with a large number of parameters is quite costly. This makes knowledge distillation difficult to apply in many scenarios.

[0028] To at least partially address one or more of the aforementioned problems and other potential issues, embodiments of this disclosure provide a low-cost, high-efficiency proxy distillation algorithm. Utilizing the technical solutions of embodiments of this disclosure, the problem of needing to train a teacher network for knowledge distillation can be effectively solved, and distillation training efficiency can be significantly improved, enhancing the usability of knowledge distillation in practical business applications.

[0029] Figure 1 This is a schematic flowchart of a model distillation method provided according to an embodiment of this disclosure. Figure 1 As shown, the method includes at least the following steps:

[0030] S101. Input the same batch of training data from the first training dataset into the first model, the second model, the third model, and the student model respectively.

[0031] Among them, the first model and the third model are pre-trained models, and the third model has more parameters than the first model; the second model is obtained by further training based on the pre-trained model.

[0032] In this embodiment, a pre-trained model refers to a model that has undergone preliminary training on a large-scale dataset. This process is typically unsupervised learning without specific task labels. In the field of large language models, pre-trained models learn the basic structure, syntax, vocabulary, and semantic features of language by processing large amounts of text data, generating model parameters rich in linguistic knowledge. The pre-training process generally does not rely on task-specific labels but learns language representations through language modeling tasks (such as predicting the next word or filling in words in a mask). Through training on large-scale data, the pre-trained model learns general language features and knowledge, which can be transferred to specific downstream tasks. Therefore, a pre-trained model can be understood as a model that has not been optimized for a specific downstream task.

[0033] In practical applications, pre-trained models are typically used as base models, which are then fine-tuned for specific tasks to adapt to their requirements. For example, models such as BERT and GPT-3 can be fine-tuned for tasks like classification, generation, and question answering after pre-training.

[0034] Both the first and third models are pre-trained models without fine-tuning. The third model has a larger number of parameters and can be used as a teacher model. The second model is obtained by further training based on the pre-trained models, and the parameter size of the second model can be the same as that of the first model. In other words, the second model can be obtained by further training based on the first model.

[0035] S102. Determine the target training result based on the prediction results output by the first model, the second model, and the third model.

[0036] In this embodiment, the first model has a small number of parameters and is not optimized for a specific downstream task; the second model has a small number of parameters but has been optimized; and the third model is a teacher network with a large number of parameters and is not optimized for a specific downstream task. Using the prediction results of the first and second models, adjustments can be made to the prediction results of the third model, so that the prediction results of the third model approach or reach the prediction results of the model after optimization for a specific downstream task. Therefore, the target training result can be considered as the prediction result of the model with a larger number of parameters and superior performance.

[0037] S103. Train the student model based on the loss value between the prediction results of the student model and the target training results to obtain a student model for performing the target natural language processing task.

[0038] The target natural language processing task includes at least one of the following: text classification, sequence labeling, text generation, text translation, text summarization, mathematical problem solving, question answering system, information retrieval, and code understanding and generation.

[0039] In this embodiment of the disclosure, the target training result can be regarded as the prediction result of the high-performing teacher model, and therefore can be used to guide the student model to perform distillation training.

[0040] According to the solution of this disclosure embodiment, without needing to train a teacher network with a large number of parameters, only a small model with fewer parameters can be trained, and the large model can be used as a teacher model to distill the student model. This effectively reduces the training cost and computational resource consumption of knowledge distillation of large language models, significantly improves distillation training efficiency, and enhances the usability of knowledge distillation in actual business.

[0041] In one possible implementation, the method of this disclosure embodiment further includes the step of:

[0042] The pre-trained model is supervisedly fine-tuned using the second training dataset to obtain the second model. The first and second training datasets contain training data of the same task type.

[0043] In this embodiment of the disclosure, before executing S101 above, a pre-trained model with a small number of parameters is first supervised fine-tuned (SFT) on the downstream task using a second training dataset containing manually labeled results. The second model obtained after SFT is regarded as the tuned model, and the original pre-trained model (first model) is regarded as the untuned model. The parameter size of the second model is the same as that of the first model. The first training dataset and the second training dataset contain training data of the same task type.

[0044] It should be noted that the target natural language processing task can also be other downstream tasks in the field of large language models.

[0045] According to the scheme of this disclosure, unlike ordinary distillation methods that require fine-tuning of the teacher network on downstream tasks, this disclosure first trains a small model on the downstream task and then fine-tunes it through a proxy, thus obtaining a large model that performs well on the downstream task as the teacher model at low cost.

[0046] In one possible implementation, step S102, based on the prediction results of the first model, the second model, and the third model, determines the target training result, and further includes the following steps:

[0047] S121. Determine the prediction difference value based on the prediction results of the first model and the second model.

[0048] S122. Based on the predicted difference value and the prediction results of the third model, the target training results are obtained.

[0049] In this embodiment of the disclosure, the prediction difference between the fine-tuned model and the unfine-tuned model is calculated during the inference stage, the prediction result of another pre-trained model with a larger number of parameters is calculated under the same input, and then the prediction difference before and after fine-tuning of the small model is superimposed on the output of the large model.

[0050] According to the scheme of this disclosure embodiment, the prediction results of the three models are superimposed so that the final generated effect is similar to the prediction result of the teacher model after SFT.

[0051] In one example, the model distillation method for text generation tasks includes at least the following steps:

[0052] Input the same batch of training data from the text training dataset into the first model, the second model, the third model, and the student model respectively.

[0053] Based on the text prediction results of the first and second models, the difference between their vectors is calculated to determine the prediction difference value.

[0054] The target text result is obtained based on the predicted difference value and the text prediction result of the third model.

[0055] The student model is trained based on the loss value between the student model's output and the target text result to obtain a student model for performing text generation tasks.

[0056] In one possible implementation, step S122, based on the predicted difference value and the prediction result of the third model, obtains the target training result, and further includes the following steps:

[0057] S122a. Multiply the predicted difference value by the first balance coefficient to obtain the adjustment value.

[0058] S122b: Add the adjustment value to the prediction result of the third model to obtain the target training result.

[0059] In this embodiment, surrogate fine-tuning is applied to the training phase. During training, each batch of training samples is input, and the forward outputs of the fine-tuned model, the unfine-tuned model, and the teacher network are calculated. Then, the output difference between the fine-tuned model and the unfine-tuned model is superimposed on the output of the teacher network. At this point, the output of the teacher network can be approximated as its SFT output. The specific formula is as follows:

[0060] M teacher = M teacher +(Mtuned -M untuned )*α (1)

[0061] Among them, M teacher M represents the output of the teacher network. tuned M represents the output of the fine-tuning network. untuned This represents the output of the unadjusted network, where α is the first balancing coefficient.

[0062] By adjusting the first balancing coefficient, the impact of the prediction difference between the fine-tuned model and the unfine-tuned model on the final result can be amplified or reduced. The first balancing coefficient can also be understood as influencing the model's learning rate of the fine-tuned prediction characteristics. A larger balancing coefficient allows the model to learn the fine-tuned characteristics faster, while a smaller balancing coefficient makes the learning process smoother and slower.

[0063] According to the scheme of this disclosure embodiment, the prediction difference before and after fine-tuning of the small model is superimposed on the output of the large model to realize surrogate fine-tuning. This can obtain a large model that performs well in downstream tasks with a lower training cost, reducing the training cost of the large model to the training cost of the small model.

[0064] In one possible implementation, step S121, based on the prediction results of the first model and the second model, determines the prediction difference value, and further includes the following steps:

[0065] S121a, Normalize the prediction results of the first model and the prediction results of the second model respectively.

[0066] S121b. Take the difference between the normalized results of the two as the predicted difference value.

[0067] In this embodiment, the output of the teacher network during proxy fine-tuning is prone to individual token disorder, resulting in garbled characters. To obtain the output of the teacher network proxy fine-tuning more stably and accurately, the model output can be normalized using logit first, and then the teacher network can be fine-tuned using proxy. The above formula (1) can be modified as follows:

[0068] M teacher =M teacher +(Norm(M tuned )-Norm(M untuned ))*α(2)

[0069] Where Norm(·) represents the normalization function.

[0070] According to the scheme of this disclosure, by performing logit normalization and then proxy fine-tuning, the difference in numerical ratio between the output results of large and small models can be reduced.

[0071] In one possible implementation, step S1211 normalizes the prediction results of the first model and the prediction results of the second model, respectively, and further includes the following steps:

[0072] Determine the first set of prediction results and the second set of prediction results output by the first model and the second model for the same batch of training data.

[0073] Determine the first mean and first standard deviation of the first set of prediction results, and the second mean and second standard deviation of the second set of prediction results.

[0074] Calculate the first difference between each prediction result in the first prediction result set and the first mean, and then calculate the quotient of the first difference and the first standard deviation.

[0075] Calculate the second difference between each prediction result in the second prediction result set and the second mean, and then calculate the quotient of the second difference and the second standard deviation to project the prediction results of the first model and the second model into the same Gaussian distribution range with a mean of 0.

[0076] In this embodiment of the disclosure, the formula for the normalization function is as follows:

[0077]

[0078] Where Norm(·) represents the normalization function, mean(·) represents taking the mean, and std(·) represents taking the standard deviation.

[0079] According to the scheme of this disclosure embodiment, the logit output of the model is projected onto the same Gaussian-like distribution range with a mean of 0, which can more accurately adjust the distribution of the teacher network output.

[0080] In one possible implementation, step S122a multiplies the predicted difference value by a first balancing coefficient to obtain an adjusted value, and further includes the following steps:

[0081] Determine the standard deviation of the prediction results of the third model to obtain the first balance coefficient.

[0082] The adjusted value is obtained by multiplying the predicted difference value by the first balance coefficient.

[0083] The balancing coefficient plays a regulatory and control role in the surrogate distillation algorithm. By setting the balancing coefficient reasonably, the influence of the prediction difference between the fine-tuned model and the unfine-tuned model on the output of the teacher model can be balanced, thereby achieving more stable and efficient model training.

[0084] In this embodiment, the standard deviation of the prediction results of the third model is used as the first balance coefficient, and the above formula (2) is further modified as follows:

[0085] Mteacher =M teacher +(Norm(M tuned )-Norm(M untuned ))*std(M teacher )

[0086] Where Norm(·) represents the normalization function, mean(·) represents taking the mean, and std(·) represents taking the standard deviation.

[0087] According to the scheme of this embodiment, the standard deviation of the teacher model's prediction results is used as the first balancing coefficient. This first balancing coefficient can be dynamically adjusted to mitigate the impact of fine-tuning differences based on the uncertainty of the teacher model's output. This allows the first balancing coefficient to adapt to different inputs and prediction scenarios. A larger standard deviation indicates higher uncertainty in the teacher model's prediction results, necessitating an increase in the balancing coefficient to emphasize the differences in the fine-tuning model. Conversely, a smaller standard deviation indicates more stable prediction results, allowing a decrease in the balancing coefficient to reduce the impact of fine-tuning differences.

[0088] In one possible implementation, the student model uses the second model as the initial model.

[0089] In this embodiment, since the student model is trained on the basis of the fine-tuned model, it already has certain downstream task knowledge. The student model inherits the optimization characteristics and performance of the fine-tuned model on downstream tasks, and can perform better in further training, and can also accelerate the convergence process of the model.

[0090] According to the scheme of this disclosure embodiment, training starts from fine-tuning the model, making full use of the learned knowledge, greatly reducing training time and reducing instability.

[0091] In one possible implementation, step S103 trains the student model based on the loss value between the student model's prediction results and the target training results, and further includes the following steps:

[0092] S131. Determine the distillation loss function based on the prediction results of the student model and the target training results.

[0093] S132. Determine the total loss function based on the product of the distillation loss function and the second balance coefficient, as well as the preset task loss function.

[0094] S133. Based on the loss value obtained from the total loss function, backpropagate the student network and update the parameters of the student network to train the student model.

[0095] In this embodiment, the preset task loss function can be understood as the preset loss function for downstream tasks, also known as the task-specific loss function, which is a loss function defined for a specific task. This loss function measures the difference between the model's predicted result and the true label to guide the model's optimization process. Different downstream tasks have different preset loss functions, and the specific choice depends on the nature and objective of the task. Below are some common downstream tasks and their corresponding preset loss functions:

[0096] 1. Text classification:

[0097] Loss function: Cross-Entropy Loss

[0098] Description: Used for multi-class classification problems, it measures the difference between the model's output probability distribution and the true label distribution.

[0099] 2. Sequence labeling (e.g., named entity recognition):

[0100] Loss function: Conditional Random Field Loss (CRF Loss) or Cross-Entropy Loss

[0101] Description: In annotation tasks, cross-entropy loss is often used for independent annotations, while CRF loss considers the dependencies between annotations.

[0102] 3. Text generation (such as machine translation and summarization):

[0103] Loss function: Cross-Entropy Loss

[0104] Description: Measures the difference between the generated sequence and the target sequence, and is often used in sequence-to-sequence generation tasks.

[0105] 4. Question and Answer System:

[0106] Loss function: Cross-Entropy Loss or Mean Squared Error Loss (MSE Loss)

[0107] Description: For extractive question answering, cross-entropy loss is used to predict the start and end positions of the answer; for generative question answering, cross-entropy loss is used to measure the accuracy of the generated answer.

[0108] like Figure 2 As shown, in this embodiment of the disclosure, during training, each batch of training samples is input, and the forward outputs of the fine-tuned network, the un-fine-tuned network, and the teacher network are calculated. Then, the output difference between the fine-tuned network and the un-fine-tuned network is superimposed on the output of the teacher network. The distillation loss is then calculated between the surrogate fine-tuned optimized teacher network output and the student network output, and distillation training is performed.

[0109] The entire training algorithm is shown below:

[0110] Input: : Teacher network; : Fine-tuned network; : Unfine-tuned network; : Student network; M T M t M ut M S

[0111] D: Training dataset; T: Number of training iterations

[0112] 1: for t=1→T do

[0113] 2: X = Sample(D) / / Sample a batch of data from the training set

[0114] 3: T = M T (X)+(Norm(M t (X))-Norm(M ut (X)))*std(M T (X))

[0115] 4: S = M S (X)

[0116] 5:L distill =Distillation_loss(S,T) / / Calculate distillation loss

[0117] 6:L total =L task +λ*L distill / / Total loss function, where λ is the balance coefficient of distillation loss.

[0118] 7: Backpropagation updates parameters of the student network

[0119] Output:

[0120] M S

[0121] According to the scheme of this disclosure, a large model can be used to distill a small model without training a teacher network with a large number of parameters, effectively reducing the training cost of knowledge distillation for large language models. Experiments on open-source datasets and business data show that agent fine-tuning has a positive gain in accuracy compared to supervised fine-tuning, and can reduce training time by at least 40% compared to ordinary knowledge distillation, reducing the occupation of server computing resources.

[0122] In one possible implementation, at least some parameters of the first, second, and third models are frozen during the training of the student model.

[0123] In this embodiment, the fine-tuned model, the un-fine-tuned model, and the teacher model are frozen, and only the student model is trained. This reduces computational complexity; by freezing the parameters of the fine-tuned, un-fine-tuned, and teacher models, it is unnecessary to calculate the gradients of these models, significantly reducing computational costs and resource consumption. Training only the student model allows for the concentration of computational resources, improving training efficiency and accelerating the training process. Furthermore, freezing the fine-tuned and teacher models ensures the stability of their prediction results, providing a stable source of knowledge to guide the student model and making the knowledge distillation process more reliable.

[0124] According to the embodiments of this disclosure, by freezing some or all of the parameters of the model, efficiency can be improved, resources can be saved, and overfitting can be prevented during the training and fine-tuning of the deep learning model.

[0125] Figure 3 This is a schematic diagram of a model distillation apparatus provided according to an embodiment of the present disclosure. Figure 3 As shown, the device includes at least:

[0126] The input module 301 is used to input the same batch of training data from the first training dataset into the first model, the second model, the third model, and the student model respectively.

[0127] The determination module 302 is used to determine the target training result based on the prediction results output by the first model, the second model, and the third model.

[0128] Training module 303 is used to train the student model based on the loss value between the prediction results of the student model and the target training results, so as to obtain a student model for performing the target natural language processing task.

[0129] The first and third models are pre-trained models, with the third model having more parameters than the first model. The second model is obtained through further training based on the pre-trained models.

[0130] In one possible implementation, the target natural language processing task includes at least one of the following: text classification, sequence labeling, text generation, text translation, text summarization, mathematical problem solving, question answering system, information retrieval, and code understanding and generation.

[0131] In one possible implementation, the device further includes:

[0132] The fine-tuning module is used to perform supervised fine-tuning on the pre-trained model using a second training dataset of the target task type to obtain a second model, wherein the first training dataset and the second training dataset belong to the same task type.

[0133] In one possible implementation, the determining module 302 includes:

[0134] The difference value determination submodule is configured to determine the predicted difference value based on the prediction results of the first model and the second model.

[0135] The result determination submodule is configured to obtain the target training result based on the predicted difference value and the prediction result of the third model.

[0136] In one possible implementation, the result determination submodule is used for:

[0137] The adjusted value is obtained by multiplying the predicted difference value by the first balance coefficient.

[0138] The adjusted value is added to the prediction result of the third model to obtain the target training result.

[0139] In one possible implementation, the difference value determination submodule is used for:

[0140] The prediction results of the first model and the prediction results of the second model are normalized respectively.

[0141] The difference between the normalized results of the two is taken as the predicted difference value.

[0142] In one possible implementation, the difference value determination submodule is further used for:

[0143] Determine the first set of prediction results and the second set of prediction results output by the first model and the second model for the same batch of training data.

[0144] Determine the first mean and first standard deviation of the first set of prediction results, and the second mean and second standard deviation of the second set of prediction results.

[0145] Calculate the first difference between each prediction result in the first prediction result set and the first mean, and then calculate the quotient of the first difference and the first standard deviation.

[0146] Calculate the second difference between each prediction result in the second prediction result set and the second mean, and then calculate the quotient of the second difference and the second standard deviation to project the prediction results of the first model and the second model into the same Gaussian distribution range with a mean of 0.

[0147] In one possible implementation, the result-determining submodule is further used for:

[0148] Determine the standard deviation of the prediction results of the third model to obtain the first balance coefficient.

[0149] The adjusted value is obtained by multiplying the predicted difference value by the first balance coefficient.

[0150] In one possible implementation, the student model uses the second model as the initial model.

[0151] In one possible implementation, the training module 330 is used for:

[0152] The distillation loss function is determined based on the prediction results of the student model and the training results of the target model.

[0153] The total loss function is determined by multiplying the distillation loss function by the second balance coefficient and the preset task loss function.

[0154] The student model will be backpropagated and its parameters updated based on the loss value obtained from the total loss function, thereby enabling the training of the student model.

[0155] In one possible implementation, at least some parameters of the first, second, and third models are frozen during the training of the student model.

[0156] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0157] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0158] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0159] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0160] like Figure 4As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0161] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0162] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the model distillation method. For example, in some embodiments, the model distillation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the model distillation method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the model distillation method by any other suitable means (e.g., by means of firmware).

[0163] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0165] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0166] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0167] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0168] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0169] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0170] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A model distillation method, comprising: The same batch of training data from the first training dataset is input into the first model, the second model, the third model, and the student model, respectively; wherein, the first model and the third model are pre-trained models, and the number of parameters of the third model is greater than the number of parameters of the first model; the second model is obtained by training based on the first model; Based on the prediction results output by the first model, the second model, and the third model, the target training result is determined, including: Based on the prediction results of the first model and the second model, the prediction difference value is determined, including: The prediction results of the first model and the prediction results of the second model are normalized respectively; The difference between the normalized results of the two is taken as the predicted difference value; Multiplying the predicted difference value by the first balance coefficient yields the adjustment value, including: Determine the standard deviation of the prediction results of the third model to obtain the first balance coefficient; The predicted difference value is multiplied by the first balance coefficient to obtain the adjustment value; The adjusted value is added to the prediction result of the third model to obtain the target training result; and The student model is trained based on the loss value between the prediction results of the student model and the target training results to obtain a student model for performing the target natural language processing task. The target natural language processing task includes at least one of the following: text classification, text generation, text translation, text summarization, mathematical problem solving, or code understanding and generation; The normalization of the prediction results of the first model and the prediction results of the second model includes: Determine the first set of prediction results and the second set of prediction results output by the first model and the second model for the same batch of training data; Determine the first mean and first standard deviation of the first set of prediction results, and the second mean and second standard deviation of the second set of prediction results; Calculate the first difference between each prediction result in the first prediction result set and the first mean, and then calculate the quotient of the first difference and the first standard deviation; Calculate the second difference between each prediction result in the second prediction result set and the second mean, and then calculate the quotient of the second difference and the second standard deviation to project the prediction results of the first model and the second model into the same Gaussian distribution range with a mean of 0.

2. The method according to claim 1, further comprising: The pre-trained model is supervisedly fine-tuned using the second training dataset to obtain the second model, wherein the first training dataset and the second training dataset contain training data of the same task type.

3. The method according to claim 1, wherein, The student model uses the second model as the initial model.

4. The method according to claim 1, wherein, The step of training the student model based on the loss value between the prediction result of the student model and the target training result includes: Based on the prediction results of the student model and the training results of the target, the distillation loss function is determined; The total loss function is determined based on the product of the distillation loss function and the second balance coefficient, as well as the preset task loss function; The student model is backpropagated based on the loss value obtained from the total loss function, and the parameters of the student model are updated to train the student model.

5. The method according to claim 1 or 4, wherein, During the training of the student model, at least some parameters of the first model, the second model, and the third model are frozen.

6. A model distillation apparatus, comprising: The input module is used to input the same batch of training data from the first training dataset into the first model, the second model, the third model, and the student model respectively; wherein, the first model and the third model are pre-trained models, and the number of parameters of the third model is greater than the number of parameters of the first model; the second model is obtained by further training based on the pre-trained model; The module to be determined includes: The difference value determination submodule is configured to determine the predicted difference value based on the prediction results of the first model and the second model; The result determination submodule is configured to obtain the target training result based on the predicted difference value and the prediction result of the third model; The difference value determination submodule is used for: The prediction results of the first model and the prediction results of the second model are normalized respectively; The difference between the normalized results of the two is taken as the predicted difference value; The result determination submodule is further used for: The predicted difference value is multiplied by the first balance coefficient to obtain the adjustment value; The adjustment value is added to the prediction result of the third model to obtain the target training result; The difference value determination submodule is further used for: Determine the first set of prediction results and the second set of prediction results output by the first model and the second model for the same batch of training data; Determine the first mean and first standard deviation of the first set of prediction results, and the second mean and second standard deviation of the second set of prediction results; Calculate the first difference between each prediction result in the first prediction result set and the first mean, and then calculate the quotient of the first difference and the first standard deviation; Calculate the second difference between each prediction result in the second prediction result set and the second mean, and then calculate the quotient of the second difference and the second standard deviation to project the prediction results of the first model and the second model into the same Gaussian distribution range with a mean of 0. The result determination submodule is further used for: Determine the standard deviation of the prediction results of the third model to obtain the first balance coefficient; The predicted difference value is multiplied by the first balance coefficient to obtain the adjustment value; The training module is used to train the student model based on the loss value between the prediction result of the student model and the target training result, so as to obtain a student model for performing the target natural language processing task. The target natural language processing task includes at least one of the following: text classification, text generation, text translation, text summarization, mathematical problem solving, or code understanding and generation.

7. The apparatus according to claim 6, further comprising: The fine-tuning module is used to perform supervised fine-tuning on the pre-trained model using the second training dataset to obtain the second model, wherein the first training dataset and the second training dataset contain training data of the same task type.

8. The apparatus according to claim 6, wherein, The student model uses the second model as the initial model.

9. The apparatus according to claim 6, wherein, The training module is used for: Based on the prediction results of the student model and the training results of the target, the distillation loss function is determined; The total loss function is determined based on the product of the distillation loss function and the second balance coefficient, as well as the preset task loss function; The student model is backpropagated based on the loss value obtained from the total loss function, and the parameters of the student model are updated to train the student model.

10. The apparatus according to claim 6 or 9, wherein, During the training of the student model, at least some parameters of the first model, the second model, and the third model are frozen.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.

Images