Knowledge distillation methods and systems

Abstract

The present invention discloses a knowledge distillation method and system, the method comprising: obtaining the output of a teacher model on a training set sample as a soft-labeled training set; designing a student model architecture and a layer correspondence between the student model and the teacher model; dividing the training set or the soft-labeled training set into super batches, and performing a first round of white-box knowledge distillation based on principal component analysis; performing a second round of black-box knowledge distillation based on the soft-labeled training set; and testing the student model performance. The present invention uses principal component analysis to correspond the teacher model and the student model with different hidden layer sizes and architectures to each other to complete the method of white-box knowledge distillation, so that white-box knowledge distillation can be applied to more student models, which is convenient for further improving the compression ratio. The distillation method of the present invention has better performance than both white-box knowledge distillation and black-box knowledge distillation.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a knowledge distillation method and system. Background Art

[0002] In the field of software engineering, pre-trained language models such as CodeBERT and CodeT5 perform significantly better than traditional models on many code tasks. Such models usually need to be pre-trained on a code corpus and then fine-tuned on specific tasks (such as code review, code summary, etc.). After these two steps of training, they can be deployed in specific applications. However, such models contain a large number of parameters, which requires sufficient computing resources to support them during the application process. Ordinary home computers can no longer meet the needs of such models, so software engineers can only use such services through cloud platforms. Not only is it difficult to optimize the model for the application scenario, but unpredictable confidentiality leaks may also occur during the development process.

[0003] Knowledge distillation is a commonly used model compression technology that can not only significantly reduce the size of the model, but also speed up the calculation speed of the inference stage, so that the generated small model can be deployed locally. For a specific task, knowledge distillation accepts a pre-trained and fine-tuned large language model (teacher model) and the dataset of the task as input, and generates a small model (student model) of a specific architecture and size according to demand. The distillation process can be roughly divided into black-box distillation and white-box distillation. The former only uses the logit value (i.e., the unstandardized probability distribution) predicted by the teacher model on the sample, while the latter uses the internal parameters of the teacher model or the intermediate results generated in addition to the logit value.

[0004] Black-box distillation is a common knowledge distillation method that is relatively simple to implement. For classification tasks, you only need to collect the logit values ​​(soft labels, i.e., the probability distribution predicted by the model) output by the teacher model on the training set and replace the reference classification results (hard labels, i.e., the probability distribution annotated by the data set) of the original data set, and then directly train the student model. For generation tasks, you can also replace the reference output with the generation results output by the teacher model on the training set, and train the student model. When using this method, the student model architecture can be different from the teacher model, so it is more applicable, but because the "knowledge" provided by the teacher model is not fully utilized, the distillation effect is not ideal.

[0005] White-box distillation requires collecting the intermediate results generated by the teacher model on the training set during training, comparing them with the intermediate results corresponding to the student model, and calculating the loss function. This technology can be directly distilled without replacing the data set, or it can be combined with black-box distillation, that is, the loss function simultaneously considers the gap between the intermediate results and logit values ​​of the teacher and student models. This method has a good distillation effect, but the existing technology requires that the hidden layer size of the teacher model architecture and the student model architecture be the same, which severely limits the choice of the student model; and because the hidden layer size of the teacher model is usually large, it also limits the compression ratio. Summary of the invention

[0006] In view of the above problems, the present invention provides an improved knowledge distillation method and system, which aims to improve the compression ratio while combining the advantages of white box distillation and black box distillation.

[0007] According to an embodiment of the present disclosure, an improved knowledge distillation method is provided, the method comprising the following steps:

[0008] Get the output of the teacher model on the training set samples, replace the original labels on the training set samples, and obtain the soft-labeled training set;

[0009] Constructing the number of layers and architecture of the student model, and setting the layer correspondence between the student model and the teacher model;

[0010] Divide the training set or the soft-labeled training set into a number of super-batches, and each super-batch is further divided into a number of batches;

[0011] Performing a first round of white-box knowledge distillation on the student model, during which principal component analysis is performed on the output results of each hidden layer of the teacher model for each super-batch sample in the training set, traversing the layer correspondence between the student model and the teacher model based on the correspondence between batch samples and super-batches, calculating the loss function of the student model on each batch of samples based on the layer correspondence, optimizing the student model parameters based on the calculation result of the loss function, and testing the performance of the student model using the original validation set after optimization;

[0012] A second round of black-box knowledge distillation is performed on the student model based on the soft labeled training set, and the loss function of the student model on each batch of samples is calculated using the output result of the student model and the soft labeled training set. The student model parameters are optimized based on the calculation result of the loss function, and the performance of the student model is tested using the original validation set after optimization.

[0013] In one embodiment, in the process of obtaining the output of the teacher model on the training set samples, for classification tasks, the output of the teacher model on the training set samples is the probability distribution of each category; for generation tasks, the teacher model uses a decoding strategy to generate a result output, wherein the decoding strategy is any one of greedy decoding, beam search, and sampling.

[0014] In one embodiment, the layer correspondence between the student model and the teacher model is set to be a linear correspondence, that is, the first layer of the student model The layer corresponds to the first This method requires the number of student model layers is the number of teacher model layers Another layer correspondence is to make the first The layer corresponds to the first layer.

[0015] In one embodiment, a first round of white-box knowledge distillation is performed on the student model, and the specific process includes the following steps:

[0016] Assume that the super batch has Samples, The samples are , using the teacher model to generate a representation function ,in for The matrix of represents the length of the teacher model output sequence, Represents the intermediate result of the hidden layer of the teacher model, and the matrix Transform to Vector , then for super batches samples, for Matrix of

[0017] right After singular value decomposition, if , U and V both represent orthogonal matrices, represents a singular value diagonal matrix, then The main component of , intercept Before Column transformation into a 3D tensor , represents the output sequence length of the student model, Represents the intermediate results of the hidden layer of the student model;

[0018] Train the student model batch by batch. If the first Samples Corresponding to the first The representations produced by the student model , then the loss function on this sample is , where MSE stands for mean square error, and the formula is (x,y is RxC matrix), are the parameters of the student model.

[0019] In one embodiment, for the classification task, a second round of black box knowledge distillation based on the soft labeled training set is performed on the student model, specifically comprising the following steps:

[0020] Assume that the jth sample in the batch is , using the student model to generate the probability distribution of each category is , then the sample The loss function is , where CE is the cross entropy, , represents the total number of categories, The teacher model pairs of samples in the soft-labeled training set The probability distribution of the prediction.

[0021] In one embodiment, for the generation task, a second round of black box knowledge distillation based on the soft labeled training set is performed on the student model, specifically comprising the following steps:

[0022] Traversing samples Each prefix of , using the student model to predict the probability distribution of the next word in the vocabulary , then the loss function of the prefix is , CE is the cross entropy, where for A one-hot vector with 1 in the first position and 0 in other positions.

[0023] In one embodiment, the teacher model is a Transformer model, and the student model is any one of a multilayer perceptron model, a recurrent neural network model, a long short-term memory network model, a gated recurrent unit model and a Transformer model.

[0024] According to another embodiment of the present disclosure, a distillation system includes:

[0025] The soft-label training set acquisition module is used to obtain the output of the teacher model on the training set samples, replace the original labels on the training set samples, and obtain the soft-label training set;

[0026] A student model construction module, used to construct the number of layers and architecture of the student model, and set the layer correspondence between the student model and the teacher model;

[0027] A training set partitioning module, used for partitioning a training set or a soft-labeled training set into a number of super batches, and each super batch is further divided into a number of batches;

[0028] A white-box knowledge distillation module is used to perform a first round of white-box knowledge distillation on the student model. During the first round of white-box knowledge distillation, principal component analysis is performed on the output results of each hidden layer of the teacher model for each super-batch sample in the training set. Based on the correspondence between batch samples and super-batches, the layer correspondence between the student model and the teacher model is traversed, and the loss function of the student model on each batch of samples is calculated based on the layer correspondence. The student model parameters are optimized based on the calculation results of the loss function, and the performance of the student model is tested using the original validation set after optimization;

[0029] A black-box knowledge distillation module is used to perform a second round of black-box knowledge distillation on the student model based on the soft-labeled training set, calculate the loss function of the student model on each batch of samples using the output result of the student model and the soft-labeled training set, optimize the student model parameters based on the calculation result of the loss function, and test the performance of the student model using the original validation set after optimization.

[0030] According to another embodiment of the present disclosure, a knowledge distillation system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the knowledge distillation method described above when executing the program.

[0031] According to another embodiment of the present disclosure, a non-temporary computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the steps of the knowledge distillation method described above.

[0032] The technical solution provided by the embodiments of the present disclosure is: an improved knowledge distillation method and system, which uses principal component analysis to correspond teacher models and student models with different hidden layer sizes and architectures to each other, so as to complete the white-box knowledge distillation method, so that white-box knowledge distillation can be applied to more student models, which is convenient for further improving the compression ratio and can also provide better performance than black-box knowledge distillation; the present invention successively uses two rounds of knowledge distillation of white-box and black-box methods, first uses the intermediate results to optimize the internal parameters of the student model, and then uses the final results to train the classification head of the student model and further fine-tune the parameters, which can further improve the performance of the student model on the basis of a single round of knowledge distillation.

[0033] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present invention and, together with the description, serve to explain the principles of the present invention;

[0035] Figure 1 is a flow chart of the knowledge distillation method according to an embodiment of the present invention;

[0036] Figure 2 Schematic diagram of the corresponding relationship between the teacher model and the student model layer in an embodiment of the present invention;

[0037] Figure 3 Schematic diagram of the white box distillation and black box distillation process in an embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of the structure of a knowledge distillation system in an embodiment of the present invention;

[0039] Figure 5 Schematic diagram of electronic equipment of the knowledge distillation system in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0041] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0042] The present invention provides a method and system for knowledge distillation and provides the following embodiments:

[0043] Example 1 is used to illustrate the knowledge distillation method. Figure 1 As shown, the method comprises the following steps:

[0044] Get the teacher model and the tokenizer it uses. The student model will use the same tokenizer.

[0045] Get the output of the teacher model on the training set samples, replace the original labels on the training set samples, and obtain the soft-labeled training set;

[0046] In the specific implementation process, the training set is preprocessed and each sample If it is a classification task, the teacher model (including the classification head) is used to generate the probability distribution of each category (C is the number of categories); if it is a generation task, the teacher model and the specified decoding strategy are used, including but not limited to greedy decoding, beam search, sampling, to generate the teacher output , the new dataset is called the soft-labeled training set.

[0047] Constructing the number of layers and architecture of the student model, and setting the layer correspondence between the student model and the teacher model;

[0048] In designing the student model, the teacher model is usually Layer Transformer, select the number of layers of the student model The student model architecture can be different from the teacher model, including but not limited to multi-layer perceptron (MLP), recurrent neural network (RNN), long short-term memory network (LSTM), gated recurrent unit (GRU), Transformer, hidden layer size It can also be compared with the hidden layer size of the teacher model Different, the hidden layer refers to the matrix output by each layer of the Transformer network. If the sequence length is M, the hidden layer matrix size is M×h, where h is the hidden layer size, which is the vector dimension of the output of each token position. Similarly, for the RNN series network, the hidden layer refers to the output result of each layer of RNN, and the size is also M×h, and the hidden layer size also refers to h.

[0049] The intermediate results of different layers of the teacher model and the student model need to correspond to each other in order to calculate the loss function and optimize the student model. There are two ways to correspond, such as Figure 2 As shown, Figure 2 In (a), a method of setting the layer correspondence between the student model and the teacher model to be a linear correspondence, that is, the first The layer corresponds to the first Layer, in a preferred embodiment, may require for factors, such as hour Optional , , the student model The layer corresponds to the first layer; Figure 2 In (b), another way is to let the first The layer corresponds to the first layer, represents the number of layers of the teacher model, Indicates the number of layers of the student model.

[0050] Divide the training set or the soft-labeled training set into a number of super-batches, and each super-batch is further divided into a number of batches;

[0051] Divide the training set or soft-labeled training set into several superbatches, as large as memory allows, and further divide each superbatch into several batches.

[0052] Based on the acquisition of the soft-labeled training set, the only difference between the training set and the soft-labeled training set is that the reference output is different, and the input is the same. White-box distillation only uses the training set input, does not use the reference output of the data set, but uses the teacher model hidden layer to calculate the loss, so you can choose either the training set or the soft-labeled training set. The purpose of the super batch is to include as many samples as possible in a single principal component analysis. However, considering that the number of samples in the super batch may be too large due to memory limitations during model calculation, the embodiment of the present application calculates the hidden layers of multiple batches separately, merges them into a super batch, and then performs PCA.

[0053] like Figure 3 As shown, the student model is subjected to a first round of white-box knowledge distillation. During the first round of white-box knowledge distillation, principal component analysis is performed on the output results of each hidden layer of the teacher model for each super-batch sample in the training set. Based on the correspondence between batch samples and super-batches, the layer correspondence between the student model and the teacher model is traversed, and the loss function of the student model on each batch of samples is calculated based on the layer correspondence. The student model parameters are optimized based on the calculation results of the loss function, and the performance of the student model is tested using the original validation set after optimization.

[0054] In the specific implementation process, the first round of white-box knowledge distillation (using intermediate results) is started: in each training period (epoch), the layer correspondence is traversed ,in Represents the student model The layer corresponds to the first layer, then the representation function and Corresponding hidden layers for student and teacher models , , ignoring higher levels of the teacher-student model. Iterate over each superbatch.

[0055] Perform the first round of white-box knowledge distillation on the student model, and perform the following steps on each corresponding layer of the teacher and student models:

[0056] Assume that the superbatch contains Samples, The samples are , using the teacher model to generate hidden layer representations ,in for The matrix ( Output sequence length for the teacher model); the matrix Write down all elements in order to get the vector , so for The matrix of .

[0057] right After performing principal component analysis (PCA) and singular value decomposition (SVD), if (U and V are orthogonal matrices, is a singular value diagonal matrix), then The main component of ,yes If the output sequence length of the student model is Then intercept Before Column, transformed into a three-dimensional tensor .

[0058] Train the student model batch by batch: If the first Samples Corresponding to the first The hidden layer representation generated by the student model , is the characterization function For example, consider the student model Layer It's a sample As input, students The hidden state of the layer. In summary, the sample The loss function is ,in Represents the three-dimensional tensor obtained by the PCA process , It is components (matrix), MSE stands for mean square error, and the formula is (x, y are RxC matrices, R and C represent the size of matrix x and matrix y, , represent the elements in matrix x and matrix y respectively), are the parameters of the student model. After calculating the loss function, back propagation is used to optimize the student model.

[0059] It should be noted that both the super batch and the batch are independent of the teacher and student models. In the embodiment, the entire super batch is applied to the PCA process, and each batch in the super batch is used for the calculation of the teacher model and the student model respectively.

[0060] After each training epoch, the performance of the student model is tested using the original validation set.

[0061] like Figure 3 As shown, the student model is subjected to a second round of black-box knowledge distillation based on the soft labeled training set, the loss function of the student model on each batch of samples is calculated using the output results of the student model and the soft labeled training set, the student model parameters are optimized based on the calculation results of the loss function, and the performance of the student model is tested using the original validation set after optimization.

[0062] In the specific implementation, the second round of black box knowledge distillation uses logit values, and the complete student model is used at this time. Function is the final output of the student model followed by the classification head. In each training epoch, we traverse each batch. Assume that The samples are .

[0063] If it is a classification task, the probability distribution of each category generated by the student model is , then the loss function of this sample is , where CE is the cross entropy, the formula is ( is the total number of categories, , represents two probability distributions), The teacher model pairs of samples in the soft-labeled training set generated in step 2 The probability distribution of the prediction.

[0064] If it is a generation task, the steps are similar to the usual training: traverse each prefix of the sample, assuming that the current processing length is e. , using the student model to predict the probability distribution of the next word in the vocabulary , then the loss function of the prefix is ,in for One-hot vector with 1 in position and 0 in other positions, sample is a token sequence, express The kth token of . Therefore, when traversing For every prefix of , the prefix of length e is The value range of e is ,in Indicates the length of the sample sequence.

[0065] After each training epoch, the performance of the student model is tested using the original validation set.

[0066] After two rounds of training, the original test set is used to test the performance of the student model.

[0067] The validation process after each training period (epoch) and the testing process after training are the same as the validation and testing process in the non-distillation scenario or the samples in the test set. (code or text) is provided to student models, and results are collected for evaluation.

[0068] For classification tasks, the samples Collect the classification results of the student model , and the reference results in the dataset Take the binary classification task as an example and calculate the true positive number. , false positive number , true negative number , false negative number , commonly used indicators include accuracy , recall rate , Accuracy , F1 score Etc. After the calculation is completed, the distillation effect is evaluated.

[0069] For the generation task, the samples Collect the generation results of the student model using the specified decoding strategy (such as greedy decoding, beamsearch, sampling, etc.) , and the reference results in the dataset For comparison, we calculated BLEU, CodeBLEU, ROUGE and other indicators, and summarized them to get the average values ​​of the corresponding indicators on the dataset, and evaluated the distillation effect.

[0070] The second embodiment of the present invention is used to illustrate a distillation system, such as Figure 4As shown, the system 400 includes: a soft-labeled training set acquisition module 410, which is used to obtain the output of the teacher model on the training set samples, replace the original labels on the training set samples, and obtain the soft-labeled training set; a student model construction module 420, which is used to construct the number of layers and architecture of the student model, and set the layer correspondence between the student model and the teacher model; a training set division module 430, which is used to divide the training set or the soft-labeled training set into a number of super batches, and each super batch is further divided into a number of batches; a white-box knowledge distillation module 440, which is used to perform a first round of white-box knowledge distillation on the student model. During the first round of white-box knowledge distillation, principal component analysis is performed on the output results of each hidden layer of the teacher model for each super batch sample in the training set, and based on the batch The method further comprises the following steps: a first step of performing a black-box knowledge distillation module 450, which is used to perform a second round of black-box knowledge distillation on the student model based on the soft labeled training set, calculate the loss function of the student model on each batch of samples based on the layer correspondence, optimize the student model parameters based on the calculation result of the loss function, and use the original validation set to test the performance of the student model after optimization; a second round of black-box knowledge distillation on the student model based on the soft labeled training set, calculate the loss function of the student model on each batch of samples based on the output result of the student model and the soft labeled training set, optimize the student model parameters based on the calculation result of the loss function, and use the original validation set to test the performance of the student model after optimization.

[0071] In addition to the upper module, the system 400 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present disclosure, their illustration and description are omitted here.

[0072] The other specific working processes of the knowledge distillation system 400 refer to the description of the above-mentioned knowledge distillation method embodiment and will not be repeated here.

[0073] Another embodiment is used to illustrate that the knowledge distillation system of the present invention can also be used by Figure 5 The architecture of the computing device shown is implemented. Figure 5 The architecture of the computing device is shown. Figure 5 As shown, a computer system 510, a system bus 530, one or more CPUs 540, an input / output 520, a memory 550, etc. The memory 550 can store various data or files used for computer processing and / or communication and program instructions including the knowledge distillation method of the embodiment executed by the CPU. Figure 5 The architecture shown is only exemplary and can be adjusted according to actual needs when implementing different devices. Figure 5The memory 550 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and modules. One or more CPUs 540 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions, and modules stored in the memory 550, that is, to implement the above-mentioned knowledge distillation method, which includes the following steps:

[0074] Get the output of the teacher model on the training set samples, replace the original labels on the training set samples, and obtain the soft-labeled training set;

[0075] Constructing the number of layers and architecture of the student model, and setting the layer correspondence between the student model and the teacher model;

[0076] Divide the training set into several super batches, and each super batch is further divided into several batches;

[0077] Performing a first round of white-box knowledge distillation on the student model, during which principal component analysis is performed on the output results of each hidden layer of the teacher model for each super-batch sample in the training set, traversing the layer correspondence between the student model and the teacher model based on the correspondence between batch samples and super-batches, calculating the loss function of the student model on each batch of samples based on the layer correspondence, optimizing the student model parameters based on the calculation result of the loss function, and testing the performance of the student model using the original validation set after optimization;

[0078] A second round of black-box knowledge distillation is performed on the student model based on the soft labeled training set, and the loss function of the student model on each batch of samples is calculated using the output result of the student model and the soft labeled training set. The student model parameters are optimized based on the calculation result of the loss function, and the performance of the student model is tested using the original validation set after optimization.

[0079] Of course, the processor of the server provided in the embodiment of the present invention is not limited to executing the method operations described above, but can also execute related operations in the knowledge distillation method provided in any embodiment of the present invention.

[0080] The memory 550 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 550 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 550 may further include a memory remotely arranged relative to one or more CPUs 540, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0081] The input / output 520 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The input / output 520 may also include a display device such as a display screen.

[0082] The embodiment of the present invention also provides a non-temporary computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the knowledge distillation method described in the above embodiment is implemented. The computer-readable storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0083] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0084] The program code contained on the storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF (Radio Frequency), etc., or any suitable combination of the above.

[0085] In addition, other specific working processes of a non-temporary computer-readable storage medium refer to the description of the above-mentioned knowledge distillation method embodiment and will not be repeated here.

[0086] Based on the technical solutions provided by the above embodiments, an improved knowledge distillation method and system use principal component analysis to correspond the teacher model and student model with different hidden layer sizes and architectures to each other, so as to complete the white-box knowledge distillation method, so that white-box knowledge distillation can be applied to more student models, which is convenient for further improving the compression ratio and can also provide better performance relative to black-box knowledge distillation. The present invention successively uses two rounds of knowledge distillation of white-box and black-box methods, first uses the intermediate results to optimize the internal parameters of the student model, and then uses the final results to train the classification head of the student model and further fine-tune the parameters, which can further improve the performance of the student model on the basis of a single round of knowledge distillation.

[0087] In this document, the terms "comprises," "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such step or method.

[0088] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A knowledge distillation method, characterized in that: The method comprises the following steps: Get the output of the teacher model on the training set samples, replace the original labels on the training set samples, and obtain the soft-labeled training set; Constructing the number of layers and architecture of the student model, and setting the layer correspondence between the student model and the teacher model; Divide the training set or the soft-labeled training set into a number of super-batches, and each super-batch is further divided into a number of batches; Performing a first round of white-box knowledge distillation on the student model, during which principal component analysis is performed on the output results of each hidden layer of the teacher model for each super-batch sample in the training set, traversing the layer correspondence between the student model and the teacher model based on the correspondence between batch samples and super-batches, calculating the loss function of the student model on each batch of samples based on the layer correspondence, optimizing the student model parameters based on the calculation result of the loss function, and testing the performance of the student model using the original validation set after optimization; Performing a second round of black-box knowledge distillation on the student model based on the soft-labeled training set, calculating the loss function of the student model on each batch of samples using the output result of the student model and the soft-labeled training set, optimizing the student model parameters based on the calculation result of the loss function, and testing the performance of the student model using the original validation set after optimization; Wherein, the training set and the verification set are both code text data.

2. A knowledge distillation method according to claim 1, characterized in that: In the process of obtaining the output of the teacher model on the training set samples, for the classification task, the output of the teacher model on the training set samples is the probability distribution of each category; for the generation task, the teacher model uses a decoding strategy to generate a result output, wherein the decoding strategy is any one of greedy decoding, beam search and sampling.

3. A knowledge distillation method according to claim 1, characterized in that: The layer correspondence between the student model and the teacher model is set to be a linear correspondence, that is, the first The layer corresponds to the first layer.

4. A knowledge distillation method according to claim 1, characterized in that: The student model The layer corresponds to the first layer, represents the number of layers of the teacher model, Indicates the number of layers of the student model.

5. A knowledge distillation method according to claim 1, characterized in that: The first round of white-box knowledge distillation is performed on the student model. The specific process includes the following steps: Assume that the super batch has Samples, The samples are , using the teacher model to generate a representation function ,in for The matrix of represents the length of the teacher model output sequence, Represents the intermediate result of the hidden layer of the teacher model, and the matrix Transform to Vector , then for super batches samples, for Matrix of right After singular value decomposition, if , U and V both represent orthogonal matrices, represents a singular value diagonal matrix, then The main component is , intercept Before Column transformation into a 3D tensor , represents the output sequence length of the student model, Represents the intermediate results of the hidden layer of the student model; Train the student model batch by batch. If the first Samples Corresponding to the first The hidden layer representation generated by the student model , then the loss function on this sample is , where MSE stands for mean square error, and the formula is , x and y are both R*C matrices, are the parameters of the student model.

6. A knowledge distillation method according to claim 1, characterized in that: For the classification task, the student model is subjected to a second round of black box knowledge distillation based on the soft labeled training set, specifically comprising the following steps: Assume that the first The samples are , using the student model to generate the probability distribution of each category is , then the sample The loss function is , where CE is the cross entropy, , represents the total number of categories, The teacher model pairs of samples in the soft-labeled training set The probability distribution of the prediction.

7. A knowledge distillation method according to claim 1, characterized in that: For the generation task, the student model is subjected to a second round of black box knowledge distillation based on the soft labeled training set, specifically comprising the following steps: Traversing samples Each prefix of , using the student model to predict the probability distribution of the next word in the vocabulary , then the loss function of the prefix is , CE is the cross entropy, where for A one-hot vector with 1 in the first position and 0 in other positions.

8. A distillation system, characterized in that: The system comprises: The soft-label training set acquisition module is used to obtain the output of the teacher model on the training set samples, replace the original labels on the training set samples, and obtain the soft-label training set; A student model construction module, used to construct the number of layers and architecture of the student model, and set the layer correspondence between the student model and the teacher model; A training set partitioning module, used for partitioning a training set or a soft-labeled training set into a number of super batches, and each super batch is further divided into a number of batches; A white-box knowledge distillation module is used to perform a first round of white-box knowledge distillation on the student model. During the first round of white-box knowledge distillation, principal component analysis is performed on the output results of each hidden layer of the teacher model for each super-batch sample in the training set. Based on the correspondence between batch samples and super-batches, the layer correspondence between the student model and the teacher model is traversed, and the loss function of the student model on each batch of samples is calculated based on the layer correspondence. The student model parameters are optimized based on the calculation results of the loss function, and the performance of the student model is tested using the original validation set after optimization; a black-box knowledge distillation module, configured to perform a second round of black-box knowledge distillation on the student model based on the soft-labeled training set, calculate the loss function of the student model on each batch of samples using the output result of the student model and the soft-labeled training set, optimize the student model parameters based on the calculation result of the loss function, and test the performance of the student model using the original validation set after optimization; Wherein, the training set and the verification set are both code text data.

9. A distillation system, characterized in that: It comprises one or more processing devices and one or more memories storing non-transient instructions. When the instructions are executed by the one or more processing devices, the one or more processing devices are configured to perform the steps of the knowledge distillation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the processor performs the steps of the knowledge distillation method as described in any one of claims 1 to 7.

Images