Over-parameterized knowledge distillation method, device, equipment and medium

Abstract

The invention relates to the technical field of knowledge distillation, and discloses an over-parameterization knowledge distillation method, device, equipment and medium, which can map a two-dimensional parameter matrix in an original student model from a two-dimensional tensor space to a high-dimensional tensor space to obtain a student model comprising a high-dimensional tensor, increase trainable parameters in the student model and improve the knowledge distillation efficiency. The knowledge capacity of the student model is improved, the student model comprising the high-dimensional tensor is trained, the high-dimensional tensor in the student model is updated until the training end condition is met, namely the training purpose is achieved, the high-order tensor in the student model after training is completed is reduced into a target two-dimensional parameter matrix, and the target two-dimensional parameter matrix is obtained; the shape of the target two-dimensional parameter matrix is the same as that of the two-dimensional parameter matrix in the original student model, and obtaining the student model with good knowledge distillation. According to the invention, the knowledge distillation effect can be enhanced under the condition of maintaining the parameter quantity during the deployment of the student model.

Description

Technical Field

[0001] The present invention relates to the field of knowledge distillation technology, and in particular to an over-parameterized knowledge distillation method, device, equipment and medium. Background Art

[0002] In the field of artificial intelligence model processing, it has become a standard paradigm to first pre-train on large-scale unlabeled corpora, and then fine-tune the pre-trained model according to downstream tasks to obtain a final model with strong performance. However, the number of parameters used in the pre-training process is large, the structure of the final model is complex, and the energy consumption, inference speed, storage requirements and edge device deployment constraints of the final model all affect the implementation of the final model, which in turn affects the execution of downstream tasks.

[0003] Related technologies can use knowledge distillation to use the large and complex final model as the teacher model, and train a smaller model with a simpler structure, namely the student model, based on the teacher model. It is expected that the knowledge and performance of the teacher model will be transferred to the student model, so that the student model has performance close to that of the teacher model. The trained student model can be deployed on the machine, effectively avoiding the impact of the above-mentioned limiting factors on the implementation of the model, and the trained student model can be used to perform downstream tasks.

[0004] However, the knowledge distillation effect of related technologies is average, and the performance of the trained student model has a significant performance difference with the teacher model. Summary of the invention

[0005] The present invention provides an over-parameterized knowledge distillation method, device, equipment and medium to solve the defects in the related technology that the knowledge distillation effect is general and the performance of the trained student model has a significant performance difference with the teacher model. The knowledge distillation effect can be enhanced while maintaining the original parameter amount of the student model, thereby improving the student model performance.

[0006] In a first aspect, the present invention provides an over-parameterized knowledge distillation method comprising: Obtain the corresponding original teacher model and original student model; Performing tensor decomposition on the two-dimensional parameter matrix in the original student model to obtain a corresponding first student high-order tensor and a first student model including the first student high-order tensor; and, performing tensor decomposition on the two-dimensional parameter matrix in the original teacher model to obtain a corresponding teacher high-order tensor and a target teacher model including the teacher high-order tensor; Based on the training data and the target teacher model, the first student model is trained and the first student high-order tensor in the first student model is updated to obtain a target student high-order tensor that meets the training end condition and a target student model including the target student high-order tensor; According to the shape information of the two-dimensional parameter matrix in the original student model, the target student high-order tensor in the target student model is contracted to obtain a corresponding target two-dimensional parameter matrix and a student model including the target two-dimensional parameter matrix; wherein the shape information of the target two-dimensional parameter matrix is ​​the same as the shape information of the two-dimensional parameter matrix in the original student model; The student model including the target two-dimensional parameter matrix is ​​determined as a student model with good knowledge distillation, so as to enhance the knowledge distillation effect while maintaining the original parameter amount of the student model.

[0007] Optionally, the high-order tensor includes auxiliary tensors and a central tensor; The step of training the first student model and updating the first student high-order tensor in the first student model based on the training data and the target teacher model to obtain a target student high-order tensor that meets a training end condition and a target student model including the target student high-order tensor comprises: Determining a first student auxiliary tensor and a first student central tensor in the first student high-order tensor; and determining a teacher auxiliary tensor in the teacher high-order tensor; Inputting first training data into the target teacher model and the first student model for processing to obtain first teacher output data of the target teacher model and first student output data of the first student model; Determining a first total loss function value based on a difference between the first student auxiliary tensor and the teacher auxiliary tensor, and based on a difference between the first student output data and the first teacher output data; Determine whether the first total loss function value converges, and obtain a first determination result; The target student high-order tensor and the target student model are obtained according to the first judgment result.

[0008] Optionally, obtaining the target student high-order tensor and the target student model according to the first judgment result includes: If the first judgment result is that the first total loss function value converges, directly determining that the first student high-order tensor and the first student model are the target student high-order tensor and the target student model respectively; If the first judgment result is that the first total loss function value has not converged, then updating the first student auxiliary tensor and the first student center tensor in the first student high-order tensor of the first student model to obtain the corresponding second student center tensor, second student auxiliary tensor, second student high-order tensor and second student model; wherein the second student model includes the second student high-order tensor, and the second student high-order tensor includes the second student auxiliary tensor and the second student center tensor; Inputting the second training data into the target teacher model and the second student model for processing to obtain second teacher output data of the target teacher model and second student output data of the second student model; Determining a second total loss function value based on a difference between the second student auxiliary tensor and the teacher auxiliary tensor, and based on a difference between the second student output data and the second teacher output data; Determine whether the second total loss function value converges until the determined total loss function value converges, and determine the latest student high-order tensor and student model as the target student high-order tensor and the target student model respectively.

[0009] Optionally, determining a total loss function value based on a difference between the student auxiliary tensor and the teacher auxiliary tensor, and based on a difference between the student output data and the teacher output data, comprises: Based on the difference between the student auxiliary tensor and the teacher auxiliary tensor, calculating the auxiliary tensor loss function value; and, based on the difference between the student output data and the teacher output data, calculating the output data loss function value; The sum of the auxiliary tensor loss function value and the output data loss function value is calculated and used as the total loss function value.

[0010] Optionally, the teacher output data includes teacher intermediate layer output data and teacher final output data, and the student output data includes student intermediate layer output data and student final output data; The calculating the output data loss function value based on the difference between the student output data and the teacher output data comprises: Calculating the intermediate layer loss function value based on the difference between the student intermediate layer output data and the teacher intermediate layer output data; and calculating the final output loss function value based on the difference between the student final output data and the teacher final output data; The sum of the intermediate layer loss function value and the final output loss function value is calculated and used as the output data loss function value.

[0011] Optionally, the original teacher model is a natural language processing model that has been pre-trained and fine-tuned and has a complex structure; the original student model is a natural language processing model that has a simple structure and corresponds to the original teacher model.

[0012] Optionally, the training data is a word vector or a sentence vector.

[0013] In a second aspect, the present invention provides an over-parameterized knowledge distillation device, comprising: An acquisition unit, used for acquiring the corresponding original teacher model and original student model; A decomposition unit, configured to perform tensor decomposition on the two-dimensional parameter matrix in the original student model to obtain a corresponding first student high-order tensor and a first student model including the first student high-order tensor; and perform tensor decomposition on the two-dimensional parameter matrix in the original teacher model to obtain a corresponding teacher high-order tensor and a target teacher model including the teacher high-order tensor; A training unit, configured to train the first student model and update the first student high-order tensor in the first student model based on training data and the target teacher model, so as to obtain a target student high-order tensor that meets a training end condition and a target student model including the target student high-order tensor; A contraction unit, used for contracting the target student high-order tensor in the target student model according to the shape information of the two-dimensional parameter matrix in the original student model, so as to obtain a corresponding target two-dimensional parameter matrix and a student model including the target two-dimensional parameter matrix; wherein the shape information of the target two-dimensional parameter matrix is ​​the same as the shape information of the two-dimensional parameter matrix in the original student model; A determination unit is used to determine the student model including the target two-dimensional parameter matrix as a student model with good knowledge distillation, so as to enhance the knowledge distillation effect while maintaining the original parameter amount of the student model.

[0014] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the over-parameterized knowledge distillation method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the over-parameterized knowledge distillation method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0016] The over-parameterized knowledge distillation method, device, equipment and medium provided by the present invention can map the two-dimensional parameter matrix in the original student model from the two-dimensional tensor space to the high-dimensional tensor space, obtain a student model including a high-dimensional tensor, increase the trainable parameters in the student model, improve the knowledge capacity of the student model, and train the student model including the high-dimensional tensor, update the high-dimensional tensor in the student model until the training end condition is met, and obtain a student model including the target student high-order tensor, and improve the performance and knowledge capacity of the student model. The present invention can also reduce the target student high-order tensor in the student model to a target two-dimensional parameter matrix for the student model of the target student high-order tensor, and the shape of the target two-dimensional parameter matrix is ​​the same as the shape of the two-dimensional parameter matrix in the original student model, so as to obtain a student model with good knowledge distillation. The present invention can enhance the knowledge distillation effect while fully maintaining the original parameter amount when the student model is deployed, and enhance the performance of the student model while retaining the advantages of the student model such as easy deployment, storage and fast reasoning speed, so that the performance of the student model is close to or exceeds the performance of the teacher model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of a process flow of an over-parameterized knowledge distillation method provided by an embodiment of the present invention; Figure 2 A schematic diagram of a process flow of another over-parameterized knowledge distillation method provided by an embodiment of the present invention; Figure 3 A flow chart of another over-parameterized knowledge distillation method provided in an embodiment of the present invention Figure 4 A schematic diagram of a tensor decomposition process provided by an embodiment of the present invention; Figure 5 A schematic diagram of the structure of an over-parameterized knowledge distillation device provided by an embodiment of the present invention; Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Combine the following Figure 1-Figure 4 The over-parameterized knowledge distillation method of the present invention is described.

[0021] like Figure 1 As shown, this embodiment proposes a first over-parameterized knowledge distillation method, which may include the following steps: S101. Obtain the corresponding original teacher model and original student model.

[0022] The original teacher model may be a large and complex model that has been pre-trained and fine-tuned according to downstream tasks. The original student model may be a student model corresponding to the original teacher model. This embodiment expects to transfer the knowledge and performance of the original teacher model to the original student model to obtain a student model whose knowledge and performance are close to the original teacher model.

[0023] Optionally, the original teacher model is a natural language processing model that has been pre-trained and fine-tuned and has a complex structure. The original student model is a natural language processing model that corresponds to the original teacher model and has a simple structure.

[0024] Specifically, the original teacher model and the original student model can be corresponding natural language processing models, which are used to process related characters in the field of natural language processing, such as artificial intelligence question answering and text recognition.

[0025] Specifically, the original teacher model can be a large model downloaded from a network channel or obtained through training. In this embodiment, a model with a small number of parameters can be built by itself as a student model.

[0026] S102. Perform tensor decomposition on the two-dimensional parameter matrix in the original student model to obtain the corresponding first student high-order tensor and the first student model including the first student high-order tensor; and perform tensor decomposition on the two-dimensional parameter matrix in the original teacher model to obtain the corresponding teacher high-order tensor and the target teacher model including the teacher high-order tensor.

[0027] Specifically, this embodiment can perform tensor decomposition on the two-dimensional parameter matrices in the original student model and the original teacher model, respectively, and map the two-dimensional parameter matrices in the original student model and the original teacher model from the two-dimensional tensor space to the high-dimensional tensor space.

[0028] The first student high-order tensor is a high-order tensor obtained by performing tensor decomposition on the two-dimensional parameter matrix in the original student model. The first student model is a student model including the first student high-order tensor.

[0029] The teacher high-order tensor is a high-order tensor obtained by performing tensor decomposition on the two-dimensional parameter matrix in the original teacher model, and the target teacher model is a teacher model including the high-order tensor.

[0030] Optionally, the tensor decomposition may be a high-order singular value decomposition or other types of decomposition.

[0031] S103. Based on the training data and the target teacher model, the first student model is trained and the first student high-order tensor in the first student model is updated to obtain a target student high-order tensor that meets the training end conditions and a target student model including the target student high-order tensor.

[0032] The training data is the data used to train the student model.

[0033] Optionally, when the teacher model and the student model are applicable to a natural language processing task in a natural language processing scenario, the training data is a word vector or a sentence vector.

[0034] Specifically, this embodiment can train the first student model and update the first student high-order tensor in the first student model based on the training data and the target teacher model, so as to transfer the knowledge and performance in the teacher model to the student model, improve the performance of the student model, and make the student model have performance close to that of the teacher model.

[0035] It should be noted that the training end condition can be a set condition for directly or indirectly evaluating whether the student model has a performance close to that of the teacher model. Specifically, in the process of training the student model, this embodiment can directly or indirectly evaluate whether the training purpose has been achieved, that is, whether the performance of the student model has approached or exceeded the performance of the teacher model, based on whether the training end condition is met. If it is determined that the training end condition is met, it can be determined that the training purpose has been achieved and the training of the student model can be terminated. If it is determined that the training end condition is not met, it can be determined that the training purpose has not been achieved and the student model needs to continue to be trained.

[0036] It should also be noted that this embodiment maps the two-dimensional parameter matrix in the original student model from the two-dimensional tensor space to the high-dimensional tensor space to obtain a student model including a high-dimensional tensor, which can effectively increase the trainable parameters in the student model and improve the knowledge capacity of the student model, thereby improving the learning ability of the student model. Thereafter, the student model including the high-dimensional tensor is trained, and the high-dimensional tensors in the student model are updated until the training end conditions are met, that is, the training purpose is achieved, so that the performance of the student model is close to or exceeds that of the teacher model, which can significantly improve the performance and knowledge capacity of the student model.

[0037] S104. According to the shape information of the two-dimensional parameter matrix in the original student model, the target student high-order tensor in the target student model is contracted to obtain the corresponding target two-dimensional parameter matrix and the student model including the target two-dimensional parameter matrix; wherein the shape information of the target two-dimensional parameter matrix is ​​the same as the shape information of the two-dimensional parameter matrix in the original student model.

[0038] The shape information of the two-dimensional parameter matrix includes the number of rows and columns of the two-dimensional parameter matrix.

[0039] It can be understood that the target two-dimensional parameter matrix and the two-dimensional parameter matrix in the original student model are matrices with the same shape information, that is, matrices with the same number of rows and columns.

[0040] Specifically, this embodiment can shrink the target student high-order tensor in the target student model into a corresponding two-dimensional parameter matrix, namely, the target two-dimensional parameter matrix, and can obtain a student model corresponding to the target student model, namely, a student model including the target two-dimensional parameter matrix.

[0041] S105. Determine the student model including the target two-dimensional parameter matrix as a student model for knowledge distillation, so as to enhance the knowledge distillation effect while maintaining the original parameter amount of the student model.

[0042] It should be noted that when the relevant technology uses knowledge distillation technology to train the student model, due to the small number of trainable parameters and limited knowledge capacity of the student model, there is still a significant performance difference between the trained student model and the teacher model. Among them, the knowledge distillation process in the relevant technology generally uses a student model with a small number of parameters to imitate the performance of the teacher model with a large number of parameters and good performance in training and data processing, so as to improve the performance of the small model and alleviate the limitations of the large model in the deployment and application stages. However, in this way, the student model's too few trainable parameters and knowledge capacity often limit the performance of the student model.

[0043] Related technologies generally directly increase the number of parameters of the student model to improve the knowledge capacity of the student model, so that the student model can better learn the performance of the classroom model during training, thereby enhancing the performance of the student model. However, increasing the number of parameters of the student model will lead to a more complicated structure of the student model, so that the deployment of the student model will still have the same problems as the deployment of the teacher model.

[0044] The over-parameterized knowledge distillation method proposed in this embodiment can map the two-dimensional parameter matrix in the original student model from the two-dimensional tensor space to the high-dimensional tensor space to obtain a student model including a high-dimensional tensor, thereby increasing the trainable parameters in the student model, improving the knowledge capacity of the student model, and training the student model including the high-dimensional tensor, updating the high-dimensional tensor in the student model until the training end condition is met, that is, the training purpose is achieved, so that the performance of the student model is close to or exceeds the teacher model, which is conducive to significantly improving the performance and knowledge capacity of the student model. At the same time, this embodiment can also losslessly shrink the high-order tensor obtained after matrix decomposition into a target two-dimensional parameter matrix in the deployment stage of the student model. The shape of the target two-dimensional parameter matrix is ​​the same as the shape of the two-dimensional parameter matrix in the original student model, and obtain a student model with good knowledge distillation. The present invention can enhance the knowledge distillation effect while fully maintaining the original parameter amount when the student model is deployed, and enhance the performance of the student model while retaining the advantages of the student model such as easy deployment, storage and fast reasoning speed, so that the performance of the student model is close to or exceeds the performance of the teacher model.

[0045] based on Figure 1 ,This embodiment proposes a second over-parameterized knowledge distillation method, in which the high-order tensor includes an auxiliary tensor and a central tensor.

[0046] The above-mentioned training of the first student model based on the training data and the target teacher model and updating of the first student high-order tensor in the first student model to obtain a target student high-order tensor that meets the training end condition and a target student model including the target student high-order tensor include: Determine a first student auxiliary tensor and a first student central tensor in a first student high-order tensor; and, determine a teacher auxiliary tensor in a teacher high-order tensor; Inputting the first training data into the target teacher model and the first student model for processing to obtain first teacher output data of the target teacher model and first student output data of the first student model; Determining a first total loss function value based on a difference between the first student auxiliary tensor and the teacher auxiliary tensor, and based on a difference between the first student output data and the first teacher output data; Determine whether the first total loss function value converges, and obtain a first determination result; The target student high-order tensor and the target student model are obtained according to the first judgment result.

[0047] Among them, the first student auxiliary tensor and the first student central tensor are the auxiliary tensor and the central tensor in the first student high-order tensor respectively.

[0048] Specifically, the first training data may be a training data. In this embodiment, the specific form of the training data may be determined according to the scenarios and task types to which the teacher model and the student model are applicable.

[0049] Specifically, in this embodiment, the first training data can be input into the target teacher model for processing, and the output data generated by the target teacher model processing the first training data can be obtained, namely, the first classroom output data. Also, in this embodiment, the first training data can be input into the first student model for processing, and the output data generated by the first student model processing the first training data can be obtained, namely, the first student output data.

[0050] Specifically, the convergence condition of the total loss function value may be that the first total loss function value is less than a preset threshold, or that the total loss function value is stable near the preset threshold.

[0051] It can be understood that when the first total loss function value does not meet the convergence condition, this embodiment can update the student auxiliary tensor and the student central tensor in the student high-order tensor of the student model based on the output data generated by processing the same training data based on the teacher model, and based on the teacher auxiliary tensor, and continue to train the updated student model until the training goal is achieved, and obtain a student model whose performance is close to or exceeds that of the teacher model.

[0052] Optionally, the step of obtaining the target student high-order tensor and the target student model according to the first judgment result includes: If the first judgment result is that the first total loss function value converges, then directly determining the first student high-order tensor and the first student model as the target student high-order tensor and the target student model respectively; If the first judgment result is that the first total loss function value has not converged, then the first student auxiliary tensor and the first student center tensor in the first student high-order tensor of the first student model are updated to obtain the corresponding second student center tensor, the second student auxiliary tensor, the second student high-order tensor and the second student model; wherein the second student model includes the second student high-order tensor, and the second student high-order tensor includes the second student auxiliary tensor and the second student center tensor; Inputting the second training data into the target teacher model and the second student model for processing to obtain second teacher output data of the target teacher model and second student output data of the second student model; Determining a second total loss function value based on a difference between the second student auxiliary tensor and the teacher auxiliary tensor, and based on a difference between the second student output data and the second teacher output data; Determine whether the second total loss function value converges until the determined total loss function value converges, and determine the latest student high-order tensor and student model as the target student high-order tensor and target student model respectively.

[0053] Specifically, in this embodiment, when it is determined that the first total loss function value has not converged, the relevant parameters in the first student model may be updated, and the updated student model may continue to be trained.

[0054] Among them, the second student center tensor is obtained by updating the first student center tensor, and the second student auxiliary tensor is obtained by updating the first student auxiliary tensor.

[0055] Among them, the second student high-order tensor is obtained by updating the first student central tensor and the first student auxiliary tensor in the first student high-order tensor, and the second student high-order tensor corresponds to the first student high-order tensor. The second student model is obtained by updating the relevant model parameters in the first student model, namely the relevant auxiliary tensor and the central tensor, and the second student model corresponds to the first student model.

[0056] Specifically, when the value of the first total loss function has not converged, this embodiment can update the first student auxiliary tensor and the first student central tensor in the first student high-order tensor of the first student model based on the gradient descent method to obtain the second student auxiliary tensor, the second student central tensor, the second student high-order tensor and the second student model.

[0057] Specifically, the second training data may be a certain training data different from the first training data. Similar to the process of processing the first training data, this embodiment may input the second training data into the target teacher model for processing, and obtain data generated by the target teacher model processing the second training data, i.e., second teacher output data, and input the second training data into the second student model for processing, and obtain data generated by the second student model processing the second training data, i.e., second student output data.

[0058] Afterwards, this embodiment can calculate the second total loss function value, and when the second total loss function value has not converged, continue to update the second student auxiliary tensor and the second student central tensor in the second student high-order tensor of the second student model based on the gradient descent method until the newly calculated total loss function value meets the convergence condition, and determine the latest student high-order tensor and student model as the target student high-order tensor and target student model, respectively.

[0059] Optionally, based on the difference between the student auxiliary tensor and the teacher auxiliary tensor, and based on the difference between the student output data and the teacher output data, a total loss function value is determined, including: Based on the difference between the student auxiliary tensor and the teacher auxiliary tensor, calculate the auxiliary tensor loss function value; and, based on the difference between the student output data and the teacher output data, calculate the output data loss function value; Calculate the sum of the auxiliary tensor loss function value and the output data loss function value, and use it as the total loss function value.

[0060] Specifically, when determining the total loss function value, this embodiment can first calculate the auxiliary tensor loss function value based on the difference between the student auxiliary tensor and the teacher auxiliary tensor, and then calculate the output data loss function value based on the difference between the student output data and the teacher output data. Finally, the sum of the auxiliary tensor loss function value and the output data loss function value is calculated and used as the total loss function value.

[0061] For example, when determining the first total loss function value, this embodiment can calculate the corresponding first auxiliary tensor loss function value based on the difference between the first student auxiliary tensor and the teacher auxiliary tensor, and then calculate the first output data loss function value based on the difference between the first student output data and the first teacher output data. Finally, the sum of the first auxiliary tensor loss function value and the first output data loss function value is calculated and used as the first total loss function value.

[0062] For another example, when determining the second total loss function value, this embodiment can calculate the corresponding second auxiliary tensor loss function value based on the difference between the second student auxiliary tensor and the teacher auxiliary tensor, and then calculate the second output data loss function value based on the difference between the second student output data and the second teacher output data. Finally, the sum of the second auxiliary tensor loss function value and the second output data loss function value is calculated and used as the second total loss function value.

[0063] Optionally, the teacher output data includes the teacher intermediate layer output data and the teacher final output data, and the student output data includes the student intermediate layer output data and the student final output data.

[0064] The above calculation of the output data loss function value based on the difference between the student output data and the teacher output data includes: Calculating the intermediate layer loss function value based on the difference between the student intermediate layer output data and the teacher intermediate layer output data; and calculating the final output loss function value based on the difference between the student final output data and the teacher final output data; Calculate the sum of the intermediate layer loss function value and the final output loss function value, and use it as the output data loss function value.

[0065] It can be understood that both the teacher model and the student model include intermediate layers and final layers. The teacher intermediate layer output data is the data output by the intermediate layer when the teacher model processes the training data, and the teacher final output data is the data output by the final layer when the teacher model processes the training data. For the introduction of the student intermediate layer output data and the student final output data, please refer to the teacher intermediate layer output data and the teacher final output data.

[0066] Specifically, when calculating the output data loss function value based on the difference between the student output data and the teacher output data, this embodiment can first calculate the intermediate layer loss function value based on the difference between the student intermediate layer output data and the teacher intermediate layer output data, and then calculate the final output loss function value based on the difference between the student's final output data and the teacher's final output data. Finally, the sum of the intermediate layer loss function value and the final output loss function value is calculated and used as the output data loss function value.

[0067] Among them, the first teacher output data may include the first teacher intermediate layer output data and the first teacher final output data, and the first student output data may include the first student intermediate layer output data and the first student final output data. In this embodiment, when calculating the first output data loss function value based on the difference between the first student output data and the first teacher output data, the first intermediate layer loss function value may be first calculated based on the difference between the first student intermediate layer output data and the first teacher intermediate layer output data, and then the first final output loss function value may be calculated based on the difference between the first student final output data and the first teacher final output data, and finally the sum of the first intermediate layer loss function value and the first final output loss function value is calculated and used as the first output data loss function value.

[0068] Specifically, the second teacher output data may include the second teacher intermediate layer output data and the second teacher final output data, and the second student output data may include the second student intermediate layer output data and the second student final output data. In this embodiment, when calculating the second output data loss function value based on the difference between the second student output data and the second teacher output data, the second intermediate layer loss function value may be first calculated based on the difference between the second student intermediate layer output data and the second teacher intermediate layer output data, and then the second final output loss function value may be calculated based on the difference between the second student final output data and the second teacher final output data, and finally the sum of the second intermediate layer loss function value and the second final output loss function value is calculated and used as the second output data loss function value.

[0069] It should be noted that, in the process of training the student model using training data, this embodiment calculates the auxiliary tensor loss function value, the intermediate layer loss function value and the final output loss function value, and takes the sum of the three as the total loss function value. The auxiliary tensors and the central tensor in the student model are updated according to the total loss function value until the training purpose is achieved. This can effectively align the auxiliary tensors in the student model and the teacher model, improve the efficiency of transferring task-related knowledge, enable the student model to learn the teacher model faster, and improve the efficiency of knowledge distillation. At the same time, since all tensors are not directly aligned, the central tensor can independently learn new knowledge from the loss of the output data of the student model and the output data of the teacher model, so that the student model has the potential to surpass the performance of the teacher model.

[0070] Among them, this embodiment calculates the auxiliary tensor loss function value and includes it in the total loss function value, and updates the model parameters of the student model based on the total loss function value. That is, the auxiliary tensor alignment strategy is used to force the alignment of the auxiliary tensors containing task-related knowledge in the student model and the teacher model, so that the student model learns the teacher model faster and improves the efficiency of knowledge distillation.

[0071] Among them, this embodiment calculates the intermediate layer loss function value and the final output loss function value, and includes them in the total loss function value, and updates the model parameters of the student model based on the total loss function value, so that in the knowledge distillation process, the central tensor of the student model can be independently trained according to the real label output by the teacher model, thereby giving the student model the possibility of surpassing the teacher model. The over-parameterized knowledge distillation method proposed in this embodiment can use the auxiliary tensor alignment strategy to force the alignment of the auxiliary tensors containing task-related knowledge in the student model and the teacher model, so that the student model can learn the teacher model faster and improve the efficiency of knowledge distillation. In the process of knowledge distillation, the central tensor of the student model can be independently trained according to the real label output by the teacher model, thereby giving the student model the possibility of surpassing the teacher model. like Figure 2 As shown, this embodiment proposes a third over-parameterized knowledge distillation method, which may include: Step 1: Model input.

[0072] Step 2: Get the student model and teacher model.

[0073] Specifically, this embodiment can obtain the corresponding original student model and original teacher model.

[0074] Step 3: Perform high-order singular value decomposition, i.e., tensor decomposition, on the two-dimensional parameter matrices in the original student model and the original teacher model.

[0075] Step 4: After tensor decomposition, the model structure represented by the tensor is obtained.

[0076] Step 5: Calculate the distillation loss function. Specifically, the distillation loss function may be an output data loss function value calculated based on the difference between the student output data and the teacher output data.

[0077] Step 6: Calculate the auxiliary tensor loss function. Specifically, the auxiliary tensor loss function may be an auxiliary tensor loss function value calculated based on the difference between the student auxiliary tensor and the teacher auxiliary tensor.

[0078] Step 7: Determine whether the training is finished. Specifically, this embodiment can calculate the sum of the distillation loss function and the auxiliary tensor loss function, that is, the total loss function value, and determine whether to end the training according to the total loss function value. If not, continue to train the student model until a student model containing the target student high-order tensor is obtained.

[0079] Step 8: Tensor merging: In this embodiment, the target student high-order tensor in the student model can be contracted and merged to obtain the corresponding target two-dimensional parameter matrix and the student model including the target two-dimensional parameter matrix.

[0080] It should be noted that both the student model and the teacher model may include multiple two-dimensional parameter matrices. When performing tensor decomposition on the two-dimensional parameter matrices in the student model and the teacher model, this embodiment can perform tensor decomposition on each two-dimensional parameter matrix in the student model and the teacher model to obtain a high-order tensor corresponding to each two-dimensional parameter matrix.

[0081] Specifically, when the student model and the teacher model include multiple two-dimensional parameter matrices, this embodiment can obtain multiple corresponding auxiliary tensors when performing tensor decomposition on the two-dimensional parameter matrices in the student model and the teacher model. At this time, when this embodiment trains the student model and calculates the auxiliary tensor loss function value, it can be calculated based on the following loss function: .

[0082] in, Auxiliary tensor loss function value, is the sequence number of the auxiliary tensor, is the total number of auxiliary tensors, . represents the mean square error function.

[0083] It is understandable that the student model including the target student high-order tensor may include multiple high-order tensors in the target student high-order tensor. At this time, when the target student high-order tensor is contracted and merged, the present embodiment may use the dot product method of the tensor to merge all the high-order tensors in the target student high-order tensor, so that the high-order tensor combination is merged into the corresponding two-dimensional parameter matrix, that is, the target two-dimensional parameter matrix.

[0084] Step 9: Obtain the model structure represented by the matrix. Specifically, the model structure represented by the matrix is ​​the student model including the target two-dimensional parameter matrix.

[0085] Step 10: Model reasoning.

[0086] Step 11: Model output.

[0087] like Figure 3 The knowledge distillation process shown in the figure, this embodiment can obtain the corresponding pre-trained teacher model according to the target task, and then fine-tune the pre-trained teacher model to obtain the above-mentioned original teacher model. Perform MPO decomposition, i.e., tensor decomposition, on the two-dimensional parameter matrix in the original teacher model to obtain the corresponding central tensor and auxiliary tensor. Perform tensor decomposition on the two-dimensional parameter matrix in the student model (i.e., the original student model) to obtain the over-parameterized student model, and then train the over-parameterized student model according to Update the student model. Figure 3 middle, and are the central tensors obtained by tensor decomposition of the two-dimensional parameter matrices in the student model and the teacher model, It is the auxiliary tensor obtained by tensor decomposition of the two-dimensional parameter matrix. is the auxiliary tensor loss function.

[0088] Among them, the pre-trained teacher model is a teacher model pre-trained on a large-scale data set, which can capture rich data features and fine-tune on multiple downstream tasks to achieve specific goals. It can generally be composed of a stacked Transformer structure, and the more parameters a model contains, the stronger the model's ability to store knowledge and information, but at the same time it will bring greater computing and storage costs. This makes how to deploy or reduce the cost of using large models on edge devices one of the key issues in the implementation of large models in related technologies. This embodiment uses a parameterized method during the training process, which effectively improves the distillation efficiency without increasing the storage space and reasoning time of the model, thereby effectively reducing the difficulty of deploying and applying pre-trained models.

[0089] like Figure 4 The tensor decomposition process shown in Figure 1 is as follows: The shape before decomposition is The parameter matrix of . , is the parameter matrix The eigenvector matrix obtained after the SVD decomposition is the high-order singular value decomposition. is the parameter matrix The eigenvalue matrix obtained after the SVD decomposition. is the normalized eigenvector matrix. is the center tensor, is an auxiliary tensor. This embodiment can be used for the two-dimensional parameter matrix Perform high-order singular value decomposition to obtain the corresponding auxiliary tensors and central tensors. In the decomposition process, high-order singular value decomposition is used to increase the trainable parameters of the model by performing multiple singular value decompositions on the target matrix. By mapping the parameter matrix with large gradient changes in the original matrix to a high-dimensional space for optimization, the optimization difficulty of the model is reduced, thereby improving the convergence speed and performance of the model.

[0090] Combination Figure 3 and Figure 4 As shown in FIG. 1 , this embodiment can perform high-order singular value decomposition on the parameter matrices of the student model and the teacher model to obtain a tensor representation of the parameters. In this way, on the one hand, the learning ability of the student model can be improved by over-parameterization during the training process, and on the other hand, the distillation efficiency can be improved by aligning the auxiliary tensors obtained by decomposition. The auxiliary tensor alignment process is shown in FIG. Figure 4 As shown in the figure, the tensor obtained after parameter matrix decomposition can be divided into a central tensor and an auxiliary tensor. During the training process, the auxiliary tensors can be aligned to make the student model learn the teacher model faster. At the same time, the auxiliary tensor alignment strategy only forces the alignment of the auxiliary tensors containing task-related knowledge in the student model and the teacher model. During the distillation process, the central tensor can be trained independently according to the true label, giving the student model the possibility of surpassing the teacher model.

[0091] This embodiment enhances the performance of the student model by increasing the capacity of the student model using over-parameterization technology during the training process. At the same time, an auxiliary tensor alignment training strategy is used during the training process to accelerate the distillation process, so that the student model has the potential to exceed the performance of the teacher model. Finally, when deploying the student model, the decomposed tensor representation can be merged into a matrix representation, which does not increase the deployment difficulty and reasoning time.

[0092] like Figure 5 As shown, this embodiment proposes an over-parameterized knowledge distillation device, which may include: An acquisition unit 501 is used to acquire the corresponding original teacher model and original student model; A decomposition unit 502 is used to perform tensor decomposition on the two-dimensional parameter matrix in the original student model to obtain a corresponding first student high-order tensor and a first student model including the first student high-order tensor; and to perform tensor decomposition on the two-dimensional parameter matrix in the original teacher model to obtain a corresponding teacher high-order tensor and a target teacher model including the teacher high-order tensor; A training unit 503 is used to train the first student model and update the first student high-order tensor in the first student model based on the training data and the target teacher model to obtain a target student high-order tensor that meets the training end condition and a target student model including the target student high-order tensor; A contraction unit 504 is used to contract the target student high-order tensor in the target student model according to the shape information of the two-dimensional parameter matrix in the original student model to obtain the corresponding target two-dimensional parameter matrix and the student model including the target two-dimensional parameter matrix; wherein the shape information of the target two-dimensional parameter matrix is ​​the same as the shape information of the two-dimensional parameter matrix in the original student model; The determination unit 505 is used to determine the student model including the target two-dimensional parameter matrix as a student model for knowledge distillation, so as to enhance the knowledge distillation effect while maintaining the original parameter amount of the student model.

[0093] It should be noted that the processing of the acquisition unit 501, the decomposition unit 502, the training unit 503, the contraction unit 504 and the determination unit 505 and the beneficial effects thereof can be respectively referred to in Figure 1 Steps S101 to S105 in the above are not described in detail.

[0094] Optionally, the high-order tensor includes auxiliary tensors and a central tensor; The training unit 503 is further used for: Determine a first student auxiliary tensor and a first student central tensor in a first student high-order tensor; and, determine a teacher auxiliary tensor in a teacher high-order tensor; Inputting the first training data into the target teacher model and the first student model for processing to obtain first teacher output data of the target teacher model and first student output data of the first student model; Determining a first total loss function value based on a difference between the first student auxiliary tensor and the teacher auxiliary tensor, and based on a difference between the first student output data and the first teacher output data; Determine whether the first total loss function value converges, and obtain a first determination result; The target student high-order tensor and the target student model are obtained according to the first judgment result.

[0095] Optionally, the training unit 503 is further configured to: If the first judgment result is that the first total loss function value converges, then directly determining the first student high-order tensor and the first student model as the target student high-order tensor and the target student model respectively; If the first judgment result is that the first total loss function value has not converged, then the first student auxiliary tensor and the first student center tensor in the first student high-order tensor of the first student model are updated to obtain the corresponding second student center tensor, the second student auxiliary tensor, the second student high-order tensor and the second student model; wherein the second student model includes the second student high-order tensor, and the second student high-order tensor includes the second student auxiliary tensor and the second student center tensor; Inputting the second training data into the target teacher model and the second student model for processing to obtain second teacher output data of the target teacher model and second student output data of the second student model; Determining a second total loss function value based on a difference between the second student auxiliary tensor and the teacher auxiliary tensor, and based on a difference between the second student output data and the second teacher output data; Determine whether the second total loss function value converges until the determined total loss function value converges, and determine the latest student high-order tensor and student model as the target student high-order tensor and target student model respectively.

[0096] Optionally, the training unit 503 is further configured to: Based on the difference between the student auxiliary tensor and the teacher auxiliary tensor, calculate the auxiliary tensor loss function value; and, based on the difference between the student output data and the teacher output data, calculate the output data loss function value; Calculate the sum of the auxiliary tensor loss function value and the output data loss function value, and use it as the total loss function value.

[0097] Optionally, the teacher output data includes the teacher intermediate layer output data and the teacher final output data, and the student output data includes the student intermediate layer output data and the student final output data; The training unit 503 is further used for: Calculating the intermediate layer loss function value based on the difference between the student intermediate layer output data and the teacher intermediate layer output data; and calculating the final output loss function value based on the difference between the student final output data and the teacher final output data; Calculate the sum of the intermediate layer loss function value and the final output loss function value, and use it as the output data loss function value.

[0098] Optionally, the original teacher model is a natural language processing model that has been pre-trained and fine-tuned and has a complex structure; the original student model is a natural language processing model that has a simple structure and corresponds to the original teacher model.

[0099] Optionally, the training data is word vectors or sentence vectors.

[0100] The over-parameterized knowledge distillation device proposed in this embodiment can map the two-dimensional parameter matrix in the original student model from the two-dimensional tensor space to the high-dimensional tensor space to obtain a student model including a high-dimensional tensor, thereby increasing the trainable parameters in the student model, improving the knowledge capacity of the student model, and training the student model including the high-dimensional tensor, updating the high-dimensional tensor in the student model until the training end condition is met, that is, the training purpose is achieved, so that the performance of the student model is close to or exceeds the teacher model, which is conducive to significantly improving the performance and knowledge capacity of the student model. At the same time, this embodiment can also losslessly shrink the high-order tensor obtained after matrix decomposition into a target two-dimensional parameter matrix in the deployment stage of the student model. The shape of the target two-dimensional parameter matrix is ​​the same as the shape of the two-dimensional parameter matrix in the original student model, and obtain a student model with good knowledge distillation. The present invention can enhance the knowledge distillation effect while fully maintaining the original parameter amount when the student model is deployed, and enhance the performance of the student model while retaining the advantages of the student model such as easy deployment, storage and fast reasoning speed, so that the performance of the student model is close to or exceeds the performance of the teacher model.

[0101] The over-parameterized knowledge distillation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0102] The embodiment of the present invention also provides a computer device having the above Figure 5 Over-parameterized knowledge distillation apparatus shown.

[0103] See also Figure 6 , a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0104] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0105] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0106] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function. The data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0107] The memory 20 may include a volatile memory, such as a random access memory. The memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive. The memory 20 may also include a combination of the above-mentioned types of memory.

[0108] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0109] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An over-parameterized knowledge distillation method, characterized in that: include: Obtain the corresponding original teacher model and original student model; Performing tensor decomposition on the two-dimensional parameter matrix in the original student model to obtain a corresponding first student high-order tensor and a first student model including the first student high-order tensor; and, performing tensor decomposition on the two-dimensional parameter matrix in the original teacher model to obtain a corresponding teacher high-order tensor and a target teacher model including the teacher high-order tensor; Based on the training data and the target teacher model, the first student model is trained and the first student high-order tensor in the first student model is updated to obtain a target student high-order tensor that meets the training end condition and a target student model including the target student high-order tensor; According to the shape information of the two-dimensional parameter matrix in the original student model, the target student high-order tensor in the target student model is contracted to obtain a corresponding target two-dimensional parameter matrix and a student model including the target two-dimensional parameter matrix; wherein the shape information of the target two-dimensional parameter matrix is ​​the same as the shape information of the two-dimensional parameter matrix in the original student model; The student model including the target two-dimensional parameter matrix is ​​determined as a student model with good knowledge distillation, so as to enhance the knowledge distillation effect while maintaining the original parameter amount of the student model.

2. The method according to claim 1, characterized in that: High-order tensors include auxiliary tensors and central tensors; The step of training the first student model and updating the first student high-order tensor in the first student model based on the training data and the target teacher model to obtain a target student high-order tensor that meets a training end condition and a target student model including the target student high-order tensor comprises: Determining a first student auxiliary tensor and a first student central tensor in the first student high-order tensor; and determining a teacher auxiliary tensor in the teacher high-order tensor; Inputting first training data into the target teacher model and the first student model for processing to obtain first teacher output data of the target teacher model and first student output data of the first student model; Determining a first total loss function value based on a difference between the first student auxiliary tensor and the teacher auxiliary tensor, and based on a difference between the first student output data and the first teacher output data; Determine whether the first total loss function value converges, and obtain a first determination result; The target student high-order tensor and the target student model are obtained according to the first judgment result.

3. The method according to claim 2, characterized in that The step of obtaining the target student high-order tensor and the target student model according to the first judgment result includes: If the first judgment result is that the first total loss function value converges, directly determining that the first student high-order tensor and the first student model are the target student high-order tensor and the target student model respectively; If the first judgment result is that the first total loss function value has not converged, then updating the first student auxiliary tensor and the first student center tensor in the first student high-order tensor of the first student model to obtain the corresponding second student center tensor, second student auxiliary tensor, second student high-order tensor and second student model; wherein the second student model includes the second student high-order tensor, and the second student high-order tensor includes the second student auxiliary tensor and the second student center tensor; Inputting the second training data into the target teacher model and the second student model for processing to obtain second teacher output data of the target teacher model and second student output data of the second student model; Determining a second total loss function value based on a difference between the second student auxiliary tensor and the teacher auxiliary tensor, and based on a difference between the second student output data and the second teacher output data; Determine whether the second total loss function value converges until the determined total loss function value converges, and determine the latest student high-order tensor and student model as the target student high-order tensor and the target student model respectively.

4. The method according to any one of claims 1 to 3, characterized in that Based on the difference between the student auxiliary tensor and the teacher auxiliary tensor, and based on the difference between the student output data and the teacher output data, determining a total loss function value, including: Based on the difference between the student auxiliary tensor and the teacher auxiliary tensor, calculating the auxiliary tensor loss function value; and, based on the difference between the student output data and the teacher output data, calculating the output data loss function value; The sum of the auxiliary tensor loss function value and the output data loss function value is calculated and used as the total loss function value.

5. The method according to claim 4, characterized in that The teacher output data includes the teacher intermediate layer output data and the teacher final output data, and the student output data includes the student intermediate layer output data and the student final output data; The calculating the output data loss function value based on the difference between the student output data and the teacher output data comprises: Calculating the intermediate layer loss function value based on the difference between the student intermediate layer output data and the teacher intermediate layer output data; and calculating the final output loss function value based on the difference between the student final output data and the teacher final output data; The sum of the intermediate layer loss function value and the final output loss function value is calculated and used as the output data loss function value.

6. The method according to claim 1, characterized in that The original teacher model is a natural language processing model with a complex structure that has been pre-trained and fine-tuned; the original student model is a natural language processing model with a simple structure that corresponds to the original teacher model.

7. The method according to claim 6, characterized in that The training data is a word vector or a sentence vector.

8. A parameterized knowledge distillation device, characterized in that: include: An acquisition unit, used for acquiring the corresponding original teacher model and original student model; A decomposition unit, configured to perform tensor decomposition on the two-dimensional parameter matrix in the original student model to obtain a corresponding first student high-order tensor and a first student model including the first student high-order tensor; and perform tensor decomposition on the two-dimensional parameter matrix in the original teacher model to obtain a corresponding teacher high-order tensor and a target teacher model including the teacher high-order tensor; A training unit, configured to train the first student model and update the first student high-order tensor in the first student model based on training data and the target teacher model, so as to obtain a target student high-order tensor that meets a training end condition and a target student model including the target student high-order tensor; A contraction unit, used for contracting the target student high-order tensor in the target student model according to the shape information of the two-dimensional parameter matrix in the original student model, so as to obtain a corresponding target two-dimensional parameter matrix and a student model including the target two-dimensional parameter matrix; wherein the shape information of the target two-dimensional parameter matrix is ​​the same as the shape information of the two-dimensional parameter matrix in the original student model; A determination unit is used to determine the student model including the target two-dimensional parameter matrix as a student model with good knowledge distillation, so as to enhance the knowledge distillation effect while maintaining the original parameter amount of the student model.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the over-parameterized knowledge distillation method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the over-parameterized knowledge distillation method described in any one of claims 1 to 7.

Images