Model distillation method based on singular value decomposition and sinker macro distance
By combining sparse hybrid expert models and singular value decomposition with Sinker macro distance, this method solves the problem that student models cannot fully inherit the representational capabilities of teacher models in existing model distillation methods, and achieves efficient training of miniaturized student models and improved performance in downstream tasks.
Patent Information
- Application Number
- CN202511387446.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-26
- Publication Date
- 2026-02-03
AI Technical Summary
Existing model distillation methods fail to fully consider the structural characteristics of large language models, resulting in student models being unable to effectively inherit the powerful representational capabilities of teacher models, and also suffer from problems such as complex deployment, easy overfitting, and hardware incompatibility.
A sparse hybrid expert model is used as the teacher model. The expert-specific feedforward network weights are merged through singular value decomposition. The difference in output distribution between the teacher and student models is minimized by combining Sinker macro distance to perform knowledge transfer and refinement. Finally, fine-tuning is performed on downstream tasks.
This achievement enables the student model to approach or even reach the performance level of the teacher model while being miniaturized and computationally efficient, thereby improving the computational efficiency of the model and the performance of downstream tasks.
Smart Images
Figure CN121456853A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of machine learning and deep learning, and in particular to a model distillation method based on singular value decomposition and Sinkhorn distance. BACKGROUND
[0002] In the prior art, in order to reduce the model training cost and improve the training efficiency, the mixed expert model and the model distillation method have attracted widespread attention due to their conditional calculation advantages, but they have problems such as complex deployment, easy overfitting, hardware unfriendliness, and difficulty in integrating the knowledge of multiple experts. Especially in the field of large language models (LLM), open source model architectures such as Mistral and have emerged, which integrate mixed expert architectures and efficient training strategies. Mistral 7B and other models show that through optimization of model architecture and knowledge distillation, small models can also approach the performance of larger models; and the architecture realizes efficient expansion of LLM performance through multiple specialized experts. Therefore, the existing model distillation methods often do not fully consider the characteristics of the model structure in the weight replication and knowledge transfer process, resulting in the student model being unable to fully inherit the strong representation ability of the teacher model. Therefore, the existing model distillation methods often do not fully consider the characteristics of the model structure in the weight replication and knowledge transfer process, resulting in the student model being unable to fully inherit the strong representation ability of the teacher model.
[0003] Therefore, the present application provides a model distillation method based on singular value decomposition and Sinkhorn distance. SUMMARY
[0004] In order to make up for the deficiencies of the prior art and solve at least one technical problem raised in the background art.
[0005] The technical scheme adopted by the present application to solve its technical problems is: the model distillation method based on singular value decomposition and Sinkhorn distance according to the present application comprises the following steps: S1, selecting a sparse mixed expert model as a teacher model, the expert activation rate of which is not more than 10% of the input sample; S2, copying the weights of all layers shared by the teacher model and the student model from the teacher model to the student model, and initializing the error to be within 1% of the pre-trained teacher model; S3, merging the multiple expert-specific feedforward network weights in the layer into a single dense feedforward network in the student model; S4, refining the student model by minimizing the Sinkhorn distance between the teacher and student output distributions; S5, fine-tuning the refined dense student model on downstream tasks; The downstream tasks include image classification, dialogue system, code completion, and search question and answer.
[0006] A further improvement to the technical solution of this invention lies in the following: the specific process of selecting the hybrid expert model as the teacher model is as follows: Based on the application scenario of the model, select a sparse hybrid expert model. The model serves as a teacher model, which is a large-scale language model employing a hybrid expert architecture. The model includes the open-source Mistral model or... Model; Each expert has specialized functions, and its embedding layer, self-attention layer, and normalization layer are identical to those of the dense student model to be trained. Each layer is a feedforward network submodule, and because there are multiple layers, the specific weights for each expert are different.
[0007] A further improvement to the technical solution of this invention lies in the following: the specific process of copying the weights of all layers structurally shared by the teacher model and the student model from the teacher model to the student model is as follows: For the teacher model and the dense student model, which have structurally identical layers—namely, the embedding layer, the self-attention layer, and the normalization layer—the weights are directly copied from the teacher model to ensure that the student model can start training.
[0008] A further improvement to the technical solution of the present invention lies in: the aforementioned... The specific process of merging multiple expert-specific feedforward network weights in a layer into a single dense feedforward network in the student model is as follows: The matrix formed by the feedforward network weights for each expert Perform singular value decomposition, the decomposition formula is as follows: in, and It is a unitary matrix. It is a diagonal matrix. and Represent matrices respectively The number of rows and columns, i.e., the dimension of the weight matrix; Choose one for each expert element The specific method is to select the first few singular values such that: in, Represents the sum of singular values. It is an adaptive ratio used to control the amount of knowledge retained; An approximate representation of the weight matrix is obtained: Next, collect the breakdown components from all the experts: in, Indicates the number of experts. Representation matrix The former List, Will and Matrix stacking: The weight matrix of the dense student model is formed by the following formula: For the bias values of individual experts, since they contain limited information, we simply consider the bias vectors of all experts. Taking the average, we obtain the bias of the student model: in, It is the bias of the student model. It is the first Bias vector of an expert Thus completed Knowledge transfer from the teacher to the student model.
[0009] A further improvement to the technical solution of this invention lies in the following: the specific process of refining the student model by minimizing the Sinkerhomian distance between the teacher and student output distributions is as follows: To calculate the difference between the teacher and student model output distributions, for an input... Calculate the output results of the teacher model output layer. and student model Application with temperature parameters The normalized exponential function yields the probability distributions of both: and These are teacher and student models that apply temperature parameters. The probability distribution after normalization of the exponential function. express function, It is a temperature parameter used to adjust the smoothness of the probability distribution. Construct a cost matrix Each element measures the absolute difference between corresponding elements in the teacher and student distributions: in , For output Dimensions Measure the absolute difference between corresponding elements in the distribution of teachers and students; The kernel matrix is constructed using a Gaussian kernel with an entropy regularization weight : : The iteration is performed The two positive vectors are initialized for a given number of iterations and and The iteration is updated as follows: where and denote the positive vectors in the iteration process, denotes the number of iterations, and the division is all element-level division After the iteration, the approximate transportation plan is calculated: The distance is given by the inner product of and the cost matrix: The soft distillation loss based on the distance is defined as follows: The total training loss is a combination of the main task loss and the distillation loss based on : where is the total training loss, is the cross-entropy loss with real labels, is used to balance the contribution of the two parts, and the total training loss is minimized in this distillation stage.
[0010] The further improvement of the technical scheme of the present application is that the specific process of fine-tuning the refined dense student model on the downstream task is: The refined dense student model is fine-tuned on the downstream task using the collected knowledge and distilled knowledge to further improve its performance.
[0011] The further improvement of the technical scheme of the present application is that the temperature parameter is applied in the function to adjust the smoothness of the probability distribution, and the value of can control the degree of differentiation of the model for different outputs.
[0012] The further improvement of the technical scheme of the present application is that the adaptive proportion The determination can be dynamically adjusted according to the model training stage and specific tasks to optimize knowledge retention and model performance.
[0013] Further improvement of the technical scheme of the application is that the iteration number The selection can be adjusted according to the computing resources and model convergence to ensure the accuracy of distance calculation while ensuring the computing efficiency.
[0014] Further improvement of the technical scheme of the application is that the balance coefficient The balance coefficient is set according to the importance of the main task and the distillation task to reasonably allocate resources and optimize the overall performance of the model.
[0015] The beneficial effects of the application are as follows: The application processes the feedforward network weight matrix of each expert in the layer through singular value decomposition technology, selects Elements and controls the amount of knowledge retention, accurately captures and retains Key knowledge in the model, effectively migrates the knowledge of the teacher model to the student model, and secondly introduces the Sinkhorn distance to minimize the difference between the output distribution of the teacher and student models. Compared with the traditional KL divergence method, the difference between the two distributions can be more accurately measured and adjusted, so that the student model can better learn the output characteristics of the teacher model and improve the performance of the student model. At the same time, the student model after knowledge distillation and fine-tuning can achieve better performance in downstream tasks, close to or even reach the level of the teacher model, with smaller model size and higher computing efficiency, and thus improve the model performance. BRIEF DESCRIPTION OF DRAWINGS
[0016] The application will be further described below with reference to the accompanying drawings.
[0017] Figure 1 The application is a model distillation flowchart. DETAILED DESCRIPTION
[0018] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.
[0019] Please refer to Figure 1 , The present application provides: A model distillation method based on singular value decomposition and Sinkhorn distance, comprising the following steps: S1, selecting a sparse mixed expert model As a teacher model, the expert activation rate of which does not exceed 10% of the input sample; The specific process of selecting a mixed expert model as a teacher model is as follows: According to the application scenario of the model, a sparse mixed expert model is selected The model as a teacher model, the teacher model is a large language model adopting a mixed expert architecture, the model includes an open source Mistral model or Model; Each expert has a specialized function, and its embedding layer, self-attention layer and normalization layer are consistent with the dense student model to be trained, only Layer, i.e. feedforward network submodule, and because multiple are set, the expert specific weights are different, at the same time, the expert activation rate of the teacher model is controlled to not exceed 10% of the input sample, so as to ensure the balance between the calculation efficiency and the knowledge richness of the model.
[0020] It should be noted that selecting a suitable teacher model is the basis of knowledge distillation. The large language model adopting the mixed expert architecture has strong language understanding and generation ability, which can provide rich knowledge for the student model, and the open source model is easy to obtain and use. The Mistral and The models such as and perform well in natural language processing tasks, and selecting them as teacher models can ensure that the student model can learn high-quality knowledge.
[0021] S2, copy the weights of all layers shared by the teacher model and the student model in structure from the teacher model to the student model, and control the initialization error within 1% of the pre-trained teacher model; The specific process of copying the weights of all layers shared by the teacher model and the student model in structure from the teacher model to the student model is as follows: For the layers that are completely same in structure between the teacher model and the dense student model, i.e. embedding layer, self-attention layer and normalization layer, the weights are directly copied from the teacher, to ensure that the student model starts training.
[0022] Weight copying can let the student model start training from the powerful representation that the teacher model has learned, avoiding the long convergence process and uncertainty caused by starting training from random initialization, and controlling the initialization error can ensure that the student model has a good performance basis in the initial stage, improving the training efficiency and the performance of the final model.
[0023] S3, the Multiple expert specific feedforward network weights in the layer are merged into a single dense feedforward network in the student model; The specific process of merging the multiple expert-specific feedforward network weights in the layers into a single dense feedforward network in the student model is as follows: For each expert, the matrix of feedforward network weights singular value decomposition is performed, and the decomposition is as follows: wherein, and are unitary matrices, is a diagonal matrix, and respectively represent the number of rows and the number of columns of the matrix , i.e., the dimension of the weight matrix; for each expert, an element is selected, and the specific method is to select the first several singular values such that: wherein, represents the sum of singular values, is an adaptive ratio for controlling the amount of retained knowledge; through singular value decomposition and selection of the element, the key information in the feedforward network weight matrix of each expert can be extracted, while the amount of retained knowledge is controlled to avoid information overload, so that the student model can efficiently learn the core knowledge of the teacher model; an approximate representation of the weight matrix is obtained: then the decomposition components of all experts are collected: wherein, represents the number of experts, represents the first columns of the matrix , and the and matrices are stacked: the weight matrix of the dense student model is formed by: for the bias values of each expert, since they contain less information, only the bias vectors of all experts are simply averaged to obtain the bias of the student model: wherein, is the bias of the student model, is the first expert bias vector thereby accomplished layer knowledge transfer from teacher to student model.
[0024] The process of collecting decomposition components and forming weight matrix realizes the coincidence of multiple expert specific feedforward network weights into a single dense feedforward network, simplifies the structure of the student model, while retaining the key knowledge of each expert, improving the computational efficiency and knowledge integration ability of the student model.
[0025] S4, refining the student model by minimizing the Sinkhorn distance between the teacher and student output distributions; The specific process of refining the student model by minimizing the Sinkhorn distance between the teacher and student output distributions is as follows: In order to calculate the difference between the teacher and student model output distributions, for an input , the output result of the output layer of the teacher model is calculated and the output of the student model , the probability distribution of the two is obtained by applying the normalized exponential function with temperature parameter : and are the probability distributions of the teacher model and the student model after applying the normalized exponential function with temperature parameter , represents function, is the temperature parameter, used to adjust the smoothness of the probability distribution, which can control the discrimination of the model to different outputs, so that the model can better capture the subtle differences between the teacher model and the student model in the training process, thereby improving the learning effect of the student model.
[0026] The application of temperature parameter in function aims to adjust the smoothness of the probability distribution, and by adjusting the value of , the discrimination of the model to different outputs can be controlled.
[0027] A cost matrix is constructed, where each element measures the absolute difference of the corresponding elements in the teacher and student distributions: where , is the dimension of the output , measures the absolute difference of the corresponding elements in the teacher and student distributions; with entropy regularization weight Gaussian kernel construction kernel matrix : Execute iteration Normalization, for a given number of iterations Initialize two positive vectors and The iterative updates are as follows: in, and This represents a positive vector during the iteration process. This indicates the number of iterations; all divisions are element-wise divisions, and the given number of iterations has elapsed. After iteration, calculate an approximate transportation plan: Number of iterations The choice can be adjusted based on computing resources and model convergence, ensuring both computational efficiency and... The accuracy of distance calculations allows for a more precise measurement of the differences between the output distributions of teacher and student models, thereby improving the refinement of student models. Distance by The inner product with the cost matrix is given by: Therefore, the definition is based on Soft distillation loss over distance: The total training loss is the sum of the main task loss and the loss based on... Combination of distillation losses: in, For total training losses, The cross-entropy loss is the one with the real labels. To balance the contributions of the two parts, this distillation phase minimizes the total training loss and the balance coefficient. Configure the settings according to the importance of the primary and distillation tasks to allocate resources reasonably and optimize the overall performance of the model.
[0028] S5. Fine-tune the refined dense student model on downstream tasks; Downstream tasks include image classification, dialogue systems, code completion, and search question answering.
[0029] The specific process of fine-tuning the refined dense student model on downstream tasks is as follows: The refined dense student model is fine-tuned on downstream tasks by using the collected knowledge and distilled knowledge to further improve its performance. The fine-tuning process can better adapt the model to specific downstream tasks. By utilizing the knowledge collected during the distillation process and the distilled knowledge, the model can further optimize its performance on specific tasks, improving the model's practicality and generalization ability. In the specific implementation process, the temperature parameters , adaptive proportion , iteration number and balance coefficient and other parameters can be dynamically adjusted according to the model training stage, specific task, computing resources and model convergence, etc. to optimize the performance and training effect of the model.
[0030] The above front, back, left, right, up, down are based on the Figure 1 in the drawings of the specification, according to the standard of the human observation angle, the side of the device facing the observer is defined as front, the left side of the observer is defined as left, and so on.
[0031] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore it cannot be understood as a limitation on the scope of protection of the present application.
[0032] The basic principles, main features and advantages of the present application are shown and described above. Those skilled in the art should understand that the present application is not limited by the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.
Claims
1. A model distillation method based on singular value decomposition and Sinkhorn macro distance, characterized in that: The method comprises the following steps: S1, select a sparse mixture expert model As a teacher model, its expert activation rate is not more than 10% of the input sample; S2, copying the weights of all layers shared in structure between the teacher model and the student model from the teacher model to the student model, and initializing the error control within 1% of the pre-trained teacher model; S3, to merge the plurality of expert-specific feedforward network weights in the layer into a single dense feedforward network in the student model; S4, refining the student model by minimizing the Sinkhorn distance between the teacher and the student output distribution; S5, fine-tuning the refined dense student model on the downstream task; The downstream task includes image classification, dialogue system, code completion and search question and answer.
2. The model distillation method based on singular value decomposition and Sinkhorn macro distance according to claim 1, characterized in that: The specific process of selecting the mixed expert model as the teacher model is: selecting a sparse mixture-of-experts model according to an application scenario of the model the model as a teacher model, the teacher model being a large language model employing a mixture-of-experts architecture, the model comprising an open-source Mistral model or the model; Each expert has a specialized function, its embedding layer, self-attention layer, normalization layer are consistent with the dense student model to be trained only layer, i.e. a feedforward network submodule, and since multiple are provided, the experts have different specific weights.
3. The model distillation method based on singular value decomposition and Sinkhorn macro distance according to claim 1, characterized in that: The specific process of copying the weights of all layers shared in structure between the teacher model and the student model from the teacher model to the student model is: For the layers of the teacher model and the dense student model which are completely the same in structure, i.e. the embedding layer, the self-attention layer and the normalization layer, the weights are directly copied from the teacher, ensuring that the student model starts training.
4. The model distillation method based on singular value decomposition and Sinkhorn macro distance of claim 1, wherein: Said The specific process of merging the multiple expert-specific feedforward network weights in the layers into a single dense feedforward network in the student model is: a matrix of feedforward network weights for each expert singular value decomposition is performed, decomposed as follows: wherein, and are unitary matrices, is a diagonal matrix, and denote the number of rows and columns of the matrix i.e. the dimension of the weight matrix, respectively. Select one for each expert elements The specific method is to select the first few singular values such that: wherein, denotes the sum of singular values, is an adaptive scale that controls the amount of knowledge retained; An approximate representation of the weight matrix is obtained: Then, collect all the expert decomposition components: wherein represents the number of experts, represents the matrix of the first column of the matrix will be described below. and Matrix stacking: The weight matrix of the dense student model is formed by the following formula: The bias values for each expert, which contain less information, are simply averaged over all experts Taking the average gives the bias of the student model: wherein, is a bias of the student model, is a bias vector of the th expert Thereby accomplished Layered knowledge transfer from teacher to student model.
5. The model distillation method based on singular value decomposition and Sinkhorn macro distance according to claim 4, characterized in that: The specific process of refining the student model by minimizing the Sinkhorn distance between the teacher and the student output distribution is: To compute the difference between the teacher and student model output distributions, for an input , the output of the output layer of the teacher model and the output of the student model are computed, applying a normalizing exponential function with temperature parameter to obtain the probability distributions of both: and are teacher and student model applications with temperature parameter normalized exponential function, denotes function, is a temperature parameter to adjust the degree of smoothing of the probability distribution Construct a cost matrix where each element measures the absolute difference between the corresponding elements in the teacher and student distributions: wherein , is the dimension of the output , measures the absolute difference of the corresponding elements in the teacher and student distribution; A kernel matrix is constructed using a Gaussian kernel with an entropy regularization weight : Performing iterations Normalizing, for a given iteration number Initialize two positive vectors and Iterative updates are as follows: wherein, and denotes the forward vector in the iteration process, denotes the iteration number, the division is all element-level division, and the given iteration number After iteration, the approximate transportation plan is calculated: The distance is given by The inner product with the cost matrix gives: Thus defining a soft distillation loss based on distance The total training loss is a combination of the main task loss and the distillation loss based on the teacher model. wherein, is the total training loss, is the cross-entropy loss with the true labels, to balance the contributions of the two parts, the total training loss is minimized at this distillation stage.
6. The model distillation method based on singular value decomposition and Sinkhorn macro distance according to claim 5, characterized in that: The specific process of fine-tuning the refined dense student model on the downstream task is: Using the collected knowledge and distilled knowledge, the refined dense student model is fine-tuned on the downstream task to further improve its performance.
7. The model distillation method based on singular value decomposition and Sinkhorn macro distance according to claim 5, characterized in that: The temperature parameter In The application in the function aims to adjust the degree of smoothness of the probability distribution, controlling the discrimination of the model for different outputs by adjusting the value of .
8. The model distillation method based on singular value decomposition and Sinkhorn macro distance of claim 5, wherein: The adaptive proportion The determination is dynamically adjusted according to the stage of model training and specific tasks to optimize knowledge retention and model performance.
9. The model distillation method based on singular value decomposition and Sinkhorn macro distance of claim 5, wherein: The iteration number The selection of the iteration number is adjusted according to the computing resources and model convergence, ensuring The accuracy of distance calculation while ensuring the computing efficiency.
10. The model distillation method based on singular value decomposition and Sinkhorn macro distance of claim 5, wherein: The balance coefficient According to the importance of the main task and the distillation task, the resources are reasonably allocated, and the overall performance of the model is optimized.