Multi-teacher model knowledge distillation method integrating semantic weight soft labels

Abstract

The invention provides a multi-teacher model knowledge distillation method integrating semantic weight soft labels, which comprises the following specific steps: firstly, inputting a data set into each teacher model of a multi-teacher module, and outputting probability distribution of labels corresponding to input data by the teacher models, namely soft labels; secondly, labels output by the teacher model are converted into semantic vector representations through a pre-training language model, similarity values among the labels are calculated through the semantic vector representations of the labels, probability distribution of the soft labels is weighted through the semantic similarity values, and therefore semantic weight soft labels are obtained; finally, in the knowledge distillation process, the student model will distill knowledge from the teacher model through the two types of losses. The two types of loss comprise cross entropy loss and weighted relative entropy loss based on label semantic vector similarity. According to the method, the semantic information of the label can be fused into the probability distribution of the soft label of the multi-teacher model, and the knowledge distillation performance is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data mining and machine learning, and in particular relates to a multi-teacher model knowledge distillation method integrating semantic weight soft labels. Background Art

[0002] Knowledge distillation technology transfers knowledge from a large teacher model to a smaller and more efficient student model, allowing the smaller model to approach the performance of the large model on specific tasks, significantly reducing the operating cost and making it more suitable for deployment on edge devices.

[0003] In traditional knowledge distillation methods, for example, in the patent document entitled "Cross-scene classification method of hyperspectral images based on bidirectional attention mechanism" with publication number CN118552853A, it is disclosed that the teacher model guides the student model by generating probability distribution soft labels reflecting the prediction probability. The student model gradually learns by imitating these probability distributions, but this method often cannot effectively extract knowledge of different dimensions in the input data. When dealing with complex tasks, the understanding is relatively superficial, and it is difficult to accurately grasp the deep characteristics of the data.

[0004] In order to overcome these limitations, some studies have proposed the use of multi-teacher models to extract features from different dimensions, such as the patent literature with publication number CN116644798A, entitled "Knowledge distillation method, device, equipment and storage medium based on multiple teachers"; and the patent literature with publication number CN117609925A, entitled "Industrial anomaly detection method and system based on knowledge distillation of multiple teacher networks", which discloses the use of a fusion of multi-teacher models to extract features and improve the training efficiency of the student model by calculating the loss of label features. However, although the multi-teacher model enhances the multi-dimensional extraction capability of features, these methods still transfer knowledge based on probability distribution and fail to deeply explore and utilize the deep semantic information of the input data. Due to the lack of understanding of deep semantic information, the semantic associations of anomaly detection may not be fully expressed in complex scenarios. Therefore, there is a need to propose a knowledge distillation method that can integrate and fuse label semantic information. Summary of the invention

[0005] In view of the above problems existing in the prior art, the purpose of the present invention is to provide a multi-teacher model knowledge distillation method integrating semantic weight soft labels, which can integrate the semantic information of the labels into the probability distribution of the soft labels of the multi-teacher model and improve the knowledge distillation performance.

[0006] The present invention provides the following technical solution: a multi-teacher model knowledge distillation method integrating semantic weight soft labels, the multi-teacher model knowledge distillation method is implemented based on functional modules, the functional modules include a multi-teacher module, a student module and a knowledge distillation module; the student module distills knowledge from the multi-teacher module through a training process, the knowledge distillation module is used to coordinate the knowledge transfer between the multi-teacher module in the multi-teacher module and the student module in the student module, and is responsible for converting the output information of the teacher model into a form that can be understood and imitated by the student model during the training process, and optimizing the parameters of the student model by calculating the loss, thereby realizing efficient knowledge transfer; the specific steps of the method are as follows:

[0007] S1, input the data into the multi-teacher module, and the multi-teacher module outputs the soft label corresponding to the data;

[0008] S2, calculate the probability distribution weight of the soft label to obtain the semantic weight soft label;

[0009] S3. By adjusting the weights of the student module to align its output with the teacher module, the purpose of distilling knowledge from the teacher module is achieved.

[0010] Furthermore, the multi-teacher module includes multiple different teacher models, each teacher model uses a heterogeneous decomposition function to process input data; the decomposition functions include but are not limited to B-spline functions, Fourier functions and Gaussian radial basis functions, each function extracts corresponding features from the input data according to its mathematical properties, and the output data of the teacher model corresponds to the probability distribution of the label, that is, the soft label.

[0011] Furthermore, the specific steps of S2 are as follows:

[0012] 2.1) Convert the labels output by the teacher model into semantic vector representations of the student model in the student module through the pre-trained language model;

[0013] 2.2) Calculate the similarity value between the semantic vector representations of each label;

[0014] 2.3) The probability distribution of the soft label is weighted by the calculated similarity value to obtain the semantic weight soft label.

[0015] Furthermore, the steps of S3 are as follows:

[0016] 3.1) Construct the neural network model of the student model in the student module;

[0017] 3.2) Determine the student model output loss;

[0018] 3.3) Adjust the weights of the student model based on the loss via error back propagation;

[0019] 3.4) Evaluate the performance of aligning the student model with the teacher model.

[0020] Furthermore, in 3.2), the student model output loss includes cross entropy loss and weighted relative entropy loss based on the similarity of label semantic vectors; by minimizing the cross entropy loss, the output of the student model is optimized so that its probability distribution can match the output distribution of the teacher model; by adopting the weighted relative entropy loss based on the similarity of label semantic vectors, it can be ensured that the higher the similarity of the semantic labels output by the teacher model, the greater its influence on the student model.

[0021] By adopting the above technology, compared with the prior art, the beneficial effects of the present invention are as follows:

[0022] 1) The method of the present invention is based on the setting of multiple teacher modules, knowledge distillation modules and student modules. By optimizing the student model, it can obtain more extensive knowledge from the diversified teacher models and achieve performance that exceeds that of a single teacher model. It can also fuse the semantic information of labels into the probability distribution of soft labels of the multiple teacher models, thereby improving the knowledge distillation performance.

[0023] 2) In the method of the present invention, the multi-teacher module integrates multiple knowledge representations, reduces the deviation of the single teacher model, and improves the generalization ability and robustness of the model;

[0024] 3) In the method of the present invention, weighted soft labels are used to fuse label semantic relationships, which enhances the accuracy and stability of the student model at the classification boundary;

[0025] 4) In the method of the present invention, the guidance intensity of the teacher model is adjusted based on semantic similarity, which reduces the semantic deviation and makes the model decision more consistent with the semantic features;

[0026] 5) In the method of the present invention, the knowledge distillation module can refine the knowledge of multiple teachers, shorten the training time, reduce the computational complexity, and achieve efficient knowledge transfer. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the algorithm structure of the present invention.

[0028] Figure 2 It is the overall flow chart of multi-teacher knowledge distillation of the present invention.

[0029] Figure 3 It is a flow chart of the teacher model optimization process of the present invention.

[0030] Figure 4 It is a flow chart of the student model optimization process of the present invention.

[0031] Figure 5 It is a flow chart of the knowledge distillation process of the multi-teacher model of the present invention.

[0032] Figure 6 It is the knowledge distillation flow chart of the multi-teacher model of the present invention.

[0033] Figure 7 1 is a schematic diagram of a visualization example of the classification task results of multiple teacher models of the present invention; wherein (a) is the classification result of the teacher model with the B-spline function as the decomposition function, (b) is the classification result of the teacher model with the Fourier function as the decomposition function, and (c) is the classification result of the teacher model with the Gaussian radial basis function as the decomposition function;

[0034] Figure 8 It is a schematic diagram of a visualization example of the classification task results of the student model of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments of the specification. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0036] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the following detailed description of the present invention. Those skilled in the art can fully understand the present invention without the description of these details.

[0037] A knowledge distillation method for multi-teacher models integrating semantic weighted soft labels, such as Figure 1 As shown in Figure 1, it is implemented based on functional modules, which include multi-teacher module, knowledge distillation module and student module. The steps of the multi-teacher model knowledge distillation method are as follows:

[0038] First, the data set is input into multiple different teacher models in the multi-teacher module. These teacher models are composed of a neural network model and a variety of different decomposition functions. Each teacher model can extract different features from the data. Secondly, the probability distribution weight of the soft label is calculated to obtain a semantic weight soft label. This soft label will be used as a target in the distillation process to guide the student model to learn. Finally, the student module weight is adjusted to align its output with the teacher module, thereby achieving the purpose of distilling knowledge from the teacher module. The optimized student model can obtain more extensive knowledge from the diverse teacher models, so it can eventually achieve performance that exceeds that of a single teacher model. This embodiment is applied to the node classification task, and a graph neural network is selected to analyze the benchmark citations, but it is not limited to this task; in this embodiment, the input data set is graph data.

[0039] like Figure 2As shown, the present embodiment provides a multi-teacher model knowledge distillation method integrating semantic weight soft labels, and the specific process of the method is as follows:

[0040] Step 1: Pre-train multiple teacher models. Figure 3 As shown in the figure, first, initialize k different teacher models, inspired by the KAN (Kolmogorov-Arnold Network) network, where the teacher model is a combination of a neural network and different decomposition functions. Input data into different teacher models, each teacher model first predicts and then obtains the model's soft label, that is, the prediction result is represented by a probability distribution, and then the prediction loss of the k teacher models is calculated. This loss value can quantify the degree of difference between the teacher model and the data label in the prediction result. According to the loss feedback, adjust the parameters of the teacher model to improve the accuracy of the prediction.

[0041] Specifically, in the present embodiment, three teacher models are set, each teacher model combines a graph neural network and a decomposition function. The graph neural network here is GCN (graph convolutional network), and the three decomposition functions are: B-spline function, Fourier function and Gaussian radial basis function. They have the following characteristics: B-spline is based on piecewise polynomials, suitable for smooth and nonlinear data with high-order locality; Fourier decomposes the function into different sub-functions through trigonometric functions, which is suitable for data with periodic characteristics. Gaussian radial basis function is mainly used for difference and classification, and is suitable for projection into nonlinear high-dimensional space. In the present invention, taking the benchmark citation graph dataset Cora as an example, the input dimension is set to 1433 and the output dimension is set to 256 according to the dataset.

[0042] Specifically, the teacher model F t1 is a graph neural network based on B-spline KAN, θ t1 is the weight of the first teacher graph neural network layer. B-spline function can be used to enhance the nonlinear representation ability of node features. After the graph convolution layer, the node features are processed by a B-spline transformation layer. It transforms the node feature vector into a new feature space. This transformation is defined by a series of control points and basis functions, which enhances the adaptability and expressiveness of the model to local data structures. The specific form is expressed as:

[0043] F t1 =GNN B样条KAN (G,θ t1 )

[0044] For a given degree p and knot vector t, the i-th B-spline basis function B i,p (t) can be defined as follows:

[0045] When p = 0 (i.e. zero order):

[0046]

[0047] When p>0 (i.e. high order):

[0048]

[0049] In this example, the B-spline function order is set to 3. Finally, the teacher model F t1 The output features are used for label prediction, and the soft label of teacher model 1 is calculated as:

[0050] SoftLabels t1 =softmax(F t1 )

[0051] Teacher Model F t2 It is a graph neural network based on Gaussian radial basis KAN, which can be formalized as:

[0052] F t2 =GNN 高斯径向基KAN (G,θ t2 )

[0053] After the graph convolution layer, the Gaussian radial basis function maps the node features to a high-dimensional space in a nonlinear way. This mapping is based on the similarity of node features. The higher the similarity between nodes, the closer their outputs are in the high-dimensional space. The function is defined as follows:

[0054]

[0055] Among them, x represents the input feature, c in this case represents the center point of the grid, σ is generally the standard deviation, which is used to control the width of the function. In this case, it is the width of the grid, representing the width of the Gaussian function. Finally, the features output by teacher model 2 are used for label prediction, and the soft label of teacher model 2 is calculated as:

[0056] SoftLabels t2 =softmax(F t2 )

[0057] Teacher Model F t3 It is a graph neural network based on Fourier KAN, which can be formalized as:

[0058] F t3 =GNN 傅立叶KAN (G,θ t3 )

[0059] Fourier transform is a mathematical tool for analyzing various frequency components and is suitable for the analysis of periodic signals. It transforms the input data into features through a linear transformation and Fourier transform. Its core is to transform the input features using the sine and cosine basis functions of the Fourier series. The specific description is the following formula:

[0060]

[0061] Among them, x i represents the input feature vector, y j represents the output feature vector, j represents the index output feature dimension. In this example, j = 1, ..., 256, i is the index input feature dimension, i = 1, ..., 1433, and k represents the index grid size. and They represent the cosine and sine Fourier coefficients of the j-th output feature dimension at the k-th grid point for the i-th input feature dimension. bias j represents the bias term of the jth output dimension. Finally, the features output by teacher model 3 are used for label prediction, and the soft label of teacher model 3 is calculated as:

[0062] SoftLabels t3 =softmax(F t3 )

[0063] Step 2: Pre-train the student model. Figure 4 As shown, the student model is initialized first, and then data is input into the student model. In this embodiment, a multilayer perceptron network is used as the student model, which is expressed as:

[0064] F s =MLP(G,θ s )

[0065] The student model predicts the data and outputs the corresponding soft label, that is, the probability distribution is used to represent the predicted category probability. The soft label of the student model is expressed as:

[0066] SoftLabels s =softmax(F s )

[0067] Next, the prediction loss of the student model is calculated to measure the difference between the output of the student model and the real data label. Based on the calculated loss value, the back propagation algorithm is used to adjust the parameters of the student model so that the model is gradually optimized, and finally it can fit the real data more accurately and improve its prediction performance. Then the data is input into the student model for prediction. The difference between the output of the model and the real label is calculated by the loss function to obtain the soft label result of the student model, which further optimizes the learning process and improves the accuracy of the model output.

[0068] Step 3: Knowledge distillation of student models from multiple teacher models. Figure 5 and Figure 6 As shown in the figure, the specific process of knowledge distillation is as follows: First, input data to multiple teacher models and student models. The teacher model generates a probability distribution of categories for the input data, that is, the teacher model soft label:

[0069]

[0070] Where: z i Category C i The corresponding logit (the unnormalized output of the teacher model), T is the temperature coefficient, which is used to adjust the smoothness of the probability distribution.

[0071] Secondly, the teacher model outputs the category label C through the pre-trained language model i Converted to semantic vector V(C i ), and then use cosine similarity to measure the similarity between the semantic vectors of each label:

[0072]

[0073] Then, the probability distribution of soft labels is weighted by the semantic similarity value to obtain semantically weighted soft labels:

[0074]

[0075] Finally, normalization is performed:

[0076]

[0077] Similarly, the student model's prediction of the input data also generates a category probability distribution, that is, the student model soft label:

[0078]

[0079] Then, the semantic vector of the student model is generated through the pre-trained language model. During the training process, two loss functions are used to optimize the student model. First, the weighted relative entropy loss based on the similarity of the label semantic vector is used to ensure that the higher the similarity of the semantic labels output by the teacher model, the greater the impact on the student model, so that the student model can better capture the semantic knowledge of the teacher model. Then, the cross entropy loss is used to ensure that the output of the student model is not only as close as possible to the soft label of the teacher model, but also consistent with the real label, to ensure the actual task performance of the student model.

[0080] In this embodiment, the specific loss calculation part uses the KL divergence based on the weighted relative entropy loss of the label semantic vector similarity to measure the distance between the semantic weight soft label of the teacher model and the output of the student model:

[0081]

[0082] This loss function ensures that the higher the semantic label similarity output by the teacher model, the greater the influence on the student model.

[0083] Second, the cross entropy loss function is used to measure the difference between the probability distributions of the student and teacher models, L CE The specific manifestations are as follows:

[0084]

[0085] The final total loss function for knowledge distillation is as follows:

[0086] L total =λL CE +(1-λ)L KL

[0087] Here, λ is a hyperparameter used to balance the contribution of the two losses.

[0088] In a specific embodiment, the present invention uses the benchmark citation graph dataset Cora to predict the graph node classification task. In order to intuitively display the learning results, this embodiment uses the t-distributed stochastic neighbor embedding (t-SNE) technology to visualize the output of the hidden layer in two dimensions. In this way, the data in Z is mapped to a two-dimensional space for observation. Each dot in the figure represents a node in the graph, and the color of the dot represents the label of the node. The final classification result is visualized as follows: Figure 7 (a), (b), (c) and Figure 8 As shown, Figure 7 (a) in the figure is the teacher model with B-spline function as the decomposition function. Figure 7 (b) in the figure is the Fourier function as the teacher model, Figure 7 (c) in the figure is the teacher model with Gaussian radial basis function as the decomposition function. Figure 8 It is the prediction result of the student model after distillation. From the results, we can see that the segmentation effect of each teacher model is good to varying degrees, but there will be differences in a specific category, and some color categories will overlap slightly, indicating that the knowledge learned by each teacher under the same data set is different. In the end, the classification results learned by the student model are relatively uniform and the segmentation is obvious, which reflects the effectiveness of the method.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A knowledge distillation method for a multi-teacher model integrating semantic weight soft labels, characterized in that: The knowledge distillation method of the multi-teacher model is implemented based on functional modules, which include a multi-teacher module, a student module and a knowledge distillation module. The student module distills knowledge from the multi-teacher module through the training process, and the knowledge distillation module is used to coordinate the knowledge transfer between the multi-teacher module and the student module. The specific steps of the method are as follows: S1, input the data into the multi-teacher module, and the multi-teacher module outputs the soft label corresponding to the data; S2, calculate the probability distribution weight of the soft label to obtain the semantic weight soft label; S3. By adjusting the weights of the student module to align its output with the teacher module, the purpose of distilling knowledge from the teacher module is achieved.

2. According to claim 1, a multi-teacher model knowledge distillation method integrating semantic weight soft labels is characterized in that: The multi-teacher module includes multiple different teacher models, each teacher model uses a heterogeneous decomposition function to process input data; the decomposition function includes but is not limited to B-spline function, Fourier function and Gaussian radial basis function, each function extracts corresponding features from the input data according to its mathematical properties, and the output data of the teacher model corresponds to the probability distribution of the label, that is, the soft label.

3. The multi-teacher model knowledge distillation method integrating semantic weight soft labels according to claim 2 is characterized in that: The specific steps of S2 are as follows: 2.1) Convert the labels output by the teacher model into semantic vector representations of the student model in the student module through the pre-trained language model; 2.2) Calculate the similarity value between the semantic vector representations of each label; 2.3) The probability distribution of the soft label is weighted by the calculated similarity value to obtain the semantic weight soft label.

4. The multi-teacher model knowledge distillation method integrating semantic weight soft labels according to claim 3 is characterized in that: The steps of S3 are as follows: 3.1) Construct the neural network model of the student model in the student module; 3.2) Determine the student model output loss; 3.3) Adjust the weights of the student model based on the loss via error back propagation; 3.4) Evaluate the performance of aligning the student model with the teacher model.

5. The multi-teacher model knowledge distillation method integrating semantic weight soft labels according to claim 4 is characterized in that: In the above 3.2), the student model output loss includes cross entropy loss and weighted relative entropy loss based on the similarity of label semantic vectors; by minimizing the cross entropy loss, the output of the student model is optimized so that its probability distribution can match the output distribution of the teacher model; by adopting the weighted relative entropy loss based on the similarity of label semantic vectors, it can be ensured that the higher the similarity of the semantic labels output by the teacher model, the greater its influence on the student model.

Images