Variation knowledge distillation method and system based on comparative learning

Abstract

The invention relates to a variation knowledge distillation method and system based on comparative learning. The method comprises the following steps: constructing a teacher model and a student model; training the teacher model by using a comparative learning method, and calculating loss through a comparative learning loss function; knowledge of the teacher model is distilled into the student model through variational reasoning, including calculating a posterior distribution of the teacher model and a priori distribution of the student model, and minimizing the difference therebetween. According to the invention, the performance and generalization ability of the student model are improved through a method of comparative learning and variation knowledge distillation.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and machine learning, and in particular relates to a variational knowledge distillation method and system based on contrastive learning. Background Art

[0002] Knowledge distillation is a technique that transfers the knowledge of a large and complex teacher model to a smaller student model. This technique can significantly reduce the number of model parameters and computational complexity while maintaining model performance, and is therefore widely used in resource-constrained environments.

[0003] However, traditional knowledge distillation methods have some limitations. For example, traditional methods usually only focus on the soft labels output by the teacher model, while ignoring the feature representation inside the model. In addition, information loss during knowledge distillation will also affect the performance of the student model.

[0004] In recent years, contrastive learning has made significant progress as a self-supervised learning method. Contrastive learning learns effective representations by maximizing the similarity of samples of the same type and minimizing the similarity of samples of different types. In addition, variational inference methods perform well in processing complex probabilistic models, especially in capturing the uncertainty of data. Based on this, the present invention proposes a variational knowledge distillation method and system based on contrastive learning, which solves the above problems existing in the prior art. Summary of the invention

[0005] In order to solve the above-mentioned problems existing in the prior art, the present invention provides a variational knowledge distillation method and system based on contrastive learning. The present invention improves the performance and robustness of the student model by combining the advantages of contrastive learning and variational reasoning technology.

[0006] In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0007] A variational knowledge distillation method based on contrastive learning, the specific steps are as follows:

[0008] Step 1: Build teacher model and student model;

[0009] Step 2: Train the teacher model using contrastive learning to obtain high-quality representations and calculate the loss using the contrastive learning loss function.

[0010] Step 3: Distill the knowledge information of the teacher model into the student model through variational inference. This step preferably includes calculating the posterior distribution of the teacher model and the prior distribution of the student model, and minimizing the difference between the two by maximizing their mutual information.

[0011] Preferably, both the teacher model and the student model adopt a deep neural network structure, such as ResNet or Wide ResNet (WRN).

[0012] Preferably, in step 2, a contrastive learning method is used to train the teacher model, and a contrastive loss function is used to optimize the representation of the teacher model, including constructing positive and negative sample pairs so that the model can distinguish between similar and dissimilar samples, defining a contrastive loss function, and training the teacher model to generate high-quality representations by minimizing the contrastive loss function.

[0013] Further preferably, in step 2, a contrast loss function is used to optimize the representation of the teacher model so that it can maintain good discrimination in a high-dimensional space, specifically comprising the following steps:

[0014] (1) Construct a dataset containing positive sample pairs and negative sample pairs. Positive sample pairs are samples from the same category, and negative sample pairs are samples from different categories.

[0015] (2) Use the teacher model to extract features from positive and negative sample pairs to obtain corresponding high-dimensional representations. In this process, the teacher model can perform feature representations that effectively distinguish samples.

[0016] (3) Define a contrast loss function to measure the distance between positive sample pairs and the distance between negative sample pairs. The contrast loss function includes the following formula:

[0017]

[0018] Where N is the batch size, logD 1,i Represents the logarithm of the positive sample loss, logD 0,i Represents the logarithm of the negative sample loss, P pos,i is the similarity score of the positive sample, P neg,i is the similarity score of the negative sample, m is the number of negative samples, P n is the noise distribution, ε is a small constant to prevent division by zero, and its value is 1e-9;

[0019] (4) By minimizing the contrast loss function, the parameters of the teacher model are optimized so that the distance between positive sample pairs in the high-dimensional space is minimized and the distance between negative sample pairs is maximized, thereby forming a clear category boundary in the high-dimensional space;

[0020] (5) After back propagation and parameter update in step (4), check whether the model meets the convergence condition. In the present invention, the preferred convergence condition is whether the model reaches the expected level on the validation set. If it has not converged yet, return to step (2), continue to generate training sample pairs, recalculate the loss and update the parameters; if the model has converged, end the iteration process and output the final teacher model. Through the above multiple rounds of iterations, the representation of the teacher model is continuously optimized to ensure that it maintains good distinguishability in high-dimensional space. In this step, the convergence condition is: whether the model reaches the expected level on the validation set. The validation set is 10% of the data extracted from the original experimental data set for division, and it is guaranteed to be independent of the training set and the test set.

[0021] Preferably, in step 3, the variational reasoning includes the following steps:

[0022] (1) Obtain the posterior distribution p of the teacher model t,θ (z|x), where θ represents the parameters of the teacher model, z represents the latent variables, and x represents the input data;

[0023] (2) Obtain the prior distribution p of the student model s,φ (z), where φ represents the parameters of the student model;

[0024] (3) Calculate the posterior distribution p of the teacher model t,θ (z|x) and the student model prior distribution p s,φ The Kullback-Leibler (KL) divergence between (z) is given by:

[0025]

[0026] Among them, L KL represents the KL divergence loss, D KL represents KL divergence;

[0027] (4) Minimize the above KL divergence and optimize the parameters φ of the student model through back propagation so that the prior distribution p of the student model s,φ (z) is closer to the posterior distribution p of the teacher model t,θ (z|x);

[0028] (5) Repeat the above steps (1) to (4), that is, recalculate the posterior distribution p of the teacher model t,θ (z|x) and the prior distribution p of the student model s,φ(z) and the KL divergence between the teacher model and the student model, continue to minimize the divergence through back propagation, and further optimize the parameters φ of the student model until the KL divergence between the prior distribution of the student model and the posterior distribution of the teacher model reaches the predetermined convergence condition, that is: the change in the KL divergence value is less than a predetermined threshold, or reaches a minimum value within a given number of iterations.

[0029] Preferably, in the actual operation of step 3, it is relatively complicated to directly calculate the KL divergence when calculating high-dimensional data, which consumes memory and time. Therefore, the following processing method is adopted: by maximizing the mutual information between the teacher model and the student model through the characteristics of the variational distribution, the output of the student model is closer to the teacher model, where the mutual information between the teacher model and the student model is expressed as I(t i ;s i )=H(t i )-H(t i |s i ), H(t i ) is the entropy of the teacher model output, For the joint distribution p(t i ,s i ), the conditional entropy defined on the mutual information is maximized by:

[0030]

[0031] Among them, q(t i |s i ) is the joint distribution p(t i ,s i ), the goal is to calculate an approximate joint distribution p(t i ,s i ) is used to realize the calculation. Experiments show that directly using this variational distribution q(t i |s i ) to replace the joint distribution p(t i ,s i ) Performing step 3 can save a lot of computing memory and time.

[0032] Preferably, in the process of training and generating the teacher model in step 2 and distilling the teacher model knowledge into the student model in step 3, noise can be introduced into the input data and the hidden layer output. The main purpose is to simulate the uncertainty in the real environment, so as to maintain high performance and robustness when facing unknown data.

[0033] Preferably, the contrast loss L is calculated by step 2 and step 3. contrastive And the distillation loss L KL , and finally the final loss is Ltotal =L contrastive +αL KL , α is the weight parameter.

[0034] The present invention also discloses a variational knowledge distillation system based on contrastive learning, which is used to execute the above method and includes the following modules:

[0035] Model building module: building teacher model and student model;

[0036] Model training module: Use contrastive learning method to train the teacher model and calculate the loss through contrastive learning loss function;

[0037] Variational module: distills the knowledge information of the teacher model into the student model through variational reasoning.

[0038] The present invention combines the technical advantages of contrastive learning and variational reasoning, and adopts the idea of ​​maximizing the mutual information between the teacher model and the student model, so that the high-quality representation of the teacher model is effectively distilled into the student model, significantly improving the performance and robustness of the student model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the variational knowledge distillation method based on contrastive learning described in the present invention, the various parts involved in the whole process will be illustrated below.

[0040] Figure 1 This is a flow chart of a variational knowledge distillation method based on contrastive learning in a preferred embodiment of the present invention.

[0041] Figure 2 Flowchart for calculating KL divergence according to a preferred embodiment of the present invention.

[0042] Figure 3 Flowchart of comparative learning and training teacher model in a preferred embodiment of the present invention.

[0043] Figure 4 It is a schematic diagram of the feature space after training based on contrastive learning in a preferred embodiment of the present invention.

[0044] Figure 5 A block diagram of a variational knowledge distillation system based on contrastive learning according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] In this example, the CIFAR-10 dataset is used to demonstrate a variational knowledge distillation method based on contrastive learning.

[0047] The CIFAR-10 dataset contains 10 categories of images, 6,000 images in each category, and a total of 60,000 32×32 color images, which are divided into training set, validation set, and test set in a ratio of 8:1:1. This example selects ResNet32×4 as the teacher model and ResNet8×4 as the student model. Figure 1-4 As shown, the specific steps of this embodiment are as follows:

[0048] Step 1: Build the teacher model and the student model

[0049] 1.1 Teacher Model

[0050] Model structure: ResNet32×4

[0051] Input data: images from the CIFAR-10 dataset

[0052] Output: predicted probability distribution for each class of images

[0053] 1.2 Student Model

[0054] Model structure: ResNet8×4

[0055] Input data: images from the CIFAR-10 dataset

[0056] Output: predicted probability distribution for each class of images

[0057] In this step, noise can be introduced: noise can be introduced into the input data and hidden layer output to simulate the uncertainty in the real environment. Gaussian noise method can be used to enable the training model to maintain high performance and robustness when facing unknown data.

[0058] Step 2: Contrastive learning method to train the teacher model Figure 3 As shown; the specific steps are as follows:

[0059] 2.1. Building a Dataset

[0060] A dataset containing positive and negative pairs is constructed from the CIFAR-10 dataset. Positive pairs are pairs of images from the same category, and negative pairs are pairs of images from different categories.

[0061] 2.2 Feature extraction Figure 4 As shown;

[0062] The teacher model is used to extract features from positive and negative sample pairs respectively to obtain the corresponding high-dimensional knowledge representation.

[0063] 2.3. Define contrast loss function

[0064] The contrast loss function measures the distance between positive sample pairs and the distance between negative sample pairs. The loss function formula is as follows:

[0065]

[0066] Where N is the batch size, logD 1,i Represents the logarithm of the positive sample loss, logD 0,i Represents the logarithm of the negative sample loss, P pos,i is the similarity score of the positive sample, P neg,i is the similarity score of the negative sample, m is the number of negative samples, P n is the noise distribution, ε is a small constant to prevent division by zero, and its value is 1e-9;

[0067] 2.4. Optimize the parameters of the teacher model by minimizing the contrast loss function, so that the distance between positive sample pairs in high-dimensional space is minimized and the distance between negative sample pairs is maximized, thereby forming a clear category boundary in high-dimensional space;

[0068] 2.5. After back propagation and parameter update in step 2.4, check whether the model meets the convergence condition. In the present invention, the preferred convergence condition is whether the model reaches the expected level on the validation set. If it has not converged, return to step 2.2, continue to generate training sample pairs, recalculate the loss and update the parameters; if the model has converged, end the iterative process and output the final teacher model. Through multiple rounds of iterations, the representation of the teacher model is continuously optimized to ensure that it maintains good discrimination in high-dimensional space.

[0069] Step 3: Variational inference Figure 2 As shown; the specific steps are as follows:

[0070] 3.1. Obtaining the posterior distribution p of the teacher model t,θ (z|x)

[0071] Use the teacher model to forward propagate the input data x to obtain the posterior distribution p of the latent variable z t,θ (z|x).

[0072] 3.2. Obtaining the prior distribution p of the student model s,φ (z)

[0073] Define the prior distribution p of the latent variable z using the parameters φ of the student model s,φ (z).

[0074] 3.3. Calculate KL divergence

[0075] The teacher model posterior distribution p is calculated using the following formula t,θ (z|x) and the student model prior distribution p s,φThe KL divergence between (z):

[0076]

[0077] Since it is more complicated to calculate KL divergence for high-dimensional data, we use the method of maximizing the lower bound of mutual information to achieve the distillation effect of making the student model closer to the teacher model. See below for details.

[0078] 3.4. Minimize KL divergence

[0079] The parameters φ of the student model are optimized through back propagation, so that the prior distribution of the student model is closer to the posterior distribution of the teacher model. In the variational inference process, the output of the student model is closer to the teacher model by maximizing the mutual information between the teacher model and the student model. The mutual information between the teacher model and the student model is expressed as:

[0080] I(T;S)=H(T)-H(T|S)

[0081] Where I(T,S) is the mutual information between the teacher model output T and the student model output S, and H(T) is the entropy of the teacher model output. H(T|S) is the conditional entropy on the joint distribution p(T,S).

[0082] Maximize the lower bound of mutual information, as follows:

[0083] The lower bound of mutual information maximization is:

[0084] I(T;S)≥H(T)+E p(T,S) [logq(T|S)]

[0085] Among them, q(T|S) is the variational distribution of the joint distribution p(T,S), and its purpose is to achieve the calculation by calculating a variational distribution that approximates the joint distribution p(T,S).

[0086] 3.5. Repeat the above steps until the KL divergence reaches the predetermined convergence condition.

[0087] The contrast loss L calculated according to step 2 and step 3 contrastive And the distillation loss L KL , and finally the final loss is L total =L contrastive +αL KL , α is the weight parameter.

[0088] Implementation results:

[0089] Through the method of the above embodiment, experiments were conducted on the CIFAR-10 dataset, and the results showed that the variational knowledge distillation method based on contrastive learning of the present invention significantly improved the performance of the student model. The specific results are as follows:

[0090] Teacher model performance: 94.8% accuracy on the CIFAR-10 test set.

[0091] Student model performance:

[0092] Without knowledge distillation, the student model has an accuracy of 86.5%.

[0093] Through traditional knowledge distillation methods, the accuracy of the student model is improved to 89.2%.

[0094] By adopting the method of the present invention, the accuracy of the student model is further improved to 92.3%.

[0095] This example shows in detail the specific implementation of the variational knowledge distillation method based on contrastive learning. By combining the technical advantages of contrastive learning and variational reasoning, the idea of ​​maximizing the mutual information between the teacher network and the student network is adopted, so that the high-quality representation of the teacher model is effectively distilled into the student model, significantly improving the performance and robustness of the student model. The experimental results verify the effectiveness and superiority of the method of the present invention.

[0096] like Figure 5 As shown, this embodiment discloses a variational knowledge distillation system based on contrastive learning, which is used to execute the above method and includes the following modules:

[0097] Model building module: building teacher model and student model;

[0098] Model training module: Use contrastive learning method to train the teacher model and calculate the loss through contrastive learning loss function;

[0099] Variational module: distills the knowledge information of the teacher model into the student model through variational reasoning.

[0100] For other contents of this embodiment, please refer to the above method embodiment.

[0101] In summary, the present invention discloses a variational knowledge distillation method and system based on contrastive learning. The present invention first constructs a teacher model and a student model, and injects noise into the input data and the hidden layer to simulate the uncertainty in the real environment, so that the training model can maintain high performance and robustness when facing unknown data; the contrastive learning method is applied to train the teacher model to obtain high-quality knowledge representation; then the knowledge of the teacher model is distilled into the student model through variational reasoning, which includes calculating the posterior distribution of the teacher model and the prior distribution of the student model, and minimizing the distillation loss by maximizing the lower bound of the mutual information between the teacher and the student. The contrastive learning method includes optimizing the representation of the teacher model using a contrastive loss function so that it maintains good distinguishability in a high-dimensional space. The variational reasoning includes using the Kullback-Leibler (KL) divergence as a metric to measure the difference between the posterior distribution of the teacher model and the prior distribution of the student model. In addition, noise is introduced during the training process to enhance the robustness of the model. The present invention improves the performance and generalization ability of the student model through contrastive learning and variational knowledge distillation methods.

[0102] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A variational knowledge distillation method based on contrastive learning, characterized in that: The following steps are involved: Step 1: Build teacher model and student model; Step 2: Use contrastive learning to train the teacher model and calculate the loss using contrastive learning loss function; Step 3: Distill the knowledge information of the teacher model into the student model through variational reasoning.

2. A variational knowledge distillation method based on contrastive learning according to claim 1, characterized in that: In step 1, both the teacher model and the student model adopt a deep neural network structure.

3. The variational knowledge distillation method based on contrastive learning according to claim 1, characterized in that: In step 2, the contrastive learning method is used to train the teacher model, and the contrastive loss function is used to optimize the representation of the teacher model, including constructing positive and negative sample pairs so that the model can distinguish between similar and dissimilar samples, defining the contrastive loss function, and training the teacher model to generate high-quality representations by minimizing the contrastive loss function.

4. A variational knowledge distillation method based on contrastive learning according to claim 3, characterized in that: Step 2 specifically includes: Step 2.1: Construct a data set containing positive sample pairs and negative sample pairs. Positive sample pairs are samples from the same category, and negative sample pairs are samples from different categories. Step 2.2: Use the teacher model to extract features from positive sample pairs and negative sample pairs to obtain corresponding high-dimensional representations; Step 2.3, define the contrast loss function. The contrast loss function formula is as follows: Where N is the batch size, logD 1,i Represents the logarithm of the positive sample loss, logD 0,i Represents the logarithm of the negative sample loss, P pos，i is the similarity score of the positive sample, P neg，i is the similarity score of the negative sample, m is the number of negative samples, P n is the noise distribution, ε is a constant to prevent division by zero; Step 2.4, optimize the parameters of the teacher model by minimizing the contrast loss function, so that the distance between positive sample pairs in high-dimensional space is minimized and the distance between negative sample pairs is maximized; Step 2.5: Check whether the model meets the convergence conditions. If not, return to step 2.2; if it meets the convergence conditions, end the iteration process and output the final teacher model.

5. A variational knowledge distillation method based on contrastive learning according to claim 4, characterized in that: In step 2.5, the convergence condition is: whether the model reaches the expected level on the validation set.

6. A variational knowledge distillation method based on contrastive learning according to claim 4 or 5, characterized in that: In step 3, the variational reasoning includes the following steps: Step 3.

1. Obtain the posterior distribution p of the teacher model t,θ (z|x), where θ represents the parameters of the teacher model, z represents the latent variables, and x represents the input data; Step 3.2: Get the prior distribution p of the student model s,φ (z), where φ represents the parameters of the student model; Step 3.3: Calculate the posterior distribution p of the teacher model t,θ (z|x) and the prior distribution p of the student model s,φ The KL divergence between (z) is: Among them, L KL represents the KL divergence loss, D KL represents KL divergence; Step 3.4, minimize the KL divergence and optimize the parameters φ of the student model through back propagation; Step 3.5, repeat steps 3.1 to 3.4 until the KL divergence between the prior distribution of the student model and the posterior distribution of the teacher model reaches a predetermined convergence condition, where the convergence condition is: the change in the KL divergence value is less than a predetermined threshold, or reaches a minimum value within a given number of iterations.

7. The variational knowledge distillation method based on contrastive learning according to claim 6, characterized in that: In step 3.4, by maximizing the mutual information between the teacher model and the student model through the characteristics of the variational distribution, the output of the student model is closer to the teacher model, where the mutual information between the teacher model and the student model is expressed as: I(t i ;s i )=H(t i )-H(t i |s i ) Among them, H(t i ) is the entropy of the teacher model output, is the conditional entropy defined on the joint distribution p(ti,si), and the mutual information maximization lower bound is: Among them, q(t i |s i ) is the joint distribution p(t i ,s i )’s variational distribution.

8. A variational knowledge distillation method based on contrastive learning according to any one of claims 1 to 5, characterized in that: In the process of training and generating the teacher model in step 2 and distilling the teacher model knowledge into the student model in step 3, noise is introduced into the input data and hidden layer output.

9. The variational knowledge distillation method based on contrastive learning according to claim 6, characterized in that: Through the contrast loss L contrastive and distillation loss L KL , and the final loss is L total =L contrastive +αL KL , α is the weight parameter.

10. A variational knowledge distillation system based on contrastive learning, used to execute the method according to any one of claims 1 to 9, characterized in that: Includes the following modules: Model building module: building teacher model and student model; Model training module: Use contrastive learning method to train the teacher model and calculate the loss through contrastive learning loss function; Variational module: distills the knowledge information of the teacher model into the student model through variational reasoning.

Images