Vector matching knowledge distillation method and system based on adaptive temperature

Abstract

The invention discloses a vector matching knowledge distillation method and system based on adaptive temperature, and the method comprises the following steps: S1, inputting an image into a given teacher model and a given student model, carrying out the convolution processing of the image through the neural networks in the teacher model and the student model, and carrying out the processing through a full-link layer, a logistic regression logit output by the model is obtained, probability distribution characteristics of logits of the student model and logits of the teacher model are extracted through a softmax function, a self-adaptive temperature mechanism is introduced after the probability distribution characteristics are extracted, and temperature distillation is carried out; s2, vector features of probability distribution of the teacher model and the student model are extracted, and cosine distance loss, cross entropy loss and KL divergence loss are introduced to calculate total loss. The probability distribution characteristics of the student model and the teacher model are fully considered, the vector characteristics of the student model and the teacher model are extracted, a knowledge distillation characteristic system is expanded, the problem of non-uniform categories caused by knowledge distillation global pooling is solved, and the problem of distillation error increase caused by excessive mismatching of single data is greatly reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification model optimization, and in particular relates to a temperature-adaptive vector matching knowledge distillation method and system. Background Art

[0002] With the continuous development of modern image recognition and classification technology, the past few decades have witnessed the prosperity of deep learning in computer vision tasks. General models are trained on many data sets, and accuracy is usually the most important criterion for verifying the quality of the model. As the accuracy of the model continues to improve, the memory it occupies is also gradually increasing. In order to meet this challenge, knowledge distillation is introduced to reduce the capacity of the model.

[0003] However, current image classification algorithms and their applications face two major problems: First, labeling large datasets requires a lot of manpower and material resources; and since there are very few companies willing to make their datasets public, most image classification models lack datasets and cannot be effectively verified. Second, as model performance continues to improve, the model becomes more complex and cannot be deployed on some devices with lower hardware configurations, which cannot meet the needs of actual applications.

[0004] In view of the above situation, how to use vector matching methods to optimize the model to reduce model memory loss and improve model accuracy has become a technical problem that needs to be solved in this field. Based on this, the present invention proposes a temperature-adaptive vector matching knowledge distillation method and system, which can significantly improve the performance of small-volume models and greatly reduce the memory usage of the model. Summary of the invention

[0005] In view of the above-mentioned status of the prior art, the present invention proposes a vector matching knowledge distillation method and system based on temperature adaptation.

[0006] The present invention adopts the following technical scheme:

[0007] A vector matching knowledge distillation method based on adaptive temperature includes the following steps:

[0008] S1: Input the image into the given teacher model and student model. The neural networks in the teacher model and the student model perform several convolution operations on the image respectively, and then process it through the full connection layer to obtain the logistic regression logit output by the model respectively. The logit of the student model and the teacher model respectively extracts the probability distribution features through the softmax function. After extracting the probability distribution features, the adaptive temperature mechanism is introduced to perform temperature distillation;

[0009] S2: Extract the vector features of the probability distribution of the teacher model and the student model and introduce cosine distance loss, cross entropy loss and KL divergence loss to calculate the total loss.

[0010] As a preferred method, in step S1, the probability distribution P(Z n k ) and P(V n k ) uses the periodic variation characteristics of the cosine time table to periodically control the distillation temperature at different distillation stages, so that the student model can fit the probability distribution characteristics of the teacher model at diverse temperatures, so that the student model can approach the teacher model in terms of the smoothness of the probability distribution. The formula is expressed as:

[0011]

[0012] Where m represents the training round, T m represents the distillation temperature, Zn represents the logit output of the student model, Vn represents the logit output of the teacher model, k represents the label, and C represents the number of classification types.

[0013] As a preferred embodiment, the adaptive temperature mechanism introduces a cosine time table into the temperature regulation mechanism, and the temperature change trend is expressed as:

[0014]

[0015] Where m represents the training round, T max and T min represent the maximum distillation temperature and the minimum distillation temperature respectively, and α represents the weight coefficient.

[0016] Preferably, vector features are introduced into the probability distributions of the teacher model and the student model so that the teacher model and the student model are aligned at the vector level.

[0017] As a preferred method, vector feature matching knowledge distillation includes introducing vector features into the teacher model and the student model, using cosine similarity and cosine distance algorithms when introducing vector features, and using cosine distance as a loss function in the output layer to narrow the feature gap between the teacher model and the student model:

[0018]

[0019] Among them, cos represents cosine similarity, L cos represents the cosine distance loss.

[0020] As a preference, KL divergence introduces relative entropy into the distillation process of the student model and the teacher model as a likelihood ratio maximization process between the student model and the teacher model. The formula can be expressed as:

[0021]

[0022] Among them, C represents the total number of samples included, L KL represents the KL divergence loss, V n k represents the normalized output features of the teacher.

[0023] As a preferred method, cross entropy introduces uncertainty into the training process of the student model to narrow the gap between the student model and the training target. The formula can be expressed as:

[0024]

[0025] Among them, L ce represents the cross entropy loss, C represents the total number of samples, and Z n k represents the normalized output features of the student, gt represents the true label, n represents the nth sample, and k represents the label.

[0026] As a preferred method, the distillation strategy of the distillation model is replanned by combining vector matching loss, KL divergence and cross entropy loss using the loss function. The formula can be expressed as:

[0027] L sum =βL KL +γL ce +λL cos

[0028] Among them, β, γ and λ represent the corresponding weight coefficients. Different weights reflect the importance of each distillation loss in knowledge distillation. The main idea of ​​this strategy is to make the output performance of the teacher model and the student model as close as possible, so that the student model can imitate the various output characteristics of the teacher model.

[0029] The present invention also discloses a vector matching knowledge distillation system based on adaptive temperature, which includes the following modules based on the above method:

[0030] Adaptive temperature distillation module: The image is input into the given teacher model and student model. The neural networks in the teacher model and the student model perform several convolution processes on the image respectively, and then process it through the full connection layer to obtain the logistic regression logit output by the model respectively. The logit of the student model and the teacher model respectively extracts the probability distribution features through the softmax function. After extracting the probability distribution features, the adaptive temperature mechanism is introduced to perform temperature distillation;

[0031] Total loss calculation module: extracts the vector features of the probability distribution of the teacher model and the student model and introduces cosine distance loss, cross entropy loss and KL divergence loss to calculate the total loss.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] (1) The present invention provides a temperature-adaptive vector matching knowledge distillation method and system, which fully considers the probability distribution characteristics of the student model and the teacher model under the original knowledge distillation system, extracts the vector features of the two, expands the feature system of knowledge distillation, solves the problem of inconsistent categories caused by global pooling of knowledge distillation, and greatly reduces the problem of increased distillation error caused by excessive mismatch of single data.

[0034] (2) The present invention provides a temperature-adaptive vector matching knowledge distillation method and system, which introduces a temperature-adaptive mechanism to enable the extraction of multi-temperature knowledge during the distillation process, thereby solving the defect that the student model only receives single-temperature knowledge during the training process and greatly improving the distillation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is a flow chart of a vector matching knowledge distillation method based on adaptive temperature according to a preferred embodiment of the present invention.

[0037] Figure 2 It is a temperature cycle variation diagram of a preferred embodiment of the present invention.

[0038] Figure 3 It is a vector feature matching diagram of the teacher model and the student model in the preferred embodiment of the present invention.

[0039] Figure 4 It is a KL divergence loss output graph of a preferred embodiment of the present invention.

[0040] Figure 5 It is a cross entropy loss output diagram of a preferred embodiment of the present invention.

[0041] Figure 6 It is a block diagram of a vector matching knowledge distillation system based on adaptive temperature according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0043] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0044] The preferred embodiment of the present invention discloses a vector matching knowledge distillation method based on adaptive temperature, the flow chart of which is as follows: Figure 1 As shown, the following steps are included:

[0045] Step 1: Input the image into the given teacher model and student model. The neural network in the model performs several convolution operations on the image, and then processes it through the full-link layer to obtain the logistic regression logit output by the model. The logit of the student and teacher models extracts the probability distribution features through the softmax function. After extracting the probability distribution features, the adaptive temperature mechanism is introduced to perform temperature distillation.

[0046] Step 2: Extract the vector features of the probability distribution of the teacher model and the student model and introduce cosine distance loss, cross entropy loss and KL divergence loss to calculate the total loss.

[0047] This embodiment uses a temperature-adaptive vector matching knowledge distillation method to achieve performance enhancement of the image classification model. The student model fits the different smoothness features of the teacher model through adaptive temperature and enhances the knowledge distillation performance through vector features.

[0048] Specifically, given a teacher model and a student model, the same batch of data sets are trained at the same time, and the logit outputs of the two models after being processed by the full-link layer are obtained. The transmission data set contains a total of N samples. x n ∈R H×W and gt = g n ∈[1,k] are the image and label of the nth sample respectively. Given an input {x n ,y n}, the teacher and the student predict the logit vector V n and Z n ∈R 1×k That is Z n =f S (xn ) and V n =f T (x n ).

[0049] For the logit vectors of the given teacher model and student model, the two are processed by the softmax function. The probability distribution of the two can be expressed as:

[0050]

[0051] Among them, Zn represents the logit output of the student model, Vn represents the logit output of the teacher model, k represents the label, and C represents the number of classification types.

[0052] Add the distillation temperature to the probability distribution formula. It can be expressed as:

[0053]

[0054] Where m represents the training round, T m Indicates the distillation temperature.

[0055] The logit outputs of students and teachers are standardized to accelerate the convergence of the algorithm. The logit outputs of the student model and the teacher model are uniformly represented as X. The output standardization formula can be expressed as:

[0056]

[0057] Among them, μ represents the mean of the logit output, σ represents the standard deviation of the logit output, and X norm Represents the standardized logit output.

[0058] After obtaining the probability distribution after logit normalization, the distillation temperature is replaced. The distillation temperature is set to an adaptive periodic change adjustment using a cosine schedule so that the student model fits the teacher model of each smoothness level, such as Figure 2 As shown, the formula for distillation temperature can be expressed as:

[0059]

[0060] Among them, T max and T min Represent the highest and lowest distillation temperatures, α represents the weight coefficient, and m represents the number of training rounds. In order to adjust the distillation temperature change speed to meet the requirements of distillation, the weight coefficient α is added. In the actual training process, the distillation temperature change speed is adjusted according to the distillation effect. The larger α is, the slower the temperature change speed is, and the smaller α is, the faster the temperature change speed is.

[0061] The obtained normalized and T m The probability distribution of the replaced teacher model and student model P(V n k ) and P(Z n k ) to extract vector features and calculate the cosine distance, so that the probability distribution of the teacher model and the student model are aligned at the vector level, and the cosine similarity between the teacher model and the student model is calculated, such as Figure 3 As shown, the formula can be expressed as:

[0062]

[0063] Among them, cos represents cosine similarity, and its value range is [-1,1]. When cos takes 1, it means that the student model and the teacher model vector are consistent. When cos takes -1, it means that the student model and the teacher model vector are opposite. Substituting cosine similarity into cosine distance, we get cosine distance loss, and its formula is expressed as:

[0064]

[0065] Among them, cos represents cosine similarity, L cos represents the cosine distance loss.

[0066] According to the concept of relative entropy between probability distributions, relative entropy is introduced into the distillation process of the student model and the teacher model, and KL divergence is used as the likelihood ratio maximization process between the student model and the teacher model, such as Figure 4 As shown, the formula can be expressed as:

[0067]

[0068] Among them, L KL represents the KL divergence loss.

[0069] Decouple its KL divergence and extract information entropy and cross entropy. The derivation process example can be expressed as:

[0070]

[0071] The first half of the equation is the information entropy H(p(x)) of P, and the second half is the cross entropy. According to the cross entropy derivation formula, cross entropy introduces uncertainty into the training process of the student model to narrow the gap between the student model and the training target, such as Figure 5 As shown, the formula can be expressed as:

[0072]

[0073] Among them, L ce represents the cross entropy loss, and gt represents the true label.

[0074] According to the above steps, the total loss function of this method can be combined, including cross entropy loss, KL divergence loss, and vector matching loss. The loss is minimized during training and distillation. The formula can be expressed as:

[0075] L sum =βL KL +γL ce +λL cos

[0076] Among them, β, γ and λ represent the corresponding weight coefficients, and different weights reflect the importance of each distillation loss in knowledge distillation. The main idea of ​​this strategy is to make the output performance of the teacher model and the student model as close as possible, so that the student model can imitate the various output characteristics of the teacher model.

[0077] like Figure 6 As shown, this embodiment discloses a vector matching knowledge distillation system based on adaptive temperature, based on the above method embodiment, including the following modules:

[0078] Adaptive temperature distillation module: The image is input into the given teacher model and student model. The neural networks in the teacher model and the student model perform several convolution processes on the image respectively, and then process it through the full connection layer to obtain the logistic regression logit output by the model respectively. The logit of the student model and the teacher model respectively extracts the probability distribution features through the softmax function. After extracting the probability distribution features, the adaptive temperature mechanism is introduced to perform temperature distillation;

[0079] Total loss calculation module: extracts the vector features of the probability distribution of the teacher model and the student model and introduces cosine distance loss, cross entropy loss and KL divergence loss to calculate the total loss.

[0080] For other contents of this embodiment, please refer to the above method embodiment.

[0081] In summary, the present invention discloses a temperature-adaptive vector matching knowledge distillation method and system, a temperature-adaptive knowledge distillation algorithm is used to realize the smoothness fitting of the teacher model and the student model, the temperature is set for periodic changes through the cosine timetable theory, the logit outputs of the student and teacher models are standardized by the normalization processing algorithm to ensure that the standardized student and teacher outputs can extract standardized vector features, the vector features of the student model output are approximated to the vector features of the teacher model by the vector matching knowledge distillation algorithm, the classification loss and the distillation loss are calculated respectively by using the cross entropy and the KL divergence, and the distillation performance is improved by adjusting the loss weight by weighing the loss ratio.

[0082] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0083] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A vector matching knowledge distillation method based on adaptive temperature, characterized in that: The following steps are involved: S1: Input the image into the given teacher model and student model. The neural networks in the teacher model and the student model perform several convolution operations on the image respectively, and then process it through the full connection layer to obtain the logistic regression logit output by the model respectively. The logit of the student model and the teacher model respectively extracts the probability distribution features through the softmax function. After extracting the probability distribution features, the adaptive temperature mechanism is introduced to perform temperature distillation; S2: Extract the vector features of the probability distribution of the teacher model and the student model and introduce cosine distance loss, cross entropy loss and KL divergence loss to calculate the total loss.

2. The method for vector matching knowledge distillation based on adaptive temperature according to claim 1, characterized in that: In step S1, the probability distribution P(Z n k ) and P(V n k ) uses the periodic variation characteristics of the cosine time table to periodically control the distillation temperature at different distillation stages, so that the student model can fit the probability distribution characteristics of the teacher model at diverse temperatures, so that the student model is close to the teacher model in terms of the smoothness of the probability distribution. The formula is expressed as: Where m represents the training round, T m represents the distillation temperature, Z n k V represents the logit output of the student model for the nth sample under the k label n k Represents the logit output of the teacher model for the nth sample under the k label, k represents the label, and C represents the total number of samples included.

3. The method for vector matching knowledge distillation based on adaptive temperature according to claim 2, characterized in that: In step S1, the adaptive temperature mechanism introduces the cosine time table into the temperature regulation mechanism, and the temperature change trend is expressed as: Where m represents the training round, T max and T min represent the maximum distillation temperature and the minimum distillation temperature respectively, and α represents the weight coefficient.

4. The method for vector matching knowledge distillation based on adaptive temperature according to claim 3, characterized in that: In step S2, vector features are introduced into the probability distribution of the teacher model and the student model so that the teacher model and the student model are aligned at the vector level.

5. The method for vector matching knowledge distillation based on adaptive temperature according to claim 4, characterized in that: When introducing vector features, cosine similarity and cosine distance algorithms are used. In the output layer, cosine distance is used as the loss function to narrow the feature gap between the teacher model and the student model: Among them, cos represents cosine similarity, Lco s represents the cosine distance loss.

6. The method for vector matching knowledge distillation based on adaptive temperature according to claim 5, characterized in that: Cross entropy introduces uncertainty into the training process of the student model to narrow the gap between the student model and the training target. The formula is: Among them, Lc e represents the cross entropy loss, C represents the total number of samples, Zn k represents the normalized output features of the student, gt represents the true label, n represents the nth sample, and k represents the label.

7. The method for vector matching knowledge distillation based on adaptive temperature according to claim 6, characterized in that: The KL divergence introduces relative entropy into the distillation process of the student model and the teacher model as a likelihood ratio maximization process between the student model and the teacher model. The formula is expressed as: Among them, L KL represents the KL divergence loss, Vn k represents the normalized output features of the teacher.

8. The method for vector matching knowledge distillation based on adaptive temperature according to claim 7, characterized in that: Combining vector matching loss, cross entropy and KL divergence, the loss function is used to re-plan the distillation strategy. The formula is expressed as: Lsum=βLKL+yLce+Lcos Among them, β, γ and λ represent the corresponding weight coefficients.

9. A vector matching knowledge distillation system based on adaptive temperature, based on the method according to any one of claims 1 to 8, characterized in that: Includes the following modules: Adaptive temperature distillation module: The image is input into the given teacher model and student model. The neural networks in the teacher model and the student model perform several convolution processes on the image respectively, and then process it through the full connection layer to obtain the logistic regression logit output by the model respectively. The logit of the student model and the teacher model respectively extracts the probability distribution features through the softmax function. After extracting the probability distribution features, the adaptive temperature mechanism is introduced to perform temperature distillation; Total loss calculation module: extracts the vector features of the probability distribution of the teacher model and the student model and introduces cosine distance loss, cross entropy loss and KL divergence loss to calculate the total loss.

Images