Method and device for diffusion and self-distillation of student model guided by teacher model

Abstract

The invention provides a teacher model-guided student model diffusion self-distillation method, which comprises the following steps of: loading a pre-trained teacher model and a pre-trained student model, and starting to train the student model; extracting teacher features and original student features from the training data; guiding the diffusion model to carry out denoising sampling on the original student features through the teacher model, and generating corresponding denoised student features; performing self-distillation training based on the de-noised student features and the original student features; calculating loss of self-distillation training of the student model, and updating parameters of the student model; and circularly executing the steps until the training of the student model is finished. The invention further provides a student model diffusion self-distillation device guided by the teacher model, a storage medium and electronic equipment. Therefore, the student model can be better trained, so that the performance of the student model is improved.

Description

Technical Field

[0001] This invention relates to the field of model knowledge distillation technology, and in particular to a teacher-model-guided student model diffusion self-distillation method, apparatus, storage medium, and electronic device. Background Technology

[0002] In recent years, with the continuous development of deep learning, the number of model parameters has been increasing, leading to a continuous rise in the cost of model training and deployment. This makes direct application difficult in scenarios with limited computing power and storage, such as mobile devices and edge devices. Therefore, compressing and accelerating models while ensuring performance as much as possible has become one of the important research directions in industry and academia.

[0003] Knowledge distillation is a classic method of model compression. Its core idea is to use a more powerful teacher model to supervise the student model during training, thereby improving the student model's performance compared to independent training. Knowledge distillation methods can be divided into output distillation, feature distillation, and relation distillation. The main idea of ​​output distillation is to allow the student model to mimic the final output of the teacher model. This is typically achieved by softening the output with a temperature coefficient and aligning it with losses such as KL divergence, thus learning the teacher's soft labels. The main idea of ​​feature distillation is to align the feature representations of the teacher and student in an intermediate layer, such as feature maps, channel responses, and attention. The teacher's representational capabilities are transferred to the student model through feature matching loss or mapping transformation. The main idea of ​​relation distillation is not to directly align the output or feature values ​​of individual samples, but rather to distill the structural relationships between samples, channels, or spatial locations, such as similarity, distance, angle, and correlation matrices, enabling the student model to learn the relative structural information in the teacher's representation.

[0004] During our research on knowledge distillation methods, we discovered certain problems and shortcomings in current feature distillation approaches. Due to differences in semantic information and mapping methods between the feature spaces of teacher and student features, directly aligning intermediate features between teacher and student models is challenging. Existing knowledge distillation techniques directly use teacher features as supervisory signals for student model distillation, leading to the student model learning redundant information that is not understood or accepted by shallow feature extractors.

[0005] In conclusion, the existing technology obviously has inconveniences and defects in practical use, so it is necessary to improve it. Summary of the Invention

[0006] To address the aforementioned shortcomings, the present invention aims to provide a teacher-model-guided student model diffusion self-distillation method, apparatus, storage medium, and electronic device, which can better train student models and thus improve the performance of student models.

[0007] To solve the above-mentioned technical problems, the present invention is implemented as follows:

[0008] In a first aspect, embodiments of the present invention provide a teacher-model-guided student model diffusion self-distillation method, comprising:

[0009] The initialization step involves loading the pre-trained teacher and student models and starting to train the student model.

[0010] The feature extraction step extracts teacher features and original student features from the training data, respectively.

[0011] The denoising sampling step involves guiding the diffusion model through the teacher model to denoise the original student features and generate corresponding denoised student features.

[0012] The self-distillation step involves self-distillation training based on the denoised student features and the original student features.

[0013] The calculation update step involves calculating the loss of the self-distillation training of the student model and updating the parameters of the student model.

[0014] The feature extraction step, the denoising sampling step, the self-distillation step, and the calculation update step are executed repeatedly until the student model training is completed.

[0015] According to the teacher-model-guided student-model diffusion self-distillation method of the present invention, the initialization step further includes:

[0016] Read the dataset;

[0017] Load the pre-trained teacher model and freeze the parameters of the teacher model;

[0018] Initialize the student model;

[0019] Begin training the student model.

[0020] According to the teacher-model-guided student-model diffusion self-distillation method of the present invention, the denoising sampling step further includes:

[0021] The diffusion model guides the teacher classifier of the teacher model to perform reverse denoising sampling on the original student features, thereby obtaining the denoised student features with teacher semantic information.

[0022] According to the teacher-model-guided student-model diffusion self-distillation method of the present invention, the denoising sampling step further includes:

[0023] The diffusion model is guided by the teacher classifier to generate denoised images.

[0024] Formula 1

[0025] Where Z is the normalization constant, It is an unconditional inverse denoising process that follows the denoising diffusion probability model, where θ is the diffusion model parameter. It is a pre-trained classifier, x t Let y represent the noisy image at step t, where y is the category label;

[0026] The original student characteristics Noisy feature x at step t t Where H, W, and C represent the height, width, and number of channels of the feature, respectively;

[0027] Using the teacher model as a pre-trained classifier, the original student features are... T-step teacher-guided diffusion sampling was performed, with the sampling format as follows:

[0028] Formula 2

[0029] The It is a conditional Markov process used to compare the noise predictor parameters θ with the teacher classifier. Under the condition that the original student characteristics are from x t+1 Sampled to x t , This refers to the teacher classifier. Based on the original student characteristics x t The inferred conditional probability of category y;

[0030] The teacher classifier typically includes a global average pooling layer and a linear weight matrix to output the class probability distribution;

[0031] Using the aforementioned diffusion model, a Gaussian distribution is used to predict the diffusion caused by x. t+1 The obtained x t :

[0032] Formula 3

[0033] in, The logarithmic form of Formula 3 is:

[0034] Formula 4

[0035] As the upper limit of the diffusion steps approaches infinity, it is derived that... Compared to this time item Its curvature is smaller, therefore in to Approximating with a first-order Taylor expansion:

[0036] Formula 5

[0037] in, Treating it as a constant, the logarithmic form of Formula 2 is further as follows:

[0038] Formula 6

[0039] in, Therefore, this conditional sampling strategy is approximately equivalent to unconditional Gaussian sampling, but its mean will be... The translation;

[0040] In addition, a gradient scaling factor is introduced. To control the guidance strength of the teacher classifier, the diffusion sampling process guided by the teacher model is represented as follows:

[0041] Formula 7

[0042] The gradient scaling factor k smooths the class probability distribution of the teacher model, with the effect being similar to... Proportional;

[0043] When k > 1, the distribution becomes sharper, the teacher classifier has stronger guidance, and can sample the denoised student features with higher fidelity. ;

[0044] When the teacher-guided diffusion sampling process passes After the step, the original student characteristics That is, x t The denoised student features will be converted into denoised versions. That is, x0, thus obtaining the denoised student features with the teacher semantic information. .

[0045] According to the teacher-model-guided student-model diffusion self-distillation method of the present invention, the computational update step further includes:

[0046] Calculate the first feature distillation loss of the denoised student features in each training session;

[0047] Calculate the second feature distillation loss of the denoised student features during overall training;

[0048] The parameters of the student model are updated based on the second feature distillation loss.

[0049] According to the teacher-model-guided student-model diffusion self-distillation method of the present invention, the step of calculating the first feature distillation loss of the denoised student features in each training session further includes:

[0050] Using the denoised student features and original student characteristics The mean square error is used to calculate the first feature distillation loss of the denoised student features in each training session. ;

[0051] Formula 8

[0052] in, These are the original student characteristics. These are the denoised student features;

[0053] The second feature distillation loss step in calculating the denoised student features during overall training further includes:

[0054] The second feature distillation loss of the denoised student features in the overall training is calculated using the overall loss function, which is as follows:

[0055] Formula 9

[0056] Where α, β, and γ are loss weights used to balance the losses of different components. The loss between the predicted values ​​of the student model and the true labels of the training data is denoted as . The loss is the difference between the probability distributions output by the student model and the teacher model. The loss of the diffusion model is trained for the teacher features.

[0057] According to the teacher-model-guided student model diffusion self-distillation method of the present invention, the method is applied to a teacher-model-guided student model diffusion self-distillation system, the system comprising: the teacher model, the student model, the diffusion model, and a student feature extractor. Teacher Feature Extractor Student classifier Teacher classifier Noise adapter ;

[0058] in, Represents student model parameters, y represents the teacher model parameters, x represents the input image, and y represents the category label corresponding to the input image.

[0059] Secondly, embodiments of the present invention provide a teacher-model-guided student-model diffusion self-distillation apparatus constructed based on any one of the methods described above, the apparatus comprising:

[0060] An initialization module is used to load the pre-trained teacher and student models and begin training the student model;

[0061] The feature extraction module is used to extract teacher features and original student features from the training data, respectively.

[0062] The denoising sampling module is used to guide the diffusion model through the teacher model to perform denoising sampling on the original student features and generate corresponding denoised student features.

[0063] The self-distillation module is used to perform self-distillation training based on the denoised student features and the original student features;

[0064] The calculation update module is used to calculate the loss of the self-distillation training of the student model and update the parameters of the student model;

[0065] The feature extraction module, the denoising sampling module, the self-distillation module, and the calculation update module are executed in a loop until the student model training is completed.

[0066] Thirdly, embodiments of the present invention provide a storage medium for storing a computer program for performing any of the methods described herein.

[0067] Fourthly, embodiments of the present invention provide an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the computer program to implement any of the methods described above.

[0068] Therefore, this invention provides a teacher-model-guided student model diffusion self-distillation technique, comprising: extracting teacher features and original student features from training data respectively; denoising and sampling the original student features using a diffusion model guided by the teacher model to generate corresponding denoised student features; performing self-distillation training based on the denoised student features and the original student features; calculating the loss of the student model's self-distillation training and updating the parameters of the student model; and repeating the steps until the student model training is completed. Therefore, this invention can eliminate the differences in mapping methods and feature distributions between the teacher model and the student model, thereby enabling better training of the student model and improving its performance. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating the teacher-guided student-model diffusion self-distillation method provided in Embodiment 1 of the present invention.

[0070] Figure 2 This is an overall structural diagram of the student model diffusion self-distillation system provided in Embodiment 2 of the present invention;

[0071] Figure 3 This is an overall flowchart of the teacher-guided student-model diffusion self-distillation method provided in Embodiment 3 of the present invention;

[0072] Figure 4 This is a schematic diagram of the structure of the teacher-guided student-model diffusion self-distillation device provided in Embodiment 4 of the present invention;

[0073] Figure 5 This is a schematic diagram of the structure of the electronic device provided in Embodiment 5 of the present invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0075] It should be noted that references to "an embodiment," "embodiment," "example embodiment," etc., in this specification refer to the described embodiment including specific features, structures, or characteristics, but not every embodiment must include these specific features, structures, or characteristics. Furthermore, such expressions do not refer to the same embodiment. Moreover, when describing specific features, structures, or characteristics in conjunction with embodiments, whether or not explicitly described, it is indicated that incorporating such features, structures, or characteristics into other embodiments is within the knowledge of those skilled in the art.

[0076] Furthermore, certain terms are used in the specification and subsequent claims to refer to specific components or parts. Those skilled in the art will understand that manufacturers may use different names or terms to refer to the same component or part. This specification and subsequent claims do not distinguish components or parts by differences in name, but rather by differences in function. The terms "comprising" and "including" used throughout the specification and subsequent claims are open-ended and should be interpreted as "including but not limited to." Additionally, the term "connection" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connections made through other means.

[0077] The following description, in conjunction with the accompanying drawings, details the teacher-guided student model diffusion self-distillation method provided by the present invention through specific embodiments and application scenarios.

[0078] Figure 1 This is a flowchart illustrating the teacher-guided student-model diffusion self-distillation method provided in Embodiment 1 of the present invention. The method includes:

[0079] Step S101: Initialization step, load the pre-trained teacher model and student model, and start training the student model.

[0080] Step S102, feature extraction step, extracts teacher features and original student features from the training data respectively.

[0081] Step S103, denoising sampling step, uses the teacher model to guide the diffusion model to denoise the original student features and generate corresponding denoised student features.

[0082] Step S104, self-distillation step, self-distillation training is performed based on denoised student features and original student features.

[0083] Step S105: Calculate the update step, calculate the loss of the student model's self-distillation training, and update the parameters of the student model.

[0084] The feature extraction step, denoising sampling step, self-distillation step, and calculation update step are executed cyclically, that is, steps S102 to S105 are executed cyclically until the student model training ends.

[0085] This invention proposes a teacher-guided student diffusion self-distillation method, designing a more comprehensive supervision signal during the knowledge distillation process to improve the performance of the student model. Unlike directly aligning intermediate features between the teacher and student models, this method utilizes a teacher classifier to guide a lightweight diffusion model, denoising and sampling student features. The denoised student features implicitly contain teacher knowledge and can serve as supervision signals for training the original student model. This invention eliminates the differences in mapping methods and feature distributions between the teacher and student models, thereby better training the student model and improving its performance.

[0086] Key Point 1: This invention proposes a teacher-model-guided student feature denoising method. Utilizing a diffusion model, student features are sampled into student features incorporating teacher knowledge through a teacher-model-guided diffusion denoising process.

[0087] Key point 2: This invention proposes a new feature distillation paradigm, which solves the problem that directly using teacher features as supervision signals leads to student models learning redundant information that is not understood or accepted by shallow feature extractors.

[0088] Technical Results: Compared with other existing knowledge distillation methods, the model optimized in this invention achieves higher accuracy in classification tasks.

[0089] Preferably, the initialization step further includes:

[0090] Sub-step 11: Read the dataset.

[0091] Sub-step 12: Load the pre-trained teacher model and freeze the parameters of the teacher model.

[0092] Sub-step 13: Initialize the student model.

[0093] Sub-step 14: Begin training the student model.

[0094] Preferably, the denoising sampling step further includes:

[0095] By using the teacher classifier of the teacher model to guide the diffusion model to perform reverse denoising sampling on the original student features, denoised student features with teacher semantic information are obtained.

[0096] Preferably, the denoising sampling step further includes:

[0097] Denoising image generation using a diffusion model guided by a teacher classifier:

[0098] Formula 1

[0099] Where Z is the normalization constant, It is an unconditional inverse denoising process that follows the denoising diffusion probability model, where θ is the diffusion model parameter. It is a pre-trained classifier, x t Let y represent the noisy image at step t, and y be the category label.

[0100] Original student characteristics Noisy feature x at step t t Where H, W, and C represent the height, width, and number of channels of the feature, respectively.

[0101] Using the teacher model as a pre-trained classifier, the original student features were analyzed. T-step teacher-guided diffusion sampling was performed, with the sampling format as follows:

[0102] Formula 2

[0103] It is a conditional Markov process used to compare the noise predictor parameters θ with the teacher classifier. Under the condition of x, the original student characteristics are changed from x t+1 Sampled to x t , Represents a teacher classifier. Based on the original student characteristics x t The inferred conditional probability of category y.

[0104] Teacher classifiers typically contain a global average pooling layer and a linear weight matrix to output the class probability distribution.

[0105] Using a diffusion model, a Gaussian distribution is used to predict the distribution of x. t+1 The obtained x t :

[0106] Formula 3

[0107] in, The logarithmic form of Formula 3 is:

[0108] Formula 4

[0109] As the upper limit of the diffusion steps approaches infinity, it is derived that... Compared to this time item Its curvature is smaller, therefore in to Approximating with a first-order Taylor expansion:

[0110] Formula 5

[0111] in, Treating it as a constant, the logarithmic form of Formula 2 is further as follows:

[0112] Formula 6

[0113] in, Therefore, this conditional sampling strategy is approximately equivalent to unconditional Gaussian sampling, but its mean will be... The translation.

[0114] In addition, a gradient scaling factor is introduced. To control the guidance strength of the teacher classifier, the diffusion sampling process guided by the teacher model is represented as follows:

[0115] Formula 7

[0116] The gradient scaling factor k smooths the class probability distribution of the teacher model, with the same effect as... Proportional.

[0117] When k > 1, the distribution becomes sharper, the teacher classifier has stronger guidance, and it can sample denoised student features with higher fidelity. .

[0118] When the teacher-guided diffusion sampling process passes After the step, the original student characteristics That is, x t These will be converted into denoised student features. That is, x0, thus obtaining denoised student features with teacher semantic information. .

[0119] Preferably, the calculation update step further includes:

[0120] Sub-step 31: Calculate the first feature distillation loss of the denoised student features in each training session.

[0121] Sub-step 32: Calculate the second feature distillation loss of the denoised student features in the overall training.

[0122] Sub-step 33: Update the parameters of the student model based on the second feature distillation loss.

[0123] Preferably, the step of calculating the first feature distillation loss of the denoised student features in each training session further includes:

[0124] Using denoised student characteristics and original student characteristics The mean square error is used to calculate the first feature distillation loss of the denoised student features in each training session. .

[0125] Formula 8

[0126] in, These are characteristics of the original students. It is a noise reduction feature for students.

[0127] Preferably, the step of calculating the second feature distillation loss of the denoised student features in the overall training further includes:

[0128] The second feature distillation loss of the denoised student features in the overall training is calculated using the overall loss function, which is as follows:

[0129] Formula 9

[0130] Where α, β, and γ are loss weights used to balance the losses of different components. The loss between the student model's predicted values ​​and the true labels of the training data. The loss is the difference between the probability distributions output by the student model and the teacher model. The loss of the diffusion model is used to train the teacher features.

[0131] Preferably, the method is applied to a teacher-model-guided student-model diffusion self-distillation system, the system comprising: a teacher model, a student model, a diffusion model, and a student feature extractor. Teacher Feature Extractor Student classifier Teacher classifier Noise adapter .

[0132] in, Represents student model parameters, y represents the teacher model parameters, x represents the input image, and y represents the category label corresponding to the input image.

[0133] Figure 2 This is an overall structural diagram of the student model diffusion self-distillation system provided in Embodiment 2 of the present invention, which consists of the following modules: Student Feature Extractor Teacher Feature Extractor Student classifier Teacher classifier Noise adapter It consists of a diffusion model. Represents student model parameters, y represents the teacher model parameters, x represents the input image, and y represents the category label corresponding to the input image.

[0134] Overall process: Before training, load the pre-trained teacher model and freeze its parameters, then initialize the student model. Then begin training the student model. Extract teacher and student features separately, and train the diffusion model using the teacher features. Denoise the student features according to the method described in the section on teacher-guided student feature denoising, and then calculate the feature distillation loss using formula (8) for the denoised student features. Finally, calculate the overall training loss using formula (9) and update the model parameters. Repeat the training steps until the student model training is complete. Refer to the execution process. Figure 3 The flowchart shown is an overall flowchart of the teacher-guided student-model diffusion self-distillation method provided in Embodiment 3 of the present invention.

[0135] The method mainly includes teacher model-guided student feature denoising and student feature self-distillation.

[0136] I. Teacher Model-Guided Student Feature Denoising: A teacher model is used to guide a diffusion model for reverse denoising of student features. Under the guidance of the teacher model, the distribution of student features shifts towards the distribution of teacher features, thus injecting meaningful semantic information from the teacher model into the student features. The classifier-guided diffusion model's denoising image generation process can be represented as follows:

[0137]

[0138] PCBA: Circuit Assembly, the first circuit board is the main board, the main board is the PCB, and the AIDC (Automatic Integrated Circuit) is the assembly of various components according to the surface-mount packaging process.

[0139] Where Z is the normalization constant. It follows the unconditional inverse denoising process of DDPM, where θ is the diffusion model parameter. It is a pre-trained classifier. t Let y represent the noisy image at step t, and y be the category label.

[0140] This invention transfers the image generation process to the field of knowledge distillation, incorporating student characteristics. Noisy feature x at step t t Where H, W, and C represent the height, width, and number of channels of the feature, respectively. A teacher model is used as the pre-trained classifier; in specific implementations, models such as ResNet34, ResNet50, or WRN-40-2 can be selected. For student features... T-step teacher-guided diffusion sampling can be performed, and the sampling form can be written as:

[0141]

[0142] here It is a conditional Markov process used to compare the noise predictor parameters θ with the teacher classifier. Under the condition of x, the student characteristics are changed from x t+1 Sampled to x t . Teacher classifier Based on student characteristics x t The conditional probability of the obtained class y is inferred. Teacher classifiers typically contain a global average pooling layer and a linear weight matrix to output the class probability distribution. Using the traditional diffusion model setting, a Gaussian distribution is used to predict the probability of class x. t+1 The obtained x t :

[0143]

[0144] in The logarithmic form of formula (3) is:

[0145]

[0146] When the upper limit of the diffusion steps approaches infinity, it can be deduced that... In this case, compared to item Its curvature is smaller. Therefore, we can... to Approximating with a first-order Taylor expansion:

[0147]

[0148] in It can be considered a constant. Furthermore, the logarithmic form of formula (2) is:

[0149]

[0150] in Therefore, this conditional sampling strategy can be approximated as unconditional Gaussian sampling, but its mean will be... The translation is as follows. Furthermore, we introduce a gradient scaling factor. This controls the guidance strength of the teacher classifier. In summary, the diffusion sampling process guided by the teacher model can be represented as:

[0151]

[0152] In theory, the gradient scaling factor k can smooth the class probability distribution of the teacher model, and its effect is similar to... Proportional. When k > 1, the distribution becomes sharper, meaning the teacher classifier has stronger guidance, thus sampling student features with higher fidelity. When the teacher-guided diffusion sampling process... After the step, the original student characteristics That is, x t These will be converted into denoised student features. That is, x0, thus obtaining student features with teacher model semantic information.

[0153] III. Student Characteristics Self-Distillation

[0154] This invention employs a diffusion model guided by a teacher classifier as a student feature sampling generator to denoise student features. The distribution is closer to the characteristics of teachers. Thus, the noise reduction student characteristics It can implicitly learn semantic information and expressive abilities similar to teacher features. Furthermore, since teacher models typically have stronger discriminative power, the classifier-guided diffusion sampling process can better preserve category-related information compared to traditional teacher-model-independent methods such as DiffKD (Knowledge Diffusion for Distillation). During the distillation process, student features are treated as "noisy versions" of teacher features. The inventors also introduced a noise adapter to initialize the noise level of student features to achieve better diffusion distillation results. After the above teacher-model-guided student feature diffusion sampling, the original student features... Denoising student features transformed into teacher-guided models Unlike most previous knowledge distillation methods that directly align student features with teacher features, the inventors' method utilizes denoised student features. Substitute teacher characteristics Distillation of original student features using simple mean squared error loss. Name the loss (DSKD, Teacher-guided studentDiffusion Self-Knowledge Distillation):

[0155]

[0156] This invention enables student models to acquire more meaningful knowledge from teacher models using a diffusion model. Traditional methods often introduce complex training schemes, loss functions, and feature alignment techniques due to significant differences between the feature representations of teacher and student models. This invention eliminates this distributional difference, making the information learned by students compatible with the teacher model. The overall loss function is as follows:

[0157]

[0158] Where α, β, and γ are loss weights used to balance the losses of different components. The loss between the student model's predicted values ​​and the true labels of the training data. The loss is the difference between the probability distributions output by the student model and the teacher model. The loss of the diffusion model is used to train the teacher features.

[0159] It should be noted that the teacher-model-guided student model diffusion self-distillation method provided in this embodiment of the invention can be executed by an electronic device, apparatus, or a control module within that apparatus for executing the method. This embodiment of the invention uses an apparatus to execute the method as an example to illustrate the teacher-model-guided student model diffusion self-distillation apparatus provided in this embodiment of the invention.

[0160] Figure 4 This is a schematic diagram of the structure of the teacher-guided student model diffusion self-distillation device provided in Embodiment 4 of the present invention. The device 100 includes an initialization module 10, a feature extraction module 20, a denoising sampling module 30, a self-distillation module 40, and a calculation and update module 50, wherein:

[0161] The initialization module 10 is used to load the pre-trained teacher model and student model, and start training the student model.

[0162] The feature extraction module 20 is used to extract teacher features and original student features from the training data, respectively.

[0163] The denoising sampling module 30 is used to denoise the original student features by guiding the diffusion model through the teacher model, and generate corresponding denoised student features.

[0164] The self-distillation module 40 is used for self-distillation training based on denoised student features and original student features.

[0165] The calculation update module 50 is used to calculate the loss of the student model's self-distillation training and update the parameters of the student model.

[0166] The feature extraction module 20, the denoising sampling module 30, the self-distillation module 40, and the calculation update module 50 are executed in a loop until the student model training is completed.

[0167] Preferably, the initialization module 10 further performs the following actions:

[0168] Action 11: Read the dataset.

[0169] Action 12: Load the pre-trained teacher model and freeze the parameters of the teacher model.

[0170] Action 13: Initialize the student model.

[0171] Action 14: Begin training the student model.

[0172] Preferably, the noise reduction sampling module 30 further performs the following actions:

[0173] By using the teacher classifier of the teacher model to guide the diffusion model to perform reverse denoising sampling on the original student features, denoised student features with teacher semantic information are obtained.

[0174] Preferably, the noise reduction sampling module 30 further performs the following actions:

[0175] Denoising image generation using a diffusion model guided by a teacher classifier:

[0176] Formula 1

[0177] Where Z is the normalization constant, It is an unconditional inverse denoising process that follows the denoising diffusion probability model, where θ is the diffusion model parameter. It is a pre-trained classifier, x t Let y represent the noisy image at step t, and y be the category label.

[0178] Original student characteristics Noisy feature x at step t t Where H, W, and C represent the height, width, and number of channels of the feature, respectively.

[0179] Using the teacher model as a pre-trained classifier, the original student features were analyzed. T-step teacher-guided diffusion sampling was performed, with the sampling format as follows:

[0180] Formula 2

[0181] It is a conditional Markov process used to compare the noise predictor parameters θ with the teacher classifier. Under the condition of x, the original student characteristics are changed from x t+1 Sampled to x t , Represents a teacher classifier. Based on the original student characteristics x t The inferred conditional probability of category y.

[0182] Teacher classifiers typically contain a global average pooling layer and a linear weight matrix to output the class probability distribution.

[0183] Using a diffusion model, a Gaussian distribution is used to predict the distribution of x. t+1 The obtained x t :

[0184] Formula 3

[0185] in, The logarithmic form of Formula 3 is:

[0186] Formula 4

[0187] As the upper limit of the diffusion steps approaches infinity, it is derived that... Compared to this time item Its curvature is smaller, therefore in to Approximating with a first-order Taylor expansion:

[0188] Formula 5

[0189] in, Treating it as a constant, the logarithmic form of Formula 2 is further as follows:

[0190] Formula 6

[0191] in, Therefore, this conditional sampling strategy is approximately equivalent to unconditional Gaussian sampling, but its mean will be... The translation.

[0192] In addition, a gradient scaling factor is introduced. To control the guidance strength of the teacher classifier, the diffusion sampling process guided by the teacher model is represented as follows:

[0193] Formula 7

[0194] The gradient scaling factor k smooths the class probability distribution of the teacher model, with the same effect as... Proportional.

[0195] When k > 1, the distribution becomes sharper, the teacher classifier has stronger guidance, and it can sample denoised student features with higher fidelity. .

[0196] When the teacher-guided diffusion sampling process passes After the step, the original student characteristics That is, x t These will be converted into denoised student features. That is, x0, thus obtaining denoised student features with teacher semantic information. .

[0197] Preferably, the calculation update module 50 further performs the following actions:

[0198] Action 51: Calculate the first feature distillation loss of the denoised student features in each training session.

[0199] Action 52: Calculate the second feature distillation loss of the denoised student features in the overall training.

[0200] Action 53: Update the parameters of the student model based on the second feature distillation loss.

[0201] Preferably, action 5 further includes:

[0202] Using denoised student characteristics and original student characteristics The mean square error is used to calculate the first feature distillation loss of the denoised student features in each training session. .

[0203] Formula 8

[0204] in, These are characteristics of the original students. It is a noise reduction feature for students.

[0205] The action 52 further includes:

[0206] The second feature distillation loss of the denoised student features in the overall training is calculated using the overall loss function, which is as follows:

[0207] Formula 9

[0208] Where α, β, and γ are loss weights used to balance the losses of different components. The loss between the student model's predicted values ​​and the true labels of the training data. The loss is the difference between the probability distributions output by the student model and the teacher model. The loss of the diffusion model is used to train the teacher features.

[0209] like Figure 2 As shown, the teacher-model-guided student-model diffusion autodistillation system 100 includes: a teacher model, a student model, a diffusion model, and a student feature extractor. Teacher Feature Extractor Student classifier Teacher classifier Noise adapter .

[0210] in, Represents student model parameters, y represents the teacher model parameters, x represents the input image, and y represents the category label corresponding to the input image.

[0211] The teacher-model-guided student-model diffusion self-distillation apparatus provided in this embodiment of the invention can achieve... Figures 1-3 The various processes implemented in the teacher-model-guided student-model diffusion self-distillation method embodiment shown are not described in detail here to avoid repetition.

[0212] The teacher-model-guided student model diffusion self-distillation apparatus provided in this invention includes: extracting teacher features and original student features from training data; denoising and sampling the original student features using a diffusion model guided by the teacher model to generate corresponding denoised student features; performing self-distillation training based on the denoised student features and the original student features; calculating the loss of the student model's self-distillation training and updating the parameters of the student model; and repeating the steps until the student model training is complete. Therefore, this invention can eliminate the differences in mapping methods and feature distributions between the teacher model and the student model, thereby enabling better training of the student model and improving its performance.

[0213] The present invention also provides a storage medium for storing, for example, Figures 1-3 The computer program for the student model diffusion from distillation method guided by any teacher model is described. For example, computer program instructions, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the present invention through the operation of the computer, achieving the same technical effect; to avoid repetition, these will not be elaborated further here. The program instructions for invoking the methods of the present invention may be stored in fixed or removable storage media, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in the storage media of a computer device operating according to the program instructions.

[0214] According to one embodiment of the present invention, the present invention also provides such a Figure 4The illustrated electronic device 400 may optionally include a storage medium 200 for storing a computer program and a processor 300 for executing the computer program. When the computer program is executed by the processor 300, it implements any of the aforementioned teacher-model-guided student-model diffusion self-distillation methods, triggering the electronic device 400 to execute methods and / or technical solutions based on the foregoing embodiments, achieving the same technical effect. To avoid repetition, these will not be elaborated further here. It should be noted that the electronic devices in this embodiment include mobile electronic devices and non-mobile electronic devices. For example, mobile electronic devices may be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, super mobile personal computers, netbooks, or personal digital assistants, etc., while non-mobile electronic devices may be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This embodiment does not specifically limit the scope of the invention.

[0215] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present invention (including associated data structures) can be stored in a computer-readable recording medium, such as RAM memory, a magnetic or optical drive, a floppy disk, or similar devices. Furthermore, some steps or functions of the present invention can be implemented in hardware, for example, as circuitry that works with a processor to perform the various steps or functions.

[0216] This invention can be implemented on a computer as a computer-based method, or in dedicated hardware, or a combination of both. Executable code or portions thereof for the method according to the invention can be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Optionally, the computer program product includes non-transitory program code components stored on a computer-readable medium so as to execute the method according to the invention when the program product is executed on a computer.

[0217] In an optional embodiment, the computer program includes computer program code components adapted to perform all the steps of the method according to the invention when the computer program is run on a computer. Optionally, the computer program is embodied on a computer-readable medium.

[0218] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0219] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A teacher-model-guided student-model diffusion self-distillation method, characterized in that, include: The initialization step involves loading the pre-trained teacher and student models and starting to train the student model. The feature extraction step extracts teacher features and original student features from the training data, respectively; The denoising sampling step involves guiding the diffusion model through the teacher model to denoise the original student features and generate corresponding denoised student features. The self-distillation step involves self-distillation training based on the denoised student features and the original student features. The calculation update step involves calculating the loss of the self-distillation training of the student model and updating the parameters of the student model. The feature extraction step, the denoising sampling step, the self-distillation step, and the calculation update step are executed repeatedly until the student model training is completed.

2. The teacher-model-guided student-model diffusion self-distillation method according to claim 1, characterized in that, The initialization step further includes: Read the dataset; Load the pre-trained teacher model and freeze the parameters of the teacher model; Initialize the student model; Begin training the student model.

3. The teacher-model-guided student-model diffusion self-distillation method according to claim 1, characterized in that, The denoising sampling step further includes: The diffusion model guides the teacher classifier of the teacher model to perform reverse denoising sampling on the original student features, thereby obtaining the denoised student features with teacher semantic information.

4. The teacher-model-guided student-model diffusion self-distillation method according to claim 3, characterized in that, The denoising sampling step further includes: The diffusion model is guided by the teacher classifier to generate denoised images. Official 1 Where Z is the normalization constant, It is an unconditional inverse denoising process that follows the denoising diffusion probability model, where θ is the diffusion model parameter. It is a pre-trained classifier, x t Let y represent the noisy image at step t, where y is the category label; The original student characteristics Noisy feature x at step t t Where H, W, and C represent the height, width, and number of channels of the feature, respectively; Using the teacher model as a pre-trained classifier, the original student features are... T-step teacher-guided diffusion sampling was performed, with the sampling format as follows: Official 2 The It is a conditional Markov process used to compare the noise predictor parameters θ with the teacher classifier. Under the condition that the original student characteristics are from x t+1 Sampled to x t , This refers to the teacher classifier. Based on the original student characteristics x t The inferred conditional probability of category y; The teacher classifier typically includes a global average pooling layer and a linear weight matrix to output the class probability distribution; Using the aforementioned diffusion model, a Gaussian distribution is used to predict the diffusion caused by x. t+1 The obtained x t : Official 3 in, The logarithmic form of Formula 3 is: Official 4 As the upper limit of the diffusion steps approaches infinity, it is derived that... Compared to this time item The curvature is smaller, therefore in to Approximating with a first-order Taylor expansion: Official 5 in, Treating it as a constant, the logarithmic form of Formula 2 is further as follows: Official 6 in, Therefore, this conditional sampling strategy is approximately equivalent to unconditional Gaussian sampling, but its mean will be... The translation; In addition, a gradient scaling factor is introduced. To control the guidance strength of the teacher classifier, the diffusion sampling process guided by the teacher model is represented as follows: Official 7 The gradient scaling factor k smooths the class probability distribution of the teacher model, with the effect being similar to... Proportional; When k > 1, the distribution becomes sharper, the teacher classifier has stronger guidance, and can sample the denoised student features with higher fidelity. ; When the teacher-guided diffusion sampling process passes After the step, the original student characteristics That is, x t The denoised student features will be converted into denoised versions. That is, x0, thus obtaining the denoised student features with the teacher semantic information. .

5. The teacher-model-guided student-model diffusion self-distillation method according to claim 1, characterized in that, The calculation and update step further includes: Calculate the first feature distillation loss of the denoised student features in each training session; Calculate the second feature distillation loss of the denoised student features during overall training; The parameters of the student model are updated based on the second feature distillation loss.

6. The teacher-model-guided student-model diffusion self-distillation method according to claim 5, characterized in that, The step of calculating the first feature distillation loss of the denoised student features in each training session further includes: Using the denoised student features and original student characteristics The mean square error is used to calculate the first feature distillation loss of the denoised student features in each training session. ; Official 8 in, These are the original student characteristics. These are the denoised student features; The second feature distillation loss step in calculating the denoised student features during overall training further includes: The second feature distillation loss of the denoised student features in the overall training is calculated using the overall loss function, which is as follows: Official 9 Where α, β, and γ are loss weights used to balance the losses of different components. The loss between the predicted values ​​of the student model and the true labels of the training data is denoted as . The loss is the difference between the probability distributions output by the student model and the teacher model. The loss of the diffusion model is trained for the teacher features.

7. The teacher-model-guided student-model diffusion self-distillation method according to claim 1, characterized in that, The method is applied to a teacher-model-guided student-model diffusion self-distillation system, the system comprising: the teacher model, the student model, the diffusion model, and a student feature extractor. Teacher Feature Extractor Student classifier Teacher classifier Noise adapter ; in, Represents student model parameters, y represents the teacher model parameters, x represents the input image, and y represents the category label corresponding to the input image.

8. A teacher-model-guided student-model diffusion self-distillation apparatus constructed based on the method described in any one of claims 1 to 7, characterized in that, The device includes: An initialization module is used to load the pre-trained teacher and student models and begin training the student model; The feature extraction module is used to extract teacher features and original student features from the training data, respectively. The denoising sampling module is used to guide the diffusion model through the teacher model to perform denoising sampling on the original student features and generate corresponding denoised student features. The self-distillation module is used to perform self-distillation training based on the denoised student features and the original student features; The calculation update module is used to calculate the loss of the self-distillation training of the student model and update the parameters of the student model; The feature extraction module, the denoising sampling module, the self-distillation module, and the calculation update module are executed in a loop until the student model training is completed.

9. A storage medium, characterized in that, Used to store a computer program for performing the method according to any one of claims 1 to 7.

10. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images