A model compression method and system based on data protection scenario
Through the combination of multi-teacher integration and comparative interaction loss functions, model compression is achieved without data, solving the accuracy and diversity of model compression in data protection scenarios, and the generated data has higher generalization and diversity , suitable for different network structures.
Patent Information
- Application Number
- CN202210220060.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-08
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-03-08
AI Technical Summary
The existing model compression method cannot effectively compress the model in data protection scenarios, and the generated data lacks diversity and generalization, resulting in low accuracy of the compressed model.
Using a multi-teacher integrated model inversion method, combining one-hot predictive loss function, feature regularization loss function and anti-distillation loss function, optimized through internal teacher comparison and teacher-student comparison loss function, the generator generates synthetic data and passes the Multi-teachers integrate distillation loss function to optimize student models to achieve model compression without data.
It improves the accuracy of model compression and the diversity and generalization of generated data, and the generated pictures are closer to the distribution of the original data sets, suitable for student networks with different network structures, reducing the computational overhead and time of generating different data sets.
Smart Images

Figure CN114565810B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of model compression, and in particular to a model compression method and system based on a data protection scenario. Background Art
[0002] In recent years, with the continuous development of deep learning computing power, deep learning models have become increasingly large, and when we need to deploy the model to the terminal device, we have to compress the model. Knowledge Distillation (KD) is a popular compression method that learns a lightweight student model by transferring knowledge from redundant teacher models to imitate representation capabilities. In most existing KD methods, knowledge is successfully transferred to the student model using logits or feature information from the teacher, but the entire training data needs to be accessed.
[0003] Unfortunately, due to privacy, confidentiality, or transmission restrictions, the original training samples of the pre-trained models are usually not available. For example, the medical data of patients are confidential and will not be shared publicly to leak the privacy of patients. Without the help of data, these methods may not be applicable.
[0004] The existing method is to replace the original data with artificially synthesized training data. However, the data generated by the existing method is somewhat different from the original data, lacking data diversity and generalization. The accuracy of the compressed model is not satisfactory. Summary of the invention
[0005] The purpose of the present invention is to provide a model compression method and system based on a data protection scenario, which is independent of the original training data of the model, effectively compresses the data-free model and improves the accuracy of model compression.
[0006] To achieve the above object, the present invention provides the following solutions:
[0007] A model compression method based on a data protection scenario, the method comprising:
[0008] Preset multiple trained teacher models and randomly initialized student models and generators under the same dataset;
[0009] Construct one-hot prediction loss function, feature regularization loss function and adversarial distillation loss function respectively;
[0010] Combining the one-hot prediction loss function, feature regularization loss function, and adversarial distillation loss function, we obtain the model inversion loss function for data-free distillation of the generator under multi-teacher conditions.
[0011] Construct a student model to mimic the multi-teacher ensemble distillation loss function output by the teacher model;
[0012] Establish the internal teacher contrast loss function and the teacher-student contrast loss function respectively;
[0013] The generator is optimized using the model inversion loss function of data-free distillation, the teacher internal contrast loss function, and the teacher-student contrast loss function. The image synthesized by the generator after this optimization is output, and the image synthesized by the generator after this optimization is input into the student model and each trained teacher model respectively. The student model is optimized using the multi-teacher integrated distillation loss function.
[0014] The optimization operation is repeated until the student model converges to obtain a compressed student model.
[0015] Optionally, the one-hot prediction loss function is In the formula, is the one-hot prediction loss, CE is the cross entropy loss, Synthesized images for the generator The output after inputting into the trained teacher model, c is the predefined class;
[0016] The feature regularization loss function is In the formula, is the feature regularization loss, Synthesized images for the generator The mean value obtained by the lth BN layer after inputting into the trained teacher model, Synthesized images for the generator The variance obtained by the lth BN layer after inputting into the trained teacher model, F(μ l (x)|X) is the mean value obtained by the lth BN layer after the input image x is input into the trained teacher model, The variance obtained by the lth BN layer after the input image x is input into the trained teacher model;
[0017] The adversarial distillation loss function is In the formula, To combat distillation loss, KL is the Kulbeck-Leible divergence, Synthesized images for the generator The output of the student model is input, and τ is the temperature.
[0018] Optionally, the one-hot prediction loss function, the feature regularization loss function and the adversarial distillation loss function are combined to obtain a model inversion loss function of the data-free distillation of the generator under multi-teacher conditions, specifically including:
[0019] Combining the one-hot prediction loss function, feature regularization loss function and adversarial distillation loss function, the model inversion loss function of the data-free distillation of the generator under the single teacher condition is obtained as In the formula, is the model inversion loss of the data-free distillation of the generator under the single teacher condition, λ1, λ2 and λ3 are the first, second and third balanced parameters respectively;
[0020] According to the model inversion loss function of the data-free distillation of the generator under the single-teacher condition, the model inversion loss function of the data-free distillation of the generator under the multi-teacher condition is constructed as In the formula, Model inversion loss for data-free distillation of generators under multi-teacher conditions, For the one-hot prediction loss with multiple teacher information, is the feature regularization loss with multi-teacher information, is the adversarial distillation loss with multi-teacher information, z is the noise input, θ g is the parameter of the generator;
[0021]
[0022]
[0023]
[0024] in, is the output of the M number of integrated teacher models, is the output of the mth trained teacher model; Synthesized images for the generator The mean value obtained by the lth BN layer after inputting into the mth trained teacher model, Synthesized images for the generator The variance obtained by the lth BN layer after inputting into the mth trained teacher model, is the mean value obtained by the lth BN layer after the input image x is input to the mth trained teacher model, It is the variance obtained by the lth BN layer after the input image x is input to the mth trained teacher model.
[0025] Optionally, the multi-teacher ensemble distillation loss function is
[0026]
[0027] In the formula, Ensemble distillation loss for multi-teachers.
[0028] Optionally, the process of establishing the teacher internal contrast loss function is:
[0029] Performing data augmentation on each image in the same batch of images synthesized by the generator, and inputting each of the images and the data augmented images into each trained teacher model respectively, to obtain a representation of each of the images output by each trained teacher model and a representation of the data augmented images;
[0030] Randomly select an image from the same batch of images synthesized by the generator as the image to be tested;
[0031] The representation of the image to be tested and the representation of the image to be tested after data enhancement are taken as positive sample pairs, the representation of images other than the image to be tested in the same batch of images synthesized by the generator are taken as negative samples, and the representation of historical images synthesized by the generator are taken as negative samples;
[0032] According to the positive sample pairs and negative samples, the teacher’s internal contrast loss function is determined as In the formula, is the intra-teacher contrast loss, is a positive sample pair, is the representation of the image to be tested output by the mth trained teacher model, is the parameter of the mth head mapping network, h is the head projection network, is the representation of the image to be tested after data augmentation output by the mth trained teacher model, is the i-th negative sample corresponding to the m-th trained teacher model, K is the number of negative samples, τ1 is the first temperature hyperparameter, and sim() is the cosine similarity.
[0033] Optionally, the process of establishing the teacher-student comparison loss function is:
[0034] Each image in the same batch of images synthesized by the generator is input into each trained teacher model and student model respectively, and the representation of each image by each trained teacher model and the representation of each image by the student model are obtained;
[0035] The representation of the same image by the student model and the trained teacher model which is isomorphic to the student model is used as a negative sample pair; the isomorphism means that the student model and the teacher model belong to the same network structure series;
[0036] The representation of the same image by a student model and a trained teacher model that is heterogeneous with the student model is defined as a positive sample pair; the heterogeneity is that the student model and the teacher model belong to different network structure series;
[0037] According to the negative sample pairs and the defined positive sample pairs, the teacher-student comparison loss function is determined as: In the formula, To compare the losses between teachers and students, is the query of the student model constructed from the rth image in the current batch of images synthesized by the generator, Synthesized images for the generator The output of the student model, θ h is the parameter of the head mapping network, h is the head projection network, is the query of the mth trained teacher model heterogeneous with the student model constructed from the rth image in the current batch of images synthesized by the generator, D(s) is the set of teacher indices heterogeneous with the student model, N is the number of images in the current batch of images, τ2 is the first temperature hyperparameter, Neg is the set of negative pairs, I(s) is the teacher index set isomorphic to the student model, is the jth negative sample in the representation of the historical image output by the student model, J is the number of negative samples in the representation of the historical image output by the student model, is the query of the student network constructed from the jth image in the historical images synthesized by the generator, is a query for a teacher model that is isomorphic to the student network model.
[0038] Optionally, the generator is optimized using the model inversion loss function, the teacher internal contrast loss function and the teacher-student contrast loss function without data distillation, the image synthesized by the generator after this optimization is output, and the image synthesized by the generator after this optimization is input into the student model and each trained teacher model respectively, and the student model is optimized by the multi-teacher integrated distillation loss function, specifically including:
[0039] According to the model inversion loss function without data distillation, the teacher internal contrast loss function and the teacher-student contrast loss function, the optimized loss function of the generator is determined as In the formula, is the optimization loss of the generator, is the total function of the internal contrast loss function of the teacher and the contrast loss function of the teacher and students, λ is and The balance parameters between
[0040] Initialize the parameters of the generator θ g , the parameters of the student model θ s and image libraries;
[0041] According to the noise input z, use the generator to synthesize the current batch of images;
[0042] According to the current batch of images, using the optimization loss function to calculate the current optimization loss of the generator;
[0043] The parameters of the generator θ g Updated to Among them, η is the coefficient, is the gradient operator;
[0044] Store the current batch of images synthesized by the generator into the image library;
[0045] Extract a batch of images from the image library;
[0046] Based on the extracted batch of images, the current loss of the student model is calculated using the multi-teacher ensemble distillation loss function;
[0047] The parameters θ of the student model are s Updated to
[0048] Optionally, the student model convergence means that the current loss of the student model is equal to the loss of the student model calculated in the previous iteration.
[0049] A model compression system based on a data protection scenario, the system comprising:
[0050] The preset module is used to preset multiple trained teacher models and randomly initialized student models and generators under the same dataset;
[0051] Three loss function building modules, used to build one-hot prediction loss function, feature regularization loss function and adversarial distillation loss function respectively;
[0052] The combination module is used to combine the one-hot prediction loss function, the feature regularization loss function and the adversarial distillation loss function to obtain the model inversion loss function of the data-free distillation of the generator under multi-teacher conditions;
[0053] A multi-teacher ensemble distillation loss function building module is used to build a multi-teacher ensemble distillation loss function where the student model imitates the output of the teacher model;
[0054] A contrast loss function establishment module is used to establish the teacher's internal contrast loss function and the teacher-student contrast loss function respectively;
[0055] The optimization module is used to optimize the generator using the model inversion loss function of data-free distillation, the teacher internal contrast loss function, and the teacher-student contrast loss function, output the image synthesized by the generator after this optimization, and input the image synthesized by the generator after this optimization into the student model and each trained teacher model respectively, and optimize the student model through the multi-teacher integrated distillation loss function;
[0056] The loop module is used to repeatedly perform optimization operations until the student model converges to obtain a compressed student model.
[0057] Optionally, the optimization module specifically includes:
[0058] The optimized loss function determination submodule is used to determine the optimized loss function of the generator according to the model inversion loss function, the teacher internal contrast loss function and the teacher-student contrast loss function of the data-free distillation model. In the formula, is the optimization loss of the generator, is the total function of the internal contrast loss function of the teacher and the contrast loss function of the teacher and students, λ is and The balance parameters between
[0059] Initialization submodule, used to initialize the parameters θ of the generator g , the parameters of the student model θ s and image libraries;
[0060] The synthesis submodule is used to synthesize the current batch of images using the generator according to the noise input z;
[0061] An optimization loss calculation submodule, used to calculate the current optimization loss of the generator using the optimization loss function according to the current batch of images;
[0062] The generator parameter update submodule is used to update the generator parameters θ g Updated to Among them, η is the coefficient, is the gradient operator;
[0063] A storage submodule, used to store the current batch of images synthesized by the generator into an image library;
[0064] The extraction submodule is used to extract a batch of images from the image library;
[0065] The student model current loss calculation submodule is used to calculate the current loss of the student model based on a batch of extracted images using a multi-teacher integrated distillation loss function;
[0066] The student model parameter update submodule is used to update the parameters θ of the student model s Updated to
[0067] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0068] The present invention discloses a model compression method and system based on a data protection scenario. First, the traditional model inversion is extended to the model inversion based on multi-teacher integration, and the richer information from the teacher is fully inverted to generate generalizable data. In addition, a contrast interaction regularization between multiple teachers and students is proposed, which includes intra-teacher contrast and teacher-student contrast. The intra-teacher contrast is used to gradually synthesize new samples with different patterns from historical samples, while the teacher-student contrast aims to push the relationship between students and isomorphic teachers away from the non-isomorphic relationship in the representation space to improve the diversity of synthetic data. And the image generation and knowledge transfer processes are trained in an adversarial manner to simultaneously learn the student model and generate synthetic data. The present invention does not rely on the original training data of the model. Through the knowledge distillation of the multi-teacher model and the introduction of a loss function based on contrastive learning, the data-free model compression method is effectively compressed and the compressed model has a higher accuracy rate. BRIEF DESCRIPTION OF THE DRAWINGS
[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0070] Figure 1 A framework diagram of a model compression method based on a data protection scenario provided by the present invention;
[0071] Figure 2 A flow chart of the optimization operation provided by the present invention;
[0072] Figure 3 A comparison diagram of the picture generation effect provided by an embodiment of the present invention;
[0073] Figure 4 A data distribution comparison diagram provided by an embodiment of the present invention; Figure 4 (a) is the data distribution diagram of MTCKI. Figure 4 (b) is the data distribution diagram of CMI. Figure 4 (c) is the data distribution diagram of CIFAR-10;
[0074] Figure 5 Training loss curves for different methods;
[0075] Figure 6 Comparison of images synthesized at different epochs; Figure 6 (a) is the image synthesized at the 10th epoch. Figure 6 (b) is the image synthesized at the 100th epoch. DETAILED DESCRIPTION
[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0077] The purpose of the present invention is to provide a model compression method and system based on a data protection scenario, which is independent of the original training data of the model, effectively compresses the data-free model and improves the accuracy of model compression.
[0078] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0079] The present invention provides a model compression method based on a data protection scenario, referring to Figure 1 , including the following steps:
[0080] Step 1: Preset multiple trained teacher models and randomly initialized student models and generators under the same dataset.
[0081] Randomly select some teacher models trained on the same dataset, as well as randomly initialized student models and generators. Input the random vector into the generator to get the synthesized image.
[0082] Step 2: Construct the one-hot prediction loss function, feature regularization loss function, and adversarial distillation loss function respectively.
[0083] 2-1, construct one-hot prediction loss function
[0084] Each image generated by the generator should belong to a category, so we calculate the cross entropy loss CE by inputting the image into the teacher network and obtaining the logits with the category with the largest logits value.
[0085] The one-hot prediction loss function is:
[0086] In the formula, is the one-hot prediction loss, CE is the cross entropy loss, Synthesized images for the generator The output after inputting into the trained teacher model, c is the predefined class;
[0087] 2-2, construct feature regularization loss function
[0088] BN layers have been widely used in CNNs to measure the running average statistics (e.g., running mean μ(x) and running variance σ) 2 (x)) during training. After training, these statistics store rich information about X.
[0089] For each batch of images generated by the generator, input to all teacher networks, the mean value obtained by the BatchNormalization (BN) layer is and variance and the mean E(μ l (x)|X) and variance Calculate the second norm and then accumulate the second norm loss obtained for each Batch Normalization layer.
[0090] The feature regularization loss function is In the formula, is the feature regularization loss, Synthesized images for the generator The mean value obtained by the lth BN layer after inputting into the trained teacher model, Synthesized images for the generator The variance obtained by the lth BN layer after inputting into the trained teacher model, E(μ l (x)|X) is the mean value obtained by the lth BN layer after the input image x is input into the trained teacher model, The variance obtained by the lth BN layer after the input image x is input into the trained teacher model;
[0091] 2-3. Constructing adversarial distillation loss function
[0092] An adversarial distillation loss is proposed to encourage synthetic images to produce larger differences between students and teachers. The distribution of the images synthesized by the generator is distanced after passing through all teacher networks and student networks to ensure that the generated images are diverse.
[0093] The adversarial distillation loss function is In the formula, To combat distillation loss, KL is the Kulbeck-Leible divergence, Synthesized images for the generator The output of the student model is input, and τ is the temperature.
[0094] The generator G can generate generalized images by minimizing Eq. 1 because it inverts the knowledge from multiple pre-trained teachers. However, the synthesized images still lack diversity, which may lead to overfitting during retraining. To this end, a contrastive interaction between multiple teachers and a student is proposed to improve data diversity and produce high-fidelity images.
[0095] Step 3: Combine the one-hot prediction loss function, feature regularization loss function, and adversarial distillation loss function to obtain the model inversion loss function for data-free distillation of the generator under multi-teacher conditions.
[0096] Multi-view structures are very common and exist in many real-world datasets. There are multiple features in these data that can be used to correctly classify images. Bird images can be classified as birds by observing the shape of their wings, body size, or beak. Even if students can extract all the features learned by their teachers, they still cannot "see" the features that the teacher did not discover, thus limiting the student's performance. Even if some models lack the views from which a single student can learn multi-view knowledge, the ensemble can collect almost all of them. We first consider multiple ensemble teachers to build a reliable multi-branch model. We choose the average final output of all teachers as the model prediction, and in addition, we use different teachers to obtain various statistical knowledge to improve the generability and diversity of synthetic images.
[0097] In one example, the model inversion loss function for data-free distillation of the generator under multi-teacher conditions is obtained by:
[0098] Combining the one-hot prediction loss function, feature regularization loss function and adversarial distillation loss function, the model inversion loss function of the data-free distillation of the generator under the single teacher condition is obtained as In the formula, is the model inversion loss of the data-free distillation of the generator under the single teacher condition, λ1, λ2 and λ3 are the first, second and third balanced parameters respectively;
[0099] According to the model inversion loss function of the data-free distillation of the generator under the single-teacher condition, the model inversion loss function of the data-free distillation of the generator under the multi-teacher condition is constructed as In the formula, Model inversion loss for data-free distillation of generators under multi-teacher conditions, For the one-hot prediction loss with multiple teacher information, is the feature regularization loss with multi-teacher information, is the adversarial distillation loss with multi-teacher information, z is the noise input, θ g is the parameter of the generator;
[0100]
[0101]
[0102]
[0103] in, is the output of the M number of integrated teacher models, is the output of the mth trained teacher model; Synthesized images for the generator The mean value obtained by the lth BN layer after inputting into the mth trained teacher model, Synthesized images for the generator The variance obtained by the lth BN layer after inputting into the mth trained teacher model, is the mean value obtained by the lth BN layer after the input image x is input to the mth trained teacher model, It is the variance obtained by the lth BN layer after the input image x is input to the mth trained teacher model.
[0104] The student network is optimized by multi-teacher ensemble distillation loss so that the student network imitates the output of the teacher network.
[0105] Step 4: Construct a student model to imitate the multi-teacher ensemble distillation loss function output by the teacher model.
[0106] For each pre-trained teacher model and student model input picture, the logits output by the teacher network and the student network are obtained, the logits output by the teacher network are averaged to obtain the integrated logits, and then the KL divergence of the integrated logits and the logits output by the student network is calculated as the multi-teacher integrated distillation loss function.
[0107] Exemplarily, the multi-teacher ensemble distillation loss function is
[0108]
[0109] In the formula, Ensemble distillation loss for multi-teachers.
[0110] Step 5: Establish the internal teacher comparison loss function and the teacher-student comparison loss function respectively.
[0111] 5-1, the process of establishing the teacher's internal contrast loss function is:
[0112] Performing data augmentation on each image in the same batch of images synthesized by the generator, and inputting each of the images and the data augmented images into each trained teacher model respectively, to obtain a representation of each of the images output by each trained teacher model and a representation of the data augmented images;
[0113] Randomly select an image from the same batch of images synthesized by the generator as the image to be tested;
[0114] The representation of the image to be tested and the representation of the image to be tested after data enhancement are taken as positive sample pairs, the representation of images other than the image to be tested in the same batch of images synthesized by the generator are taken as negative samples, and the representation of historical images synthesized by the generator are taken as negative samples;
[0115] According to the positive sample pairs and negative samples, the teacher’s internal contrast loss function is determined as In the formula, is the intra-teacher contrast loss, is a positive sample pair, is the representation of the image to be tested output by the mth trained teacher model, is the parameter of the mth head mapping network, h is the head projection network, is the representation of the image to be tested after data augmentation output by the mth trained teacher model, is the i-th negative sample corresponding to the m-th trained teacher model, K is the number of negative samples, τ1 is the first temperature hyperparameter, and sim() is the cosine similarity.
[0116] 5-2, the process of establishing the teacher-student comparison loss function is:
[0117] Each image in the same batch of images synthesized by the generator is input into each trained teacher model and student model respectively, and the representation of each image by each trained teacher model and the representation of each image by the student model are obtained;
[0118] The representation of the same image by the student model and the trained teacher model which is isomorphic to the student model is used as a negative sample pair; the isomorphism means that the student model and the teacher model belong to the same network structure series; which network structure series the student model and the teacher model belong to is known at the time of selection;
[0119] The representation of the same image by a student model and a trained teacher model that is heterogeneous with the student model is defined as a positive sample pair; the heterogeneity is that the student model and the teacher model belong to different network structure series;
[0120] According to the negative sample pairs and the defined positive sample pairs, the teacher-student comparison loss function is determined as: In the formula, To compare the losses between teachers and students, is the query of the student model constructed from the rth image in the current batch of images synthesized by the generator, Synthesized images for the generator The output of the student model, θ h is the parameter of the head mapping network, h is the head projection network, is the query of the mth trained teacher model heterogeneous with the student model constructed from the rth image in the current batch of images synthesized by the generator, D(s) is the set of teacher indices heterogeneous with the student model, N is the number of images in the current batch of images, τ2 is the first temperature hyperparameter, Neg is the set of negative pairs, I(s) is the teacher index set isomorphic to the student model, is the jth negative sample in the representation of the historical image output by the student model, J is the number of negative samples in the representation of the historical image output by the student model, is the query of the student network constructed from the jth image in the historical images synthesized by the generator, is a query for a teacher model that is isomorphic to the student network model.
[0121] Step 6: Use the model inversion loss function of data-free distillation, the teacher internal contrast loss function, and the teacher-student contrast loss function to optimize the generator, output the image synthesized by the generator after this optimization, and input the image synthesized by the generator after this optimization into the student model and each trained teacher model respectively, and optimize the student model through the multi-teacher integrated distillation loss function.
[0122] Exemplarily, step 6 specifically includes:
[0123] According to the model inversion loss function without data distillation, the teacher internal contrast loss function and the teacher-student contrast loss function, the optimized loss function of the generator is determined as In the formula, is the optimization loss of the generator, is the total function of the internal contrast loss function of the teacher and the contrast loss function of the teacher and students, λ is and The balance parameter between; preferably, the value of λ is 0.2;
[0124] Initialize the parameters of the generator θ g , the parameters of the student model θ s and image libraries;
[0125] According to the noise input z, use the generator to synthesize the current batch of images;
[0126] According to the current batch of images, using the optimization loss function to calculate the current optimization loss of the generator;
[0127] The parameters of the generator θ g Updated to Among them, η is the coefficient, is the gradient operator;
[0128] Store the current batch of images synthesized by the generator into the image library;
[0129] Extract a batch of images from the image library;
[0130] Based on the extracted batch of images, the current loss of the student model is calculated using the multi-teacher ensemble distillation loss function;
[0131] The parameters θ of the student model are s Updated to
[0132] Step 7: Repeat the optimization operation until the student model converges to obtain a compressed student model.
[0133] The student model converges when the current loss of the student model is equal to the loss of the student model calculated in the previous iteration.
[0134] Reference Figure 2 , the overall process of the data-free distillation method (Multi-teacher contrastiveKnowledgeinversion, MTCKI) in step 6 and step 7 is:
[0135] Input: pre-trained teacher model, subset size M
[0136] Randomly select a teacher model
[0137] Initialization: G(·;θ g ), z←(0,1), student model f s (·;θ s ), image library.
[0138] For e=1: maximum epoch
[0139] For i=1: maximum number of iterations
[0140] For t=1: maximum number of steps
[0141] Generate a batch of samples G(z) from the noise z.
[0142] Use the equation Calculate the loss LG.
[0143] Update the parameters of G
[0144] Store the sample G(z) into the image library.
[0145] Finish
[0146] For k = 1: Maximum number of steps
[0147] Extract a batch of images G(z) from the image library.
[0148] Use the equation Calculate the loss Ls.
[0149] Update the parameters of S
[0150] Finish
[0151] Finish
[0152] Output: Student model f S (·;θ s ) and image libraries.
[0153] The compressed student model and the synthesized data are tested using the test set data, and the test indicators are classification accuracy and FID.
[0154] The purpose of the present invention is to propose a new data-free distillation method to address the deficiencies of the prior art. The model compression method is based on multi-teacher contrastive learning. During the implementation process, a student can access multiple teachers, and multiple teacher networks provide comprehensive guidance, which is conducive to training a student model that is robust to model deviations. First, the traditional model inversion is extended to model inversion based on multi-teacher integration, and richer information from teachers is fully inverted to generate generalizable data. In addition, a contrastive interaction regularization between multiple teachers and students is proposed, which includes intra-teacher contrast and teacher-student contrast to improve the diversity of synthetic data. Specifically, intra-teacher contrast is used to gradually synthesize new samples with different patterns from historical samples, while teacher-student contrast aims to push the relationship between students and isomorphic teachers away from non-isomorphic relationships in the representation space. The process of image generation and knowledge transfer is trained in an adversarial manner to simultaneously learn student models and generate synthetic data.
[0155] The present invention does not rely on the original training data of the model. Through the knowledge distillation of multi-teacher models and the introduction of a loss function based on contrastive learning, the data-free model compression method is effectively compressed and the compressed model has a higher accuracy, and the inverted images are diverse and generalizable.
[0156] Compared with the prior art, the compressed model of the present invention has higher accuracy and is applicable to student networks with different network structures. It can generate a variety of student networks at one time and reduce the computational overhead and time of generating different data sets multiple times. This advantage comes from the diversity and generalization of the images synthesized by the generator. Since the images synthesized by the generator are compared with historical images, the difference between the images and the previously generated images is as large as possible, resulting in richer styles of images. Due to the regularization of multiple teachers and the distance between the image distribution of homologous students and their teacher networks, the disadvantage that the generated data is only applicable to a specific network is eliminated, making the images more generalized and closer to the real data set distribution, thereby improving network performance.
[0157] The following embodiments are tested on three public data sets, namely CIFAR10, CIFAR100, and Caltech-101. The model is compressed to obtain different student networks. The test results are shown in Table 1 below:
[0158] Table 1 Comparison of performance improvements on the dataset
[0159]
[0160] The evaluation index of image synthesis is usually measured by FID. The lower the value, the closer the synthesized image is to the real image, and the better the generation quality. The present invention compresses the model through data-free knowledge distillation. The accuracy of the model compressed by this method is the highest, and the FID value of the synthesized image is lower, which means that the synthesized image is closer to the original data set.
[0161] Summarizing the above comparison results on CIFAR-10, CIFAR-100 and Caltech-101. It can be observed that: (1) The method of the present invention outperforms the existing methods on all three datasets. For example, when extracting the same WRN-16-1 on CIFAR-10, the method of the present invention achieves 91.59%. (2) When the multi-teacher ensemble of the present invention and ResNet-34 on CIFAR-10 have almost the same accuracy (~95.7%), the accuracy of the method of the present invention when compressed to the same small network is significantly improved over other baselines. This also shows that the performance improvement comes from the multi-teacher structure and the proposed contrastive interaction loss, rather than the improvement of the teacher's prediction. (3) The isomorphic structure between the teacher and the student helps to further improve the performance of the student in all baselines. For example, compared with ResNet-34, the present invention uses the same WRN-16-1 as the student and WRN-40-2 as the teacher to significantly improve the accuracy of WRN-16-1 on CIFAR-10. (4) The pre-trained teacher of the present invention does not use MobileNet-V2; the synthetic images of the present invention can still effectively train the model. The method of the present invention achieves a 3.22% accuracy improvement over MobileNet-V2 trained using the original data. This means that the synthetic images using the proposed MTCKI have high generalization for the training of various models. (5) Compared with other methods, the FID values of the datasets generated by the present invention using different student models are all the lowest, and the variance is much smaller. This means that the synthetic images of the present invention are closer to the original dataset.
[0162] See attached Figure 3, the present invention can better synthesize the details of the image and has superiority in visual effects. The synthesis quality of the method is compared with that of the SOTA method. Obviously, the image quality of the MTCKI inversion of the present invention is the highest. For example, DAFL generates noise-like images using a pre-trained teacher on CIFAR-10. Deepinv is able to generate satisfactory visual images, but the object color is close to the background color and the style is single. Therefore, it is far from the original CIFAR-10 dataset. The comparison between the synthetic images of DFQ and MTCKI shows that MTCKI can generate more diverse images, while DFQ suffers from more serious mode collapse. Although the images sampled by CMI seem to have some improvements in color and style, they are still too blurry to be distinguished. The method of the present invention improves image quality in terms of the clarity of object contours, the rationality of color matching, and rich and detailed information. For CIFAR-10, MTCKI generates more diverse semantic images, such as close-ups of horses in different poses and various types of trucks. Even tiny details like the skyline behind the ship can be synthesized. For CIFAR-100, the synthetic images provide rich semantic information, and the present invention can easily identify the objects shown in the images, such as pandas, bicycles, and flowers.
[0163] See attached Figure 4 The present invention draws a t-sne graph of the characteristics of the synthetic data set through VGG16. It can be seen that the clustering characteristics of the same category of the data synthesized by the present invention are obvious, which is similar to the distribution of the original data.
[0164] See attached Figure 5 , we further analyze the convergence of our method and the changes in synthetic images. Compared with other baselines, our method requires fewer training epochs to converge and also achieves the lowest loss. It is worth noting that during the training process, due to the effectiveness of rich multi-teacher information and contrastive interactions, such as Figure 6 As shown in Figure 2, the images synthesized in the 10th epoch already have diverse semantic information and well-organized object contours.
[0165] The method of the present invention extracts model-specific knowledge from available teacher models and fuses it into the student model to eliminate model bias. In addition, contrastive interactions between multiple teachers and students are used to improve the diversity of synthesized images, which encourages synthesized images to be distinguished from previously stored images. Compared with the prior art, the present invention has the advantages of being able to produce more diverse and generalized pictures during the image generation process, and only needs to be generated once to provide comprehensive guidance for various networks rather than specific networks. Extensive experiments show that the method not only generates visually pleasing images, but also outperforms the existing state-of-the-art methods.
[0166] The method of the present invention is a new framework for data-free distillation, which extracts "multi-perspective" knowledge from multiple available teacher models and integrates it into a well-performing student model. The above method designs a comparative interaction to make full use of the knowledge from multiple teachers and students to generate synthetic data with high generalization and diversity. A large number of experiments show that the above method (MTCKI) is superior to the existing state-of-the-art methods. Not only is high-fidelity data closer to the distribution of the original data set synthesized, but also results comparable to the pre-trained model trained on the original data set are produced. Compared with the prior art, the compressed model of the present invention has a higher accuracy rate and is suitable for student networks with different network structures. It can generate a variety of different student networks at one time, and the computational overhead and time of generating different data sets multiple times. This advantage comes from the diversity and generalization of the pictures synthesized by the generator. Since the pictures synthesized by the generator are compared with the historical pictures, the difference between the pictures and the previously generated pictures is as large as possible, and pictures with richer styles are generated. Due to the regularization of multiple teachers and the distance between the picture distribution of the homologous students and their teacher networks, the disadvantage that the generated data is only applicable to a specific network is eliminated, making the pictures more generalized and closer to the real data set distribution, so that the network performance is improved.
[0167] The present invention also provides a model compression system based on a data protection scenario, the system comprising:
[0168] The preset module is used to preset multiple trained teacher models and randomly initialized student models and generators under the same dataset;
[0169] Three loss function building modules, used to build one-hot prediction loss function, feature regularization loss function and adversarial distillation loss function respectively;
[0170] The combination module is used to combine the one-hot prediction loss function, the feature regularization loss function and the adversarial distillation loss function to obtain the model inversion loss function of the data-free distillation of the generator under multi-teacher conditions;
[0171] A multi-teacher ensemble distillation loss function building module is used to build a multi-teacher ensemble distillation loss function where the student model imitates the output of the teacher model;
[0172] A contrast loss function establishment module is used to establish the teacher's internal contrast loss function and the teacher-student contrast loss function respectively;
[0173] The optimization module is used to optimize the generator using the model inversion loss function of data-free distillation, the teacher internal contrast loss function, and the teacher-student contrast loss function, output the image synthesized by the generator after this optimization, and input the image synthesized by the generator after this optimization into the student model and each trained teacher model respectively, and optimize the student model through the multi-teacher integrated distillation loss function;
[0174] The loop module is used to repeatedly perform optimization operations until the student model converges to obtain a compressed student model.
[0175] Optimization modules include:
[0176] The optimized loss function determination submodule is used to determine the optimized loss function of the generator according to the model inversion loss function, the teacher internal contrast loss function and the teacher-student contrast loss function of the data-free distillation model. In the formula, is the optimization loss of the generator, is the total function of the internal contrast loss function of the teacher and the contrast loss function of the teacher and students, λ is and The balance parameters between
[0177] Initialization submodule, used to initialize the parameters θ of the generator g , the parameters of the student model θ s and image libraries;
[0178] The synthesis submodule is used to synthesize the current batch of images using the generator according to the noise input z;
[0179] An optimization loss calculation submodule, used to calculate the current optimization loss of the generator using the optimization loss function according to the current batch of images;
[0180] The generator parameter update submodule is used to update the generator parameters θ g Updated to Among them, η is the coefficient, is the gradient operator;
[0181] A storage submodule, used to store the current batch of images synthesized by the generator into an image library;
[0182] The extraction submodule is used to extract a batch of images from the image library;
[0183] The student model current loss calculation submodule is used to calculate the current loss of the student model based on a batch of extracted images using a multi-teacher integrated distillation loss function;
[0184] The student model parameter update submodule is used to update the parameters θ of the student models Updated to
[0185] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0186] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of the present invention. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A model compression method based on a data protection scenario, characterized in that: The method comprises: Preset multiple trained teacher models and randomly initialized student models and generators under the same dataset; Construct one-hot prediction loss function, feature regularization loss function and adversarial distillation loss function respectively; Combining the one-hot prediction loss function, feature regularization loss function, and adversarial distillation loss function, we obtain the model inversion loss function for data-free distillation of the generator under multi-teacher conditions. Construct a student model to mimic the multi-teacher ensemble distillation loss function output by the teacher model; Establish the internal teacher contrast loss function and the teacher-student contrast loss function respectively; The generator is optimized using the model inversion loss function of data-free distillation, the teacher internal contrast loss function, and the teacher-student contrast loss function. The image synthesized by the generator after this optimization is output, and the image synthesized by the generator after this optimization is input into the student model and each trained teacher model respectively. The student model is optimized using the multi-teacher integrated distillation loss function. Repeat the optimization operation until the student model converges to obtain a compressed student model; The establishment process of the teacher's internal contrast loss function is: Performing data augmentation on each image in the same batch of images synthesized by the generator, and inputting each of the images and the data augmented images into each trained teacher model respectively, to obtain a representation of each of the images output by each trained teacher model and a representation of the data augmented images; Randomly select an image from the same batch of images synthesized by the generator as the image to be tested; The representation of the image to be tested and the representation of the image to be tested after data enhancement are taken as positive sample pairs, the representation of images other than the image to be tested in the same batch of images synthesized by the generator are taken as negative samples, and the representation of historical images synthesized by the generator are taken as negative samples; According to the positive sample pairs and negative samples, the teacher’s internal contrast loss function is determined as In the formula, is the intra-teacher contrast loss, is a positive sample pair, is the representation of the image to be tested output by the mth trained teacher model, is the output of the mth trained teacher model, is the parameter of the mth head mapping network, h is the head projection network, is the representation of the image to be tested after data augmentation output by the mth trained teacher model, is the i-th negative sample corresponding to the m-th trained teacher model, K is the number of negative samples, τ1 is the first temperature hyperparameter, sim() is the cosine similarity, and M is the number of integrated teacher models; The establishment process of the teacher-student comparison loss function is: Each image in the same batch of images synthesized by the generator is input into each trained teacher model and student model respectively, and the representation of each image by each trained teacher model and the representation of each image by the student model are obtained; The representation of the same image by the student model and the trained teacher model which is isomorphic to the student model is used as a negative sample pair; the isomorphism means that the student model and the teacher model belong to the same network structure series; The representation of the same image by a student model and a trained teacher model that is heterogeneous with the student model is defined as a positive sample pair; the heterogeneity is that the student model and the teacher model belong to different network structure series; According to the negative sample pairs and the defined positive sample pairs, the teacher-student comparison loss function is determined as: In the formula, To compare the losses between teachers and students, is the query of the student model constructed from the rth image in the current batch of images synthesized by the generator, Synthesized images for the generator The output of the student model, θ s are the parameters of the student model, θ h are the parameters of the head mapping network, is the query of the mth trained teacher model heterogeneous with the student model constructed from the rth image in the current batch of images synthesized by the generator, D(s) is the set of teacher indices heterogeneous with the student model, N is the number of images in the current batch of images, τ2 is the first temperature hyperparameter, Neg is the set of negative pairs, I(s) is the teacher index set isomorphic to the student model, is the jth negative sample in the representation of the historical image output by the student model, J is the number of negative samples in the representation of the historical image output by the student model, is the query of the student network constructed from the jth image in the historical images synthesized by the generator, is a query for a teacher model that is isomorphic to the student network model.
2. The model compression method based on the data protection scenario according to claim 1 is characterized in that: The one-hot prediction loss function is In the formula, is the one-hot prediction loss, CE is the cross entropy loss, Synthesized images for the generator The output after inputting into the trained teacher model, c is the predefined class; The feature regularization loss function is In the formula, is the feature regularization loss, Synthesized images for the generator The mean value obtained by the lth BN layer after inputting into the trained teacher model, Synthesized images for the generator The variance obtained by the lth BN layer after inputting into the trained teacher model, E(μ l (x)X) is the mean value obtained by the lth BN layer after the input image x is input into the trained teacher model, The variance obtained by the lth BN layer after the input image x is input into the trained teacher model; The adversarial distillation loss function is In the formula, To combat distillation loss, KL is the Kulbeck-Leible divergence, Synthesized images for the generator The output of the student model, θ s are the parameters of the student model and τ is the temperature.
3. The model compression method based on the data protection scenario according to claim 2 is characterized in that: The one-hot prediction loss function, feature regularization loss function and adversarial distillation loss function are combined to obtain the model inversion loss function of the data-free distillation of the generator under multi-teacher conditions, which specifically includes: Combining the one-hot prediction loss function, feature regularization loss function and adversarial distillation loss function, the model inversion loss function of the data-free distillation of the generator under the single teacher condition is obtained as In the formula, is the model inversion loss of the data-free distillation of the generator under the single teacher condition, λ1, λ2 and λ3 are the first, second and third balanced parameters respectively; According to the model inversion loss function of the data-free distillation of the generator under the single-teacher condition, the model inversion loss function of the data-free distillation of the generator under the multi-teacher condition is constructed as In the formula, Model inversion loss for data-free distillation of generators under multi-teacher conditions, For the one-hot prediction loss with multiple teacher information, is the feature regularization loss with multi-teacher information, is the adversarial distillation loss with multi-teacher information, z is the noise input, θ g is the parameter of the generator; in, is the output of the M number of integrated teacher models, is the output of the mth trained teacher model; Synthesized images for the generator The mean value obtained by the lth BN layer after inputting into the mth trained teacher model, Synthesized images for the generator The variance obtained by the lth BN layer after inputting into the mth trained teacher model, is the mean value obtained by the lth BN layer after the input image x is input to the mth trained teacher model, It is the variance obtained by the lth BN layer after the input image x is input to the mth trained teacher model.
4. The model compression method based on the data protection scenario according to claim 3 is characterized in that: The multi-teacher ensemble distillation loss function is In the formula, Ensemble distillation loss for multi-teachers.
5. The model compression method based on the data protection scenario according to claim 4 is characterized in that: The generator is optimized using the model inversion loss function, the teacher internal contrast loss function and the teacher-student contrast loss function without data distillation, and the image synthesized by the generator after this optimization is output, and the image synthesized by the generator after this optimization is input into the student model and each trained teacher model respectively, and the student model is optimized by the multi-teacher integrated distillation loss function, specifically including: According to the model inversion loss function without data distillation, the teacher internal contrast loss function and the teacher-student contrast loss function, the optimized loss function of the generator is determined as In the formula, is the optimization loss of the generator, is the total function of the internal contrast loss function of the teacher and the contrast loss function of the teacher and students, λ is and The balance parameters between Initialize the parameters of the generator θ g , the parameters of the student model θ s and image libraries; According to the noise input z, use the generator to synthesize the current batch of images; According to the current batch of images, using the optimization loss function to calculate the current optimization loss of the generator; The parameters of the generator θ g Updated to Among them, η is the coefficient, is the gradient operator; Store the current batch of images synthesized by the generator into the image library; Extract a batch of images from the image library; Based on the extracted batch of images, the current loss of the student model is calculated using the multi-teacher ensemble distillation loss function; The parameters θ of the student model are s Updated to 6. The model compression method based on data protection scenario according to claim 5 is characterized in that: The student model convergence means that the current loss of the student model is equal to the loss of the student model calculated in the previous iteration.
7. A model compression system based on a data protection scenario, characterized in that: The system comprises: The preset module is used to preset multiple trained teacher models and randomly initialized student models and generators under the same dataset; Three loss function building modules, used to build one-hot prediction loss function, feature regularization loss function and adversarial distillation loss function respectively; The combination module is used to combine the one-hot prediction loss function, the feature regularization loss function and the adversarial distillation loss function to obtain the model inversion loss function of the data-free distillation of the generator under multi-teacher conditions; A multi-teacher ensemble distillation loss function building module is used to build a multi-teacher ensemble distillation loss function where the student model imitates the output of the teacher model; A contrast loss function establishment module is used to establish the teacher's internal contrast loss function and the teacher-student contrast loss function respectively; The optimization module is used to optimize the generator using the model inversion loss function of data-free distillation, the teacher internal contrast loss function, and the teacher-student contrast loss function, output the image synthesized by the generator after this optimization, and input the image synthesized by the generator after this optimization into the student model and each trained teacher model respectively, and optimize the student model through the multi-teacher integrated distillation loss function; A loop module is used to repeatedly perform optimization operations until the student model converges to obtain a compressed student model; The establishment process of the teacher's internal contrast loss function is: Performing data augmentation on each image in the same batch of images synthesized by the generator, and inputting each of the images and the data augmented images into each trained teacher model respectively, to obtain a representation of each of the images output by each trained teacher model and a representation of the data augmented images; Randomly select an image from the same batch of images synthesized by the generator as the image to be tested; The representation of the image to be tested and the representation of the image to be tested after data enhancement are taken as positive sample pairs, the representation of images other than the image to be tested in the same batch of images synthesized by the generator are taken as negative samples, and the representation of historical images synthesized by the generator are taken as negative samples; According to the positive sample pairs and negative samples, the teacher’s internal contrast loss function is determined as In the formula, is the intra-teacher contrast loss, is a positive sample pair, is the representation of the image to be tested output by the mth trained teacher model, is the output of the mth trained teacher model, is the parameter of the mth head mapping network, h is the head projection network, is the representation of the image to be tested after data augmentation output by the mth trained teacher model, is the i-th negative sample corresponding to the m-th trained teacher model, K is the number of negative samples, τ1 is the first temperature hyperparameter, sim() is the cosine similarity, and M is the number of integrated teacher models; The establishment process of the teacher-student comparison loss function is: Each image in the same batch of images synthesized by the generator is input into each trained teacher model and student model respectively, and the representation of each image by each trained teacher model and the representation of each image by the student model are obtained; The representation of the same image by the student model and the trained teacher model which is isomorphic to the student model is used as a negative sample pair; the isomorphism means that the student model and the teacher model belong to the same network structure series; The representation of the same image by a student model and a trained teacher model that is heterogeneous with the student model is defined as a positive sample pair; the heterogeneity means that the student model and the teacher model belong to different network structure series; According to the negative sample pairs and the defined positive sample pairs, the teacher-student comparison loss function is determined as: In the formula, To compare the losses between teachers and students, is the query of the student model constructed from the rth image in the current batch of images synthesized by the generator, Synthesized images for the generator The output of the student model, θ s are the parameters of the student model, θ h are the parameters of the head mapping network, is the query of the mth trained teacher model heterogeneous with the student model constructed from the rth image in the current batch of images synthesized by the generator, D(s) is the set of teacher indices heterogeneous with the student model, N is the number of images in the current batch of images, τ2 is the first temperature hyperparameter, Neg is the set of negative pairs, I(s) is the teacher index set isomorphic to the student model, is the jth negative sample in the representation of the historical image output by the student model, J is the number of negative samples in the representation of the historical image output by the student model, is the query of the student network constructed from the jth image in the historical images synthesized by the generator, is a query for a teacher model that is isomorphic to the student network model.
8. The model compression system based on data protection scenario according to claim 7 is characterized in that: The optimization module specifically includes: The optimized loss function determination submodule is used to determine the optimized loss function of the generator according to the model inversion loss function, the teacher internal contrast loss function and the teacher-student contrast loss function of the data-free distillation model. In the formula, is the optimization loss of the generator, is the total function of the internal contrast loss function of the teacher and the contrast loss function of the teacher and students, λ is and The balance parameters between Initialization submodule, used to initialize the parameters θ of the generator g , the parameters of the student model θ s and image libraries; The synthesis submodule is used to synthesize the current batch of images using the generator according to the noise input z; An optimization loss calculation submodule, used to calculate the current optimization loss of the generator using the optimization loss function according to the current batch of images; The generator parameter update submodule is used to update the generator parameters θ g Updated to Among them, η is the coefficient, is the gradient operator; A storage submodule, used to store the current batch of images synthesized by the generator into an image library; The extraction submodule is used to extract a batch of images from the image library; The student model current loss calculation submodule is used to calculate the current loss of the student model based on a batch of extracted images using a multi-teacher integrated distillation loss function; The student model parameter update submodule is used to update the parameters θ of the student model s Updated to
