A knowledge distillation method driven by perturbation generative adversarial networks

Abstract

The present invention discloses a knowledge distillation method driven by an interference generative adversarial network. The method first obtains an image classification dataset consisting of three-channel RGB format images. Secondly, an interference generative adversarial network AD‑GAN is constructed, including a teacher model, a student model and an interference generative model. The inputs of the three models are all images in the image classification dataset. The outputs of the teacher and student models are confidence vectors for image category predictions, and the output of the interference generative model is an image with interference. Finally, the three models are subjected to adversarial training. A training batch is divided into an interference generative model training phase and a student model distillation training phase. The training of the two uses independent optimizers respectively. The interference generative model is used as a generator in adversarial training, and the teacher and student models are used together as discriminators. The present invention enhances the generalization ability and noise resistance of the student model, and provides a practical and efficient solution for resource-constrained scenarios.

Description

Technical Field

[0001] The present invention relates to the field of knowledge distillation based on deep learning, and in particular to a knowledge distillation method driven by an interference generative adversarial network. Background Art

[0002] In the image classification task based on deep learning, although deep neural networks (DNN) perform well on training data, high-performance models have problems such as high complexity, large amount of computation, high latency, and strict deployment conditions. In order to achieve lightweight models, that is, to provide strong generalization performance while maintaining low computing resource consumption, researchers have proposed a variety of methods, among which the knowledge distillation (KD) method has attracted much attention for its ability to efficiently simplify the model while maintaining performance. In this method, the teacher model is usually a large and complex deep neural network that has high pattern recognition capabilities after being trained with large-scale data sets; the student model is lightweight and aims to learn the knowledge and generalization capabilities of the teacher model in order to balance high performance and high efficiency in practical applications, thereby improving the flexibility of model deployment, especially providing technical support for the needs of the edge computing field.

[0003] However, traditional methods mainly rely on guiding the probability distribution or feature distribution of the student model output to be as similar as possible to the teacher model on the original data set, so that the student model can show similar decision logic and feature extraction mechanism as the teacher model. Although this method can make the student model achieve relatively consistent performance with the teacher model in an ideal environment, it has obvious limitations in actual application scenarios. Faced with data with complex structure, uncertain features and irregular distribution, especially in the presence of severe noise interference, the student model is over-adapted to the characteristics of clean data and it is difficult to flexibly adjust its feature extraction strategy, resulting in its ability to capture key features. It lacks an effective noise filtering mechanism, and ultimately makes the student model perform far worse than the teacher model in complex environments. In addition, existing knowledge distillation frameworks usually assume that the predictions of the teacher model are correct, which is not necessarily true when processing complex data or noisy data. When the teacher model makes a misjudgment, these erroneous information will be directly passed to the student model, causing the learning direction of the student model to shift and affecting its final performance.

[0004] At the same time, the combination of Generative Adversarial Networks (GAN) and knowledge distillation provides new ideas for improving the method. For example, by generating adversarial samples, the student model can correctly classify the adversarial samples, thereby continuously optimizing its internal feature representation and decision-making process during the training process, so as to improve the generalization ability of the student model and enhance its performance in complex data environments. However, existing technologies currently focus on generating challenging adversarial samples, while ignoring the consistency of the information distribution of the generated adversarial samples. This approach causes the student model to over-adapt to the perturbed features in the adversarial samples during training, deviating from the learning of the core features of the original samples, thereby causing problems such as overfitting or feature loss, reducing the effectiveness and practicality of the model. Therefore, how to maintain the consistency of sample information in the process of generating adversarial samples has become a key issue in improving the generalization ability of the student model. Summary of the invention

[0005] In view of the shortcomings of existing knowledge distillation methods in improving the generalization ability and robustness of student models, the present invention proposes a knowledge distillation method driven by an Adversarial Disturbance GAN (hereinafter referred to as AD-GAN). The method constructs a trainable interference generation model as the generator of AD-GAN, which generates perturbation samples based on clean samples and one-dimensional random vectors. The perturbation intensity is controlled by hyperparameters to ensure that the samples are visually similar and expose the weaknesses of the student model. The teacher model and the student model jointly serve as the discriminator of AD-GAN. The teacher model ensures that the sample information distribution does not deviate from the clean data set, and the student model identifies the perturbation samples to expose its own weaknesses. Through adversarial training, the interference generation model adjusts the perturbation noise, reduces the recognition accuracy of the student model and maintains the consistency of the sample distribution. The present invention designs and adopts an alternating training strategy, so that the interference generation model and the student model are alternately trained in the batch and use independent optimizers and loss back propagation. In order to avoid excessive reliance on interference samples in the early stage of the student model, a progressive training strategy is adopted to gradually increase the perturbation intensity. For loss function design, the loss function of the interference generation model combines the cross entropy loss and Hinge loss of the teacher and student models to improve the stability of adversarial training. The loss function of the distillation learning part includes distillation loss and cross entropy loss to ensure the classification accuracy of the student model.

[0006] The specific steps of the method of the present invention include:

[0007] S1. Obtain an image classification dataset consisting of three-channel RGB format images.

[0008] S2. Construct an interference generative adversarial network AD-GAN, including a teacher model, a student model, and an interference generation model. The inputs of the three models are all images in the image classification dataset. The outputs of the teacher model and the student model are confidence vectors of image category predictions. The output of the interference generation model is an image with interference. The specific design and construction are as follows:

[0009] S21. Construct two independent neural network models as the teacher model and the student model, respectively. The number of parameters and accuracy of the teacher model are greater than those of the student model, while the inference speed and memory usage of the student model are smaller than those of the teacher model. For example, use ResNet101, WRN40-2, VGG13 and other models as the teacher model, and use ResNet20, WRN16-2, VGG8 and other models as the student model.

[0010] S22. Construct an interference generation model, which is used as a generator in AD-GAN to generate perturbation samples. The specific design of the interference generation model is as follows:

[0011] S221. The input of the interference generation model is the original clean sample image and a one-dimensional random vector ,in is the number of input channels, and are the length and width of the input image, respectively. Represents a one-dimensional random vector Dimensions of random vectors. go through The embedding layer is mapped to a One-dimensional vector of ,Right now .

[0012] S222. One-dimensional vector Folded into a single channel and have the same length and width as the input image ( ) ,Right now Then use Splicing operation, the input sample With random feature map Splice in the channel direction to get the spliced ​​feature map , output the concatenated feature map .

[0013] S223. The feature map output from step S132 Input to the feature extractor of the interference generation model. The feature extractor contains four feature extraction modules. The first three feature extraction modules (i=1,2,3) contains convolution transformations cascaded in sequence , normalization function , ReLU activation function, which is used to extract features from the input feature map. The formula is expressed as, the fourth feature extraction module only contains convolution transformation, and the input and output length and width of the four blocks are the same, which are , and the number of channels of each module input and output is different. The number of channels of the final output of the feature extractor is a hyperparameter The change process of the number of channels from input to output is:

[0014] Initial input layer: The number of channels of the input feature map is , because in the original An additional random channel is added to the first feature extraction block: This module takes the input Channels converted to This conversion is achieved through convolution operations. The second and third feature extraction blocks: Same as the first feature extraction block, the number of channels is changed from Convert to , and then , which is also achieved through convolution operation. The last module will Channels converted to channels to produce the final feature map. Among them, Indicates The number of channels of the feature map output by the feature extraction module, .

[0015] Feature Map After inputting into the feature extractor, the output of the feature extractor is obtained ;

[0016] S224. Output of the feature extractor Input to In the activation function, the generated disturbance information is obtained .

[0017] S225. The disturbance information generated in step S224 Interference intensity coefficient Multiply and get interference :

[0018]

[0019] Among them, interference The value range is Interference intensity coefficient ,Right now is a random variable that follows a uniform distribution, and each sample Independent of each other. For this round The maximum value of Dynamic updates are performed during the training process, that is, as the training rounds proceed, Gradually increases, with an upper limit of The formula is:

[0020]

[0021] in, is the current training round; For the warm-up period, the number of rounds with progressively increasing perturbation intensity is defined; is the maximum perturbation intensity hyperparameter.

[0022] S226. The generated disturbance With the original sample Add and use clipping operation To ensure that the element values ​​of the perturbation sample are In the range, the disturbance samples are obtained .

[0023] S3. Perform adversarial training optimization on the three models in AD-GAN. In a training batch, it is divided into two stages: the interference generation model training stage and the student model distillation training stage. The training of the two uses independent optimizers, that is, the training parameter updates of the interference generation model and the student model do not interfere with each other. The interference generation model is used as the generator in the adversarial training, while the teacher model and the student model are used as discriminators. The specific steps of AD-GAN adversarial training optimization include:

[0024] S31. In the interference generation model adversarial training stage, the parameters of the teacher model and the student model are fixed and both are set to inference mode, and the interference generation model is set to training mode. Interference , generate interference samples , will interfere with the sample Input into the teacher model and the student model to guide the interference generation model to adjust the interference. The specific steps are as follows:

[0025] S311. Generation and batch size The same number of one-dimensional random vectors , and with all clean samples Input them into the interference generation model together to obtain interference samples .

[0026] S312. Interference samples Input them into the teacher model and student model with fixed parameters respectively to obtain the category prediction probability distribution and Then, we calculate the probability distribution and the true label The cross entropy loss of the teacher model is obtained and the cross entropy loss of the student model .

[0027] S313. Combining Hinge Loss and Cross Entropy Loss of Teacher Model To ensure the stability of generator training, the optimized teacher model loss is obtained , the formula is:

[0028]

[0029] in, As the tolerance threshold, a certain increase in the loss of the teacher model on the perturbation samples is allowed to improve the stability of adversarial training. For hinge The weight of the loss controls the penalty for increasing the loss after exceeding the threshold.

[0030] S314. Total loss during the training phase of the interference generation model Defined as:

[0031]

[0032] The total loss during the training phase of the perturbation generation model Perform back propagation to update the parameters of the interference generation model.

[0033] S32. During the student model distillation training phase, the parameters of the teacher model and the interference generation model are fixed and both are set to inference mode, and the student model is set to training mode. The clean sample is interfered with to obtain the interfered sample, and then mixed with the remaining clean sample to obtain the mixed sample Mix the sample Participate in the distillation training of the teacher model and the student model. The specific steps are as follows:

[0034] S321. In a batch, first clean samples Select The number of samples, where is a hyperparameter that determines the percentage of the number of interference samples in the batch. At the same time, a one-dimensional random vector with the same number of interference samples is generated. ,and The number of samples are input into the interference generation model together to obtain interference samples , With sample The remaining clean samples Mix to get a mixed sample :

[0035]

[0036] S322. Mix the samples Input the teacher model and student model respectively to get the predicted category output of the teacher model and student model And feature output .

[0037] S323. Output the features of the student model The feature output of the teacher model Alignment, calculation of distillation loss , the formula is:

[0038]

[0039] in, Represents the distillation method. Different distillation methods can be selected according to needs, such as the Logits-based distillation method, the intermediate feature-based distillation method, or the attention-based distillation method.

[0040] S324. Prediction category output using student model With the true label Calculate the cross entropy loss to get the classification loss of the student model .

[0041] S325. Total loss of the student model during training Distillation loss With classification loss The weighted sum of is expressed as:

[0042]

[0043] in, To control distillation losses With classification loss The relative importance of the hyperparameter. The total loss during the training phase of the student model is used Perform back propagation to update the parameters of the student model.

[0044] Beneficial effects of the present invention

[0045] The knowledge distillation method based on interference generative adversarial network (AD-GAN) achieves efficient information transfer and improvement of student model capabilities by constructing a trainable interference generative model as a generator and combining teacher and student models as discriminators. The method first applies a perturbation of controllable intensity to the clean sample to generate perturbation samples that both maintain the original information distribution and expose the weaknesses of the student model; then, the perturbation noise is adjusted through adversarial training so that the student model learns to resist these perturbations, thereby improving its feature extraction ability and decision logic. The present invention has the following benefits: 1. Introducing the adversarial training mechanism and ensuring that the information distribution of the perturbation sample is close to the original data set, strengthening the learning quality of the student model and improving its anti-interference ability; 2. Using a progressive training strategy to gradually increase the perturbation intensity, avoiding the difficulty of the student model to establish basic feature extraction capabilities due to excessive interference in the early stage of training, thereby ensuring the stability and effectiveness of the training process; 3. Designing a stable loss function, integrating hinge loss and cross entropy loss enables the interference generative model to maintain the stability of training while revealing the weaknesses of the student model, preventing mode collapse; 4. The alternating training strategy ensures that the two models are optimized independently, further improving the training efficiency and the quality of the final model. In summary, the present invention enhances the generalization ability and noise resistance of the student model through an innovative AD-GAN framework and an efficient training mechanism, provides a practical and efficient solution for resource-constrained scenarios, and shows certain application prospects and use value. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 An architecture diagram of a knowledge distillation method driven by a perturbation generative adversarial network;

[0047] Figure 2 Interference generation model structure diagram;

[0048] Figure 3 Process diagram of the interference generation model training phase;

[0049] Figure 4 Process diagram of the student model training phase. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] As a preferred implementation form of the present invention, a knowledge distillation method driven by an interference generation adversarial network is provided, and its architecture is as follows: Figure 1 As shown, the following steps are included:

[0052] S1. The knowledge distillation method of the present invention is applied to an image classification dataset consisting of three-channel RGB format images.

[0053] S2. Construct an interference generative adversarial network AD-GAN, including a teacher model, a student model, and an interference generation model. The inputs of the three models are all images in the image classification dataset. The outputs of the teacher model and the student model are confidence vectors of image category predictions. The output of the interference generation model is an image with interference. The specific design and construction are as follows:

[0054] S21. Construct two independent neural network models as the teacher model and the student model, respectively. The number of parameters and accuracy of the teacher model are greater than those of the student model, while the inference speed and memory usage of the student model are less than those of the teacher model. This implementation uses ResNet32x4, WRN40-2, and VGG13 models as teacher models, and uses ResNet20, ResNet8x4, ResNet32, WRN16-2, WRN-40-1, VGG8, and SHN-V2 models as student models.

[0055] S22. Construct interference generation model, such as Figure 2 As shown in Figure 1, the model is used as a generator in AD-GAN to generate perturbation samples that are as "real" as possible, that is, they do not deviate from the original data set in terms of information distribution, but can expose the weaknesses of the student model and are used to optimize the performance of the student model against perturbation samples, thereby improving its generalization. The specific design of the interference generation model is as follows:

[0056] S221. The input of the interference generation model is the original clean sample image and a one-dimensional random vector , where the input image has 3 channels, the length and width are both 32, and the one-dimensional random vector The dimension of is 128. The random vector go through The embedding layer is mapped to a One-dimensional vector of ,Right now .

[0057] S222. One-dimensional vector Folded into a single channel, random feature map with length and width of 32 ,Right now Then use Splicing operation, the input sample With random feature map Splice in the channel direction to get the spliced ​​feature map , the specific formula is as follows:

[0058]

[0059] Output the concatenated feature map .

[0060] S223. The feature map output from step S132 Input to the feature extractor of the interference generation model. The feature extractor contains four feature extraction modules. The first three feature extraction modules (i=1,2,3) contains convolution transformation, normalization function, and ReLU activation function, which are used to extract features from the input feature map. The formula is:

[0061]

[0062] The fourth feature extraction module only contains convolution transformation, and the formula is expressed as:

[0063]

[0064] Among them, the input and output length and width of the four blocks are the same, all of which are , and the number of channels of each module input and output is different. The number of channels of the final output of the feature extractor is a hyperparameter The change process of the number of channels from input to output is:

[0065] Initial input layer: The number of channels of the input feature map is , because an extra random channel is added to the original three channels.

[0066] The first feature extraction block: This module takes the input Channels converted to This transformation is achieved through a convolution operation.

[0067] The second and third feature extraction blocks are the same as the first feature extraction block, but the number of channels is changed from Convert to , and then , which is also achieved through convolution operation.

[0068] Output Module: The last module will Channels converted to channels to produce the final feature map.

[0069] Feature Map After inputting into the feature extractor, the output of the feature extractor is obtained , the formula is:

[0070]

[0071] S224. Output of the feature extractor Input to In the activation function, the generated disturbance information is obtained The formula is:

[0072]

[0073] use The activation function can ensure the output of disturbance information The element values ​​of are restricted to [-1,1] and the size is , ensuring the stability of the output of the interference generation model.

[0074] S225. The disturbance information generated in step S224 Interference intensity coefficient Multiply and get interference :

[0075]

[0076] Among them, interference The value range is Interference intensity coefficient ,Right now is a random variable that follows a uniform distribution, and each sample Independent of each other. For this round The maximum value of Dynamic updates are performed during the training process, that is, as the training rounds proceed, Gradually increases, with an upper limit of The formula is:

[0077]

[0078] in, is the current training round; the warm-up period is 80 training rounds, and the disturbance intensity increases gradually during the warm-up period; is the maximum disturbance intensity.

[0079] S226. The generated disturbance With the original sample Add and use clipping operation To ensure that the element values ​​of the perturbation sample are In the range, the disturbance samples are obtained The formula is as follows:

[0080]

[0081] in, and are the minimum and maximum values ​​of the input sample elements respectively.

[0082] S3. Perform adversarial training optimization on the three models built in AD-GAN. In a training batch, it is divided into two stages: the interference generation model training stage and the student model distillation training stage. The training of the two uses independent optimizers, that is, the training parameter updates of the interference generation model and the student model do not interfere with each other. The interference generation model is used as the generator in the adversarial training, while the teacher model and the student model are used as discriminators. The specific steps of AD-GAN adversarial training optimization include:

[0083] S31. In the interference generation model adversarial training stage, the parameters of the teacher model and the student model are fixed and both are set to inference mode, and the interference generation model is set to training mode. Figure 3 As shown, the interference generation model is effective for clean samples Interference , generate interference samples , will interfere with the sample Input into the teacher model and the student model to guide the interference generation model to adjust the interference and ensure the perturbation sample It does not deviate from the original sample information distribution, but can also expose the weaknesses of the student model. The specific steps are as follows:

[0084] S311. Generate 64 one-dimensional random vectors , and with all clean samples Input them into the interference generation model together to obtain interference samples .

[0085] S312. Interference samples Input them into the teacher model and student model with fixed parameters respectively to obtain the category prediction probability distribution and Then, we calculate the probability distribution and the true label The cross entropy loss of the teacher model is obtained and the cross entropy loss of the student model The cross entropy loss formula is expressed as:

[0086]

[0087]

[0088] Among them, the batch size is 64, the number of output categories is 100, It is a sample The true label is in the category The indicator function on The true category is ,but ,otherwise ); The model is a sample Belongs to category The predicted probability of .

[0089] S313. Combining Hinge Loss and Cross Entropy Loss of Teacher Model To ensure the stability of generator training, the optimized teacher model loss is obtained , the formula is:

[0090]

[0091] Among them, the tolerance threshold is 0.2, which means that the loss of the teacher model on the perturbation sample is allowed to be within 0.2, improving the stability of adversarial training. The weight of the loss is 15, which controls the penalty for increasing the loss after exceeding the threshold range.

[0092] S314. In order to ensure that the perturbed samples do not deviate from the information distribution of the original samples, the hinge loss At the same time, in order to allow the perturbation information to expose the weaknesses of the student model, the cross entropy loss of the student model Need to be increased. Therefore, the total loss of the interference generation model training phase Defined as:

[0093]

[0094] The total loss during the training phase of the perturbation generation model Perform back propagation to update the parameters of the interference generation model.

[0095] S32. During the student model distillation training phase, the parameters of the teacher model and the interference generation model are fixed and both are set to inference mode, and the student model is set to training mode. Figure 4 As shown in Figure 2, the interference generation model is The clean sample is interfered to obtain the interfered sample, and then mixed with the remaining clean sample to obtain the mixed sample Mix the sample Participate in the distillation training of the teacher model and the student model. The specific steps are as follows:

[0096] S321. In a batch, first clean samples Select of samples, where is the percentage of interference samples to the number of batch samples. At the same time, generate a one-dimensional random vector with the same number of interference samples ,and The number of samples are input into the interference generation model together to obtain interference samples , With sample The remaining clean samples Mix to get a mixed sample :

[0097]

[0098] S322. Mix the samples Input the teacher model and student model respectively to get the predicted category output of the teacher model and student model and logits output .

[0099] S323. Output the logits of the student model Logits output from the teacher model Alignment, calculation of distillation loss , the formula is:

[0100]

[0101] in, Represents the Kullback-Leibler divergence calculation function, which is used to measure the difference between two probability distributions. It is a hyperparameter used to adjust the smoothness of the probability distribution. An appropriate smoothness can retain more relative relationship information and improve learning efficiency.

[0102] S324. Prediction category output using student model With the true label Calculate the cross entropy loss to get the classification loss of the student model .

[0103] S325. Total loss of the student model during training Distillation loss With classification loss The weighted sum of is expressed as:

[0104]

[0105] in, Represents distillation loss With classification loss The total loss during the training phase of the student model is used Perform back propagation to update the parameters of the student model.

[0106] Example:

[0107] The steps of this embodiment are the same as those of the specific implementation method, and will not be repeated here. The implementation process and implementation results are shown below.

[0108] The technology of the present invention is implemented in the dataset CIFAR100 (Canadian Institute for Advanced Research 100 Class Dataset), which contains 50,000 training samples and 10,000 test samples, covering 100 categories, and the size of each sample is 32×32 pixels. As a standard image classification dataset, CIFAR100 is highly challenging and widely used to evaluate the performance of image classification models. For the model training process, the total training rounds are set to 240 rounds, and the batch size of each round is 64. The student model uses the stochastic gradient descent SGD optimizer, and the weight decay is , the momentum is set to 0.9, the initial learning rate is 0.05, and the learning rate is reduced to one tenth at the 150th, 180th, and 210th rounds. The interference generation model uses the adaptive moment estimation Adam optimizer, the learning rate is set to 0.001, and the weight decay is .

[0109] In order to evaluate the effect of interference generative adversarial network (AD-GAN) in different knowledge distillation frameworks KD: a knowledge distillation method based on neural network and DKD: a knowledge distillation method based on target decoupling, the experimental setting includes two main schemes: homogeneous network comparison experiment and heterogeneous network comparison experiment. In homogeneous networks, the teacher model and the student model use the same network architecture combination, while in heterogeneous networks, the network structure of the teacher model and the student model is different. The network architecture includes: ResNet: residual network, WRN: wide residual network, VGG: visual geometry perception neural network, ShuffleNet: lightweight neural network based on channel shuffle. At the same time, it is compared with other classic knowledge distillation methods, such as FitNet: a knowledge distillation method for narrow and deep neural networks, AT: a knowledge distillation method based on attention transfer, RKD: a knowledge distillation method based on relationship construction, CRD: a knowledge distillation method based on representation information comparison, and ReviewKD: a knowledge distillation method based on knowledge review.

[0110] The experimental results are shown in Tables 1 and 2, where the teacher model and the student model use the same model architecture combination in Table 1 and different model architecture combinations in Table 2. The experimental indicator uses the Top-1 accuracy (%), which refers to the ratio of the category with the highest prediction probability in the model prediction results to the true label. The designed interference generation adversarial training strategy is applied to the existing KD and DKD methods respectively. The KD+present invention and DKD+present invention in the table represent the improved results after the introduction of the present invention, and the performance gain is shown by ∆. Indicates the maximum perturbation intensity. Experiments were conducted at perturbation intensities of 0.5 and 0.8, respectively. The optimal and suboptimal results are marked in bold and underlined, respectively.

[0111] The experimental results show that after the AD-GAN strategy of the present invention is introduced into the KD method and the DKD method, the accuracy of the student model on the test set is improved in both the homogeneous network combination and the heterogeneous network combination comparison test. Compared with other distillation methods, it is found that the present invention shows a more outstanding effect in improving the performance of the student model, which proves that the present invention provides an efficient improvement solution for knowledge distillation technology.

[0112] Table 1. Comparative experiments on isomorphic model combinations under different knowledge distillation methods

[0113]

[0114] Table 2. Comparative experiments of heterogeneous model combinations under different knowledge distillation methods

[0115]

Claims

1. A knowledge distillation method driven by interference generative adversarial network, characterized in that The following steps are involved: S1. Obtain an image classification dataset consisting of three-channel RGB format images; S2. Construct an interference generative adversarial network AD-GAN, including a teacher model, a student model and an interference generation model. The inputs of the three models are all images in the image classification dataset. The outputs of the teacher model and the student model are confidence vectors of image category predictions. The output of the interference generation model is an image with interference. The specific implementation process of the interference generation model is as follows: The input of the interference generation model is the original sample image and a one-dimensional random vector z∈R 1 ×latent_dim , where c in is the number of input channels, h and w are the length and width of the input image respectively, latent_dim represents the dimension of the one-dimensional random vector z; the random vector z is mapped to a one-dimensional vector z of dimension h*w through the embedding layer. emb ; The one-dimensional vector z emb Folded into a single channel, a random feature map r with the same length and width as the input image, that is, r∈R 1×h×w ; Then use the concat operation to concatenate the input sample x clean It is concatenated with the random feature map r in the channel direction to obtain the concatenated feature map x concat ; The feature map x concat Input to the feature extractor of the interference generation model; The feature extractor contains four feature extraction modules. The first three feature extraction modules F i , i = 1, 2, 3, including the convolution transformation CONV, the normalization function BN, and the ReLU activation function cascaded in sequence to extract features from the input feature map; The fourth feature extraction module only contains convolutional transformations. The input and output dimensions of the four blocks are the same, h×w, but the number of input and output channels of each module is different; the number of channels output by the feature extractor is the hyperparameter c out ; Feature map x concat After inputting into the feature extractor, the output of the feature extractor is obtained F conv Input into the tanh activation function to get the generated disturbance information F perturb ; The disturbance information F perturb Interference intensity coefficient Multiply them together to get the interference δ: Among them, the value range of interference δ is ; Interference intensity coefficient ,Right now is a random variable that follows a uniform distribution, and each sample Independent of each other; For the current round The maximum value of Dynamic updates are performed during the training process, that is, as the training rounds proceed, Gradually increases, with an upper limit of ; The formula is: Among them, epoch is the current training round; warmup_epochs is the warm-up period, which defines the number of rounds with gradually increasing perturbation intensity; is the maximum perturbation intensity hyperparameter; The generated perturbation δ is compared with the original sample x clean Add and use the clipping operation clamp() to ensure that the element values ​​of the perturbed sample are within [x min ,x max ] range, get the perturbation sample x adv ; S3. The three models constructed in AD-GAN are optimized through adversarial training. One training batch is divided into: interference generation model training stage and student model distillation training stage. The training of the two uses independent optimizers, that is, the training parameter updates of the interference generation model and the student model do not interfere with each other. The interference generation model serves as the generator in adversarial training, and the teacher model and the student model serve as discriminators together.

2. The knowledge distillation method driven by interference generative adversarial network according to claim 1, characterized in that: The specific implementation process of step S2 is as follows: S21. Construct two independent neural network models as a teacher model and a student model, wherein the number of parameters and accuracy of the teacher model are greater than those of the student model, and the reasoning speed and memory usage of the student model are less than those of the teacher model; S22. Construct an interference generation model, which is used as a generator in AD-GAN to generate perturbation samples and optimize the performance of the student model against perturbation samples.

3. The knowledge distillation method driven by interference generative adversarial network according to claim 2, characterized in that: The specific implementation process of step S3 is as follows: S31. In the interference generation model adversarial training stage, the parameters of the teacher model and the student model are fixed and both are set to inference mode, and the interference generation model is set to training mode; the interference generation model is used for sample x clean Apply interference δ to generate interference samples x adv , will interfere with the sample x adv Input into the teacher model and the student model to guide the interference generation model to adjust the interference to ensure that the perturbation sample x adv It does not deviate from the original sample information distribution, but can also expose the weaknesses of the student model; S32. In the student model distillation training stage, the parameters of the teacher model and the interference generation model are fixed and both are set to inference mode, and the student model is set to training mode; the interference generation model interferes with the clean samples of the batch sample number α% to obtain interference samples, and mixes them with the remaining clean samples to obtain mixed samples x mix ; Mix the sample x mix Participate in the distillation training of teacher models and student models.

4. The knowledge distillation method driven by interference generative adversarial network according to claim 3 is characterized in that: The specific implementation process of step S31 is as follows: S311. Generate the same number of one-dimensional random vectors z∈R as the batch size b b×1×latent_dim and with all samples Input them into the interference generation model together to obtain interference samples ; S312. Interference sample x adv Input them into the teacher model and student model with fixed parameters respectively to obtain the category prediction probability distribution and Calculate the probability distribution and the true label y∈R b The cross entropy loss of the teacher model is obtained ce_tea and the cross entropy loss L of the student model ce_stu ; S313. Combining hinge loss and cross entropy loss of teacher model L ce_tea , and get the optimized teacher model loss L hinge_tea , the formula is: in, is the tolerance threshold, allowing the loss of the teacher model on the perturbation sample to increase to improve the stability of adversarial training; hinge is the weight of hinge loss; S314. Total loss L during the training phase of the interference generation model ad-gan Defined as: L ad-gan =L hinge_tea -L ce_stu The total loss L during the training phase of the perturbation generation model is used ad-gan Perform back propagation to update the parameters of the interference generation model.

5. The knowledge distillation method driven by interference generative adversarial network according to claim 4, characterized in that: The specific implementation process of step S32 is as follows: S321. In a batch, firstly, Select α% number of samples from , where α is a hyperparameter that determines the percentage of interference samples in the batch; Generate a one-dimensional random vector z∈R with the same number of interference samples (α％×b)×1×latent_dim , together with α% of samples, are input into the interference generation model to obtain the interference sample x adv_α , x adv_α With sample x clean The remaining clean samples x 1-α Mix and get mixed sample x mix : x mix ={x 1-α ,x adv_α } S322. Mix the sample x mix Input the teacher model and the student model respectively, and get the predicted category output p of the teacher model and the student model tea ,p stu And the feature output f tea ,f stu ; S323. Output the features of the student model f stu and the feature output f of the teacher model tea Align and calculate the distillation loss L KD , the formula is: L KD =Distill(f tea ,f stu ) Where Distill(·) represents the distillation method; S324. Use the predicted category output p of the student model stu With the true label y∈R b Calculate the cross entropy loss and get the classification loss L of the student model ce_stu ; S325. Total loss L during the student model training phase stu is the distillation loss L KD With the classification loss L ce_stu The weighted sum of is expressed as: L stu =L KD +β*L ce_stu Where β is the control distillation loss L KD With the classification loss L ce_stu Relative importance of hyperparameters; using the total loss L during the student model training phase stu Perform back propagation to update the parameters of the student model.

Images