A layered knowledge distillation method, system, device and storage medium for enhancing adversarial robustness of student models

Abstract

The application discloses a layered knowledge distillation method and system for enhancing the adversarial robustness of a student model, a device and a storage medium, and belongs to the field of artificial intelligence model optimization; the method comprises the following steps: constructing a layered knowledge guidance framework, extracting and transferring knowledge to the student model by using the layered knowledge guidance framework, starting a feature fusion learning mechanism, introducing an attention mechanism to assign weights to features for feature conversion, optimizing the feature learning of the student model, adaptively normalizing Logit, extracting Logit information from a teacher model and performing distribution standardization processing, calculating the loss by using KL divergence and optimizing the Logit learning of the student model, mixing the layered knowledge guidance framework, and outputting a student model generated after distillation; the application has higher robustness by using the decoupling method to aggregate two kinds of knowledge, and does not introduce additional adversarial disturbance data, so that the efficient recognition rate on clean samples is retained.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence model optimization, and particularly relates to a layered knowledge distillation method for enhancing the adversarial robustness of a student model, a system, a device and a storage medium. BACKGROUND

[0002] Computer vision applications are widespread, and DNN is an indispensable component. It plays a key role in various real-world tasks such as image classification, semantic segmentation, and object detection. However, completing these tasks often requires a large amount of computational power. Therefore, model distillation has emerged. Model distillation is a model compression technique aimed at transferring the knowledge of a large, complex model (teacher model) to a smaller, more efficient model (student model). Although knowledge distillation has made significant progress in improving model performance, studies have shown that even deep neural network (DNN) models obtained through model distillation can be vulnerable to adversarial perturbations. By adding tiny, imperceptible perturbations to input data, it can cause the model to output incorrect prediction results.

[0003] Therefore, when designing a model distillation method, the robustness of the student model generated by distillation is necessary to consider. Existing methods such as RSLAD (Robust Soft Label Adversarial Distillation) in defense distillation improve robustness by using robust soft labels, and MTARD (Multi-Teacher Adversarial Robust Distillation) improves model robustness by using a mixture of robust teacher models and normal teacher models for teaching. Both methods introduce additional training data and models, and have poor recognition efficiency on clean samples. Therefore, the existing technology mainly studies distillation methods that do not introduce additional adversarial data samples.

[0004] Current mainstream knowledge distillation methods include Logit distillation and Feature distillation. In the existing technology, the knowledge is first transferred through soft labels, and a temperature parameter is introduced to soften the output distribution of the teacher model, thereby capturing the similarity information between classes. Later, many methods based on Logit response have emerged. In terms of feature distillation, the FitNet distillation method is representative in proposing to use the feature map of the intermediate layer of the teacher as a "hint" to guide the student to learn the corresponding layer features. However, the current mainstream knowledge distillation methods all ignore the problem of the robustness of the student model, which is risky in the application of DNN models. SUMMARY

[0005] In view of the problems mentioned in the background art, the present application proposes a layered knowledge distillation method for enhancing the adversarial robustness of a student model, a system, a device and a storage medium, which teaches the student by decoupling the Logit knowledge and Feature knowledge of the teacher, achieving higher robustness; and without introducing additional adversarial data samples, maintaining high recognition rate on clean samples.

[0006] Technical scheme: In order to solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0007] A hierarchical knowledge distillation method for enhancing the adversarial robustness of a student model, comprising the following steps:

[0008] S1: Construct a hierarchical knowledge guidance framework, decouple semantic knowledge, and use the hierarchical knowledge guidance framework to extract and transfer knowledge to the student model;

[0009] S2: Start the feature fusion learning mechanism, introduce the attention mechanism to assign weights to the features and perform feature conversion, and based on the multi-scale mean square error loss, optimize the feature learning of the student model;

[0010] S21: Introduce the attention mechanism to assign feature weights, and convert important features of the teacher model into learnable feature knowledge for the student model;

[0011] S22: Fuse the initial features of the student model with the current features;

[0012] Measure the consistency between the student features and the teacher features through multi-scale mean square error loss;

[0013] S23: Calculate the feature loss and optimize the student model;

[0014] S3: Adaptive temperature standardization Logit, extract Logit information from the teacher model and perform distribution standardization processing, calculate the loss through KL divergence and optimize the Logit learning of the student model;

[0015] S31: Extract Logit information from the teacher model;

[0016] S32: Introduce a dynamic temperature parameter to dynamically adjust the temperature value according to the sharpness of the Logit;

[0017] S33: Perform distribution standardization processing on the Logit value to make the student model focus on learning the relative relationship between Logits;

[0018] S34: Calculate the probability distribution difference between the student model and the teacher model standardized Logit through KL divergence, and optimize the Logit learning of the student model;

[0019] S4: Mixed hierarchical knowledge guidance framework, fuse Feature knowledge and Logit knowledge, and optimize the mixed knowledge distillation loss;

[0020] S5: Output the final generated distilled student model.

[0021] As preferred, in S21, the specific content of introducing attention mechanism to assign feature weights is that:

[0022] Given an input image and a teacher network ; The output Logit of the teacher represents; Divided into different parts , The classifier represents, Different stages separated by down-sampling layers; therefore, the output of The output of

[0023] ,

[0024] The function symbol is used to represent the nesting of functions, The output is represented as:

[0025] ,

[0026] The intermediate feature level of the teacher represents The i-th layer feature of the teacher represents:

[0027] ,

[0028] The single-level feature knowledge transferred by the teacher model represents:

[0029] ,

[0030] Wherein, The i-th layer feature of the student represents, The i-th layer feature of the teacher represents; The important feature knowledge of the i-th layer of the student represents; The important feature knowledge of the i-th layer of the teacher represents; The distance function measures the difference between the feature of the student network and the feature of the teacher network; The first layer feature of the teacher represents; The n-th layer feature of the teacher represents; The n-th stage of the teacher network T represents; The i-th stage of the teacher network T represents.

[0031] As preferred, in S23, the specific content of calculating feature loss and optimizing the student model is that:

[0032] ​The first stage feature of the network includes low-level detailed information, and the first stage feature is denoted as a starting feature; by introducing low semantic information of the fused starting feature layer, the student network can better understand high-level semantic information of the teacher network; in each stage of knowledge transmission of the teacher model, the student feature is fused with the feature and the feature loss between the teacher feature is calculated;

[0033] The loss generated by the starting feature fusion learning mechanism in each stage is denoted as:

[0034] ,

[0035] The total distilled knowledge loss of the multi-level knowledge distillation is denoted as:

[0036] ,

[0037] The distance sum formula between the starting feature of the student and the current student feature and the teacher feature is denoted as the distance between the current student fusion feature and the teacher feature:

[0038] ,

[0039] The final feature loss is denoted as:

[0040] ,

[0041] wherein, represents important feature knowledge of the i-th layer of the student; represents important feature knowledge of the i-th layer of the teacher; represents the i-th layer feature of the student; represents the i-th layer feature of the teacher; represents a distance function for measuring the difference between the student network feature and the teacher network feature; n represents a sample capacity; represents a feature fusion module; represents the first layer feature of the student; represents important feature knowledge of the first layer of the student.

[0042] As a preferred, in S33, specifically:

[0043] The Logit value is standardized to zero mean and unit standard deviation, so that the student model focuses on learning the relative relationship between Logits, and the calculation formula of the distribution standardization function D1 is:

[0044]

[0045] Where K represents the total number of categories; Let x represent the k-th component vector, and let x represent the original Logit value. Let x represent the mean of the Logit vector. The standard deviation of the Logit vector x is represented by x. This indicates an adaptive temperature.

[0046] As a preferred option, in S34, the difference in probability distribution between the standardized Logit of the student model and the teacher model is calculated using KL divergence. The specific content for optimizing the Logit learning of the student model is as follows:

[0047] First, obtain the softmax outputs for students and teachers, specifically:

[0048] ,

[0049] ,

[0050] in, This represents the Logit value after processing with the student distribution standardization function D1; This represents the Logit value after processing by the teacher distribution standardization function D1; This represents the soft prediction for the i-th stage of the student network; This represents the soft prediction for the i-th stage of the teacher network; This represents the normalized exponential function;

[0051] Then, the probability distribution difference is calculated using KL divergence, specifically:

[0052] ,

[0053] Where KL represents the KL divergence; This represents the normalized exponential function; This represents the Logit value after processing with the student distribution standardization function D1; This represents the Logit value after processing by the standardized function D1, representing the teacher distribution c. Indicates the number of sample categories;

[0054] Finally, considering the effect of temperature on label smoothing, the final loss of Logit knowledge is set. for:

[0055] ,

[0056] in, Indicates the sample size, and KL represents the KL divergence; This indicates the temperature parameter of the student network; temperature parameter of the teacher network; P represents the soft prediction of the student network; Q represents the soft prediction of the teacher network.

[0057] As preferred, in S4, the specific implementation process is:

[0058] S41: Fuse Feature knowledge and Logit knowledge, and adjust the proportion of learning Feature knowledge and Logit knowledge of the student model by decoupling;

[0059] S42: Balance the feature distillation loss and the Logit distillation loss using hyperparameters;

[0060] S43: Optimize the mixed knowledge distillation loss.

[0061] As preferred, in S43, the specific content of optimizing the mixed knowledge distillation loss is:

[0062] ,

[0063] ,

[0064] wherein, represents the cross-entropy loss, N represents the number of samples; C represents the number of sample categories; represents the true label of whether the ith sample belongs to category c; represents the probability of the student network predicting the ith sample as category c; represents the mixed knowledge distillation loss; represents the final feature loss; represents the final Logit loss; , both represent hyperparameters; represents a dynamic factor.

[0065] A hierarchical knowledge distillation system for enhancing the adversarial robustness of a student model, which implements the hierarchical knowledge distillation method for enhancing the adversarial robustness of a student model described in any of the above, and the system comprises a hierarchical knowledge guidance module, a starting feature fusion learning module, a self-adaptive temperature standardization Logit module, a mixed knowledge fusion optimization module, and an output module.

[0066] The hierarchical knowledge guidance module is used to extract and deliver the Feature knowledge and Logit knowledge of the teacher model to the student model, and through hierarchical learning and decoupling, the student model learns the knowledge of the teacher model from different dimensions.

[0067] The starting feature fusion learning module optimizes the feature learning of the student model by introducing an attention mechanism to assign weights to features and convert them, and combining a multi-scale mean square error loss.

[0068] Adaptive temperature normalization Logit module: extract Logit information from the teacher model, introduce a dynamic temperature parameter to adjust the temperature value, and distribute the Logit value, and calculate the loss by KL divergence to optimize the Logit learning of the student model;

[0069] Hybrid knowledge fusion optimization module: fuse Feature knowledge and Logit knowledge, balance the corresponding distillation loss with hyperparameters, and optimize the hybrid knowledge distillation loss;

[0070] Output module: output the student model obtained by final distillation.

[0071] An electronic device includes a processor, a memory, a user interface, and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to make the electronic device execute the method described in any of the above.

[0072] A computer-readable storage medium stores a computer program capable of being loaded and executed by a processor to execute the method described in any of the above.

[0073] Advantages: Compared with the prior art, the present application has the following advantages:

[0074] (1) The knowledge distillation method of the present application creates a model trained only on clean samples, aiming to retain the high recognition rate of the model, while the model should have good robustness to resist potential adversarial perturbations.

[0075] (2) By decoupling the Logit knowledge and Feature knowledge of the teacher to teach the student, higher robustness can be achieved; and by not introducing additional adversarial data samples, the high recognition rate on clean samples is retained; and the Logit layer and the Feature layer have their unique focuses in knowledge representation, so by decoupling after hierarchical learning, the student model can learn the strong points of the teacher.

[0076] (3) By constructing two hierarchical knowledge extraction methods LMIFF to extract low-level semantic feature knowledge information, and ATNL to extract high-level semantic Logit knowledge information, the present application proves that the two kinds of knowledge have different focuses on adversarial perturbation pictures, and by extracting the knowledge focuses of the two and decoupling, the adversarial robustness is enhanced.

[0077] (4) How to extract useful information from the teacher's Logit knowledge level and Feature knowledge level and pass it to the student is a challenging problem. To solve these problems, the present application proposes a starting feature fusion learning mechanism in Feature knowledge distillation, an adaptive temperature standardization method in Logit knowledge distillation to learn useful knowledge, and finally aggregates the two kinds of knowledge through a decoupling method. The hierarchical knowledge guidance framework proposed by the present application enables the student to learn knowledge in multiple dimensions, which greatly improves the effectiveness of learning, and most importantly, it achieves higher robustness performance. BRIEF DESCRIPTION OF DRAWINGS

[0078] Figure 1 is a general flowchart of the present application;

[0079] Figure 2 is a schematic diagram of the Feature layer knowledge extraction module of the present application;

[0080] Figure 3 is a schematic diagram of the Logit layer knowledge extraction module of the present application;

[0081] Figure 4 is a t-SNE visualization comparison chart of the HIKD method and the Vanilla KD method of the present application;

[0082] Figure 5 is an effect display chart of five perturbation methods;

[0083] Figure 6 is a t-SNE visualization comparison chart of the HIKD method and the Vanilla KD method of the present application on the CIFAR-10 dataset after FGSM processing. DETAILED DESCRIPTION

[0084] The present application will be further illustrated below in combination with specific embodiments, which are implemented on the premise of the technical solutions of the present application, and it should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0085] As shown in Figure 1 , the hierarchical knowledge distillation method for enhancing the adversarial robustness of the student model provided by the present embodiment mainly includes the following steps:

[0086] S1: Construct a hierarchical knowledge guidance framework (Hierarchical Integration Knowledge Distillation, HIKD);

[0087] By decoupling low-level semantic knowledge and high-level semantic knowledge, the robustness of the student model is improved while retaining high recognition rate.

[0088] Specific operation: a framework is constructed to extract and pass the Feature knowledge and Logit knowledge of the teacher model to the student model respectively. Through hierarchical learning and decoupling, the student model learns the knowledge of the teacher model from different dimensions.

[0089] S2: Learning Mechanism of Initial Feature Fusion (LMIFF);

[0090] S21: introduce attention mechanism to assign feature weights, and convert important features of the teacher model into learnable feature knowledge of the student model;

[0091] In this embodiment, given an input picture and a teacher network ; taking the teacher network as an example, represents the output Logit of the teacher. It can be divided into different parts , wherein is a classifier, is different stages separated by down-sampling layers. Therefore, the output of is expressed as:

[0092]

[0093] represented by the function symbol as the nesting of functions, can be expressed as:

[0094]

[0095] wherein, is the output Logit of the teacher, and the intermediate feature level of the teacher is represented as , so the i-th layer feature of the teacher is represented as:

[0096]

[0097] Therefore, the single-level feature knowledge passed by the teacher model is expressed as:

[0098]

[0099] wherein, represents the i-th layer feature of the student, represents the i-th layer feature of the teacher; represents the important feature knowledge of the i-th layer of the student; represents the important feature knowledge of the i-th layer of the teacher; represents the first layer feature of the teacher; represents the n-th layer feature of the teacher; represents the n-th stage of the teacher network T; represents the i-th stage of the teacher network T. The attention mechanism is used to allocate feature weight fusion. This design is the main method of the B module. First, the input feature x is processed by the first convolutional layer Conv1. Then, the input feature x is spliced with the residual feature y which is upsampled to the same size by bilinear interpolation in the channel dimension, and the attention weight calculated by an attention convolutional layer and the learnable weight are used to dynamically adjust the contribution ratio of x and y.

[0100] represents the distance function measuring the difference between the student network feature and the teacher network feature.

[0101] The attention mechanism is used to allocate feature weight, and the important features of the teacher model are converted into learnable feature knowledge of the student model. As shown in Figure 2 represents the important features of the teacher are converted into important feature knowledge that the student can learn.

[0102] S22: Fuse the starting feature of the student model with the current feature.

[0103] The consistency between the student feature and the teacher feature is measured by the multi-scale mean square error loss (MSE).

[0104] S23: Calculate the feature loss and optimize the student model.

[0105] The first stage feature of the network mainly includes low-level detail information such as edges and textures. The first stage feature is the starting feature, which retains the most detailed information. By introducing the low semantic information of the starting feature layer, the student network can better understand the high-level semantic information of the teacher network. Low semantic information can be used as an auxiliary signal to help the student network better capture the feature representation of the teacher network. In each stage of the teacher model knowledge transfer, the student feature is fused with feature and the feature loss between the student feature and the teacher feature is calculated.

[0106] By optimizing the feature loss, the student model can better learn the low-level semantic information of the teacher model.

[0107] The loss generated by the starting feature fusion learning mechanism in each stage

[0108] , ​

[0109] All distilled knowledge losses of multi-level knowledge distillation is denoted as:

[0110] ,

[0111] The distance between the student's initial feature and the current student feature and the distance between the teacher feature are summed up respectively, denoted as the distance between the student fusion feature and the current teacher feature:

[0112] ,

[0113] Final feature loss is denoted as:

[0114] ,

[0115] wherein, denotes the important feature knowledge of the i-th layer of the student; denotes the important feature knowledge of the i-th layer of the teacher; denotes the i-th layer feature of the student; denotes the i-th layer feature of the teacher; denotes the distance function to measure the difference between the student network feature and the teacher network feature; n denotes the sample capacity; denotes the feature fusion module for fusion between two features; denotes the first layer feature of the student; denotes the important feature knowledge of the first layer of the student.

[0116] The feature fusion module first processes the input feature x through the first convolutional layer Conv1. Then the input feature x is spliced with the residual feature y which is up-sampled to the same size by bilinear interpolation in the channel dimension, and the contribution ratio of x and y is dynamically adjusted by the attention weight calculated through an attention convolutional layer and the learnable weight.

[0117] Low semantic information can be used as an auxiliary signal to help the student network better capture the learning signals contained in the current teacher network feature representation.

[0118] S3: Adaptive Temperature Normalization of Logit (ATNL);

[0119] S31: Extract Logit information from the teacher model.

[0120] These Logit information contains high-level semantic knowledge. Specifically as follows:

[0121]

[0122] in, It is the predicted probability of the i-th category. It is the logit value of the model corresponding to the i-th category. It is a temperature parameter used to control the smoothness of the probability distribution.

[0123] S32: Introduces dynamic temperature parameters.

[0124] The temperature value is dynamically adjusted based on the sharpness (dispersion) of the Logit. When the sharpness is high, the Logit value varies more, and the temperature parameter is adjusted accordingly.

[0125] The specific temperature setting is as follows:

[0126]

[0127] in, Indicates adaptive temperature; This represents the logit value of the model corresponding to the i-th category. This represents the logarithm of the sum of the exponents of all Logit values, used to quantify Logit sharpness. This represents the decay rate that varies with the number of training rounds. The sharpness of the labels is dynamically adjusted to help students better learn and adapt to the label information. It is a numerical stability constant to prevent numerical underflow / overflow.

[0128] S33: Standardize the distribution of Logit values.

[0129] like Figure 3 As shown, standardizing the Logit values ​​to zero mean and unit standard deviation allows the student model to focus on learning the relative relationships between Logit values. Specifically:

[0130] The formula for calculating the distribution standardization function D1 is:

[0131]

[0132] Where K represents the total number of categories; Indicates the first A subset vector, This represents the original Logit value. Represents the Logit vector The mean, Represents the Logit vector standard deviation This indicates the aforementioned adaptive temperature.

[0133] After such processing, the Logit vector of each sample will have a zero mean and a unit standard deviation. After processing by the standardization function D1, the logic value of the student model no longer needs to match the absolute size of the teacher model, but can be freely selected within the range suitable for its capacity.

[0134] S34: Calculate the loss by KL divergence and optimize the student model.

[0135] By optimizing the Logit loss, the student model learns better the advanced semantic information of the teacher model.

[0136] First, get the softmax output of the student and teacher:

[0137]

[0138]

[0139] where, represents the Logit value of the student distribution after processing by the standardization function D1; represents the Logit value of the teacher distribution after processing by the standardization function D1; represents the soft prediction of the i-th stage of the student network; represents the soft prediction of the i-th stage of the teacher network. represents the normalized exponential function.

[0140] Then calculate the probability distribution difference by KL divergence, specifically:

[0141]

[0142] where, represents the normalized exponential function; represents the Logit value of the student distribution after processing by the standardization function D1; represents the Logit value of the teacher distribution after processing by the standardization function D1; C represents the number of sample categories; KL represents the KL divergence.

[0143] The formula is simplified as:

[0144]

[0145] where, P represents the soft prediction of the student network; Q represents the soft prediction of the teacher network; represents the soft prediction of the i-th stage of the student network; represents the soft prediction of the i-th stage of the teacher network; C represents the number of sample categories.

[0146] Finally, considering the effect of temperature on label smoothing, the final Logit knowledge loss is set as: ​

[0147]

[0148] wherein, is the number of samples, KL is the Kullback-Leibler divergence; represents the temperature parameter of the student network, used to control the softening degree of the student output softmax; represents the temperature parameter of the teacher network, used to control the softening degree of the student output softmax; P represents the soft prediction of the student network; Q represents the soft prediction of the teacher network.

[0149] S4: Mixed Hierarchical Integration Framework;

[0150] S41: Fusion of Feature knowledge and Logit knowledge.

[0151] By decoupling, the proportion between learning Feature knowledge and Logit knowledge of the student model is adjusted.

[0152] S42: Use of hyperparameters to balance the loss of different knowledge.

[0153] Use hyperparameters (such as a and β) to balance the feature distillation loss and the Logit distillation loss, and use the dynamic factor λ to balance the feature distillation loss and the original loss.

[0154] S43: Optimization of mixed knowledge distillation loss.

[0155] The sensitivity of feature distillation and Logit distillation to noise and perturbation methods is different, and the two distillation methods have their own strong fields when verifying pictures. By fusing the different key knowledge learned by the two distillation methods, model distillation is realized only on clean samples to improve the robustness of the student model. By optimizing the mixed knowledge distillation loss, the student model obtains high recognition rate on clean samples while improving the adversarial robustness.

[0156] The specific calculation process is as follows:

[0157]

[0158]

[0159] wherein, represents the cross-entropy loss, N represents the number of samples; C represents the number of sample categories; represents the true label of whether the ith sample belongs to category c (0 or 1, usually using one-hot encoding). represents the probability that the student network predicts the i-th sample as class c (softmax output); represents the mixed knowledge distillation loss; represents the final feature loss; represents the final Logit loss; 、 represents the hyperparameter; represents the dynamic factor.

[0160] S5: Output the final generated distilled student model.

[0161] The present application uses accuracy and robustness as evaluation criteria:

[0162] Accuracy: Accuracy is a key indicator of model performance, used to evaluate the proportion of correct classification or prediction of the model on test data. Specifically, it quantifies the accuracy of the model by calculating the proportion of model output results consistent with the true label. The higher the accuracy, the higher the correctness of the model in processing data, and the more effective it can complete the task.

[0163] Robustness: Robustness is used to evaluate the stability of the model in the face of various disturbances and abnormal situations. It measures the performance of the model under adverse conditions such as noise, data distribution shift, and adversarial perturbation. A model with high robustness can maintain a high level of performance in complex and variable environments, even when the input data has some degree of disturbance, it can still output reliable results.

[0164] The present application uses Tiny-Imagenet and CIFAR-10 datasets. CIFAR-10 contains 50,000 training images, 5,000 training images per class, and 10,000 test images. Tiny-ImageNet is a challenging classification dataset that provides 100,000 images for training and 10,000 images for verification of 200 classes.

[0165] The results of the hierarchical integration knowledge distillation (HIKD) framework proposed in the present application are compared with a variety of existing knowledge distillation methods. Among these methods, including traditional Logit distillation methods (such as soft label method KD, decoupled knowledge distillation DKD, Wasserstein knowledge distillation WKD) and feature-based distillation methods.

[0166] For example, FitNet guides the student network to learn the feature representation of the teacher network by aligning the intermediate feature maps layer by layer. CRD introduces a contrastive learning mechanism, using positive and negative sample pairs to gradually bring the features of the teacher network and the student network closer or further apart in contrastive learning, thereby transferring knowledge. OFD sets factors such as feature position, connector composition, and loss function of various distillation algorithms to reduce information loss. RKD focuses on transferring structural relationship knowledge between the intermediate features of the teacher network and the student network. ReviewKD uses knowledge review to make each layer of the teacher model not only match the corresponding layer of the student network, but also match all previous layers. AT converts the feature map into a spatial attention map, which is used as valuable information to guide knowledge transfer. These methods cover the current mainstream knowledge distillation techniques, among which the soft label method KD is the classic Logit distillation method, which transfers knowledge by softening the output distribution of the teacher model. The FitNet and CRD methods are feature matching distillation methods that transfer knowledge by matching the intermediate features of the teacher and student models.

[0167] Training implementation details: The model generated by the algorithm of the present application is tested through different teacher-student architectures and compared with the baselines of the teacher and student to demonstrate the superiority of the algorithm. Considering the influence of different architectures on knowledge distillation, the present application constructs six different architectures, namely [ResNet34, ResNet18], [ResNet50, ResNet18], [ResNet34, MobileNetv1], [ResNet50, MobileNetv1], [ResNet101, ResNet18], and [ResNet101, ResNet50]. The present application uses an untrained teacher network as a prerequisite. The student uses SGD as the optimizer, sets the initial momentum to 0.9, and sets the weight decay to 1e-4. The initial learning rate is set to 0.1, and is halved every 30 periods during training. The present application compares multiple baselines, including KD, DKD, and WKD for Logit distillation, and CRD, AT, ReviewKD, OFD, and RKD for feature distillation.

[0168] Results on CIFAR-10: On the CIFAR-10 dataset, the present application sets [ResNet50, MobileNetv1], [ResNet34, ResNet18], and [ResNet50, ResNet18] as the comparison architecture, and the comparison method includes mainstream Logit distillation and Feature distillation.

[0169] As shown in Table 1, the method of the present application is superior to all the comparison methods in top-1 accuracy on CIFAR-10. The present application uses [ResNet101, ResNet50] as a new architecture scheme to perform visualization verification, because the gap between the model quantities tests the efficiency of the teacher to transfer knowledge. Finally, the comparison results are shown in Figure 4 The figure shows that the hierarchical knowledge distillation method based on KIKD of the present application is more separable than the baseline Vanilla KD (original knowledge distillation method), proving that the HIKD of the present application is beneficial to the distinguishability of deep features.

[0170] Table 1 sets [ResNet50, MobileNetv1], [ResNet34, ResNet18] and [ResNet50, ResNet18].

[0171] Table 1 Experimental results of three architectures on CIFAR-10 dataset

[0172]

[0173] Results on Tiny-Imagenet: The number of classes of CIFAR-10 is less, so the present application performs experiments on Tiny-Imagenet with 200 classes, and compared with the architecture method on CIFAR-10, the present application adds two architectures [ResNet34, MobileNetv1] and [ResNet101, ResNet18] to better verify the superior generalization of the present application method.

[0174] As shown in Table 2, the method of the present application is only lower than the Review KD method in top-1 (the probability that the class with the highest confidence is consistent with the real label) of [ResNet50, ResNet18] architecture and top-5 (the probability that the real label is included in the top 5 classes with the highest confidence) of [ResNet34, MobileNetv1] architecture, and is superior to the comparison method in other aspects. And the newly set [ResNet101, ResNet18] tests the efficiency of the teacher to transfer knowledge in the distillation method because of the large gap between the model quantities. Compared with other methods, the hierarchical guidance framework of the present application performs more outstandingly on the more challenging [ResNet101, ResNet18] distillation architecture. This shows that the method of the present application is excellent in the efficiency of the teacher to transfer knowledge.

[0175] Table 2 Settings [ResNet34, ResNet18], [ResNet50, ResNet18], [ResNet34, MobileNetvl], [ResNet50, MobileNetvl] and [ResNet101, ResNet18].

[0176] Table 2 Experimental results of five architectures on Tiny-Imagenet dataset

[0177]

[0178] Robustness evaluation: In order to verify the robustness of the comparative method, five kinds of perturbation methods are set in this application, which are FGSM (fast gradient sign method), PGD (projected gradient descent), Gaussian Noise Attack (Gaussian noise), Salt-and-Pepper Noise Attack (salt and pepper noise) and Poisson Noise Attack (Poisson noise).

[0179] The specific perturbation addition parameters are as follows: the perturbation amplitude of FGSM is 0.3. The perturbation amplitude of PGD is 0.3, the iteration step is 0.1, and the iteration number is 40. The standard deviation of Gaussian noise method is set to 25, and the mean is set to 0. In the Poisson noise method, the mean of the Poisson distribution is set to 0.1. In the salt and pepper noise method, the salt noise and pepper noise are 1:1, and the noise intensity is set to 0.04.

[0180] This application adopts 5 kinds of comparative perturbation methods, such as Figure 5As shown, first is PGD (projected gradient descent), which gradually adjusts the input data along the gradient direction under given constraints through iterative optimization, generating adversarial samples that can make the model prediction error. Then is FGSM (fast gradient sign method), which uses the gradient information of the model to directly add perturbations opposite to the gradient sign on the input data, quickly generating adversarial samples to test the sensitivity of the model to gradient direction perturbations. Then is Gaussian Noise (Gaussian noise), which is a random noise conforming to normal distribution, widely existing in natural environment. By adding Gaussian noise to the input data, the robustness of the model can be evaluated by simulating random interference in the actual scene. Next is Salt-and-Pepper Noise (Salt-and-Pepper noise), which contains randomly distributed white and black points, which can simulate image sensor failure or transmission error, etc. By adding salt-and-pepper noise, the anti-interference ability of the model to such extreme noise can be tested. Finally, Poisson Noise (Poisson noise) is a random noise related to signal strength, conforming to Poisson distribution, commonly used to simulate noise in photon counting and other physical processes. By adding Poisson noise, the performance of the model under low signal-to-noise ratio conditions can be evaluated. And all experiments are conducted on the data sets processed by the five methods.

[0181] Table 3 Experiments on Tiny-Imagenet dataset using [ResNet50, ResNet18] teacher-student architecture

[0182]

[0183] Table 4 Experiments on CIFAR-10 dataset using [ResNet50, MobileNetv1] teacher-student architecture

[0184]

[0185] And in order to more intuitively display the method of the present application, the present application processes the training data of CIFAR-10 using FGSM and visualizes the perturbation effect of Vanilla KD (original knowledge distillation method) and the hierarchical knowledge distillation method based on HIKD of the present application through t-SNE (t-distributed stochastic neighbor embedding) for comparison, as shown in Figure 6 As shown, the HIKD method of the present application exhibits better separability effect than the Vanilla KD method on the FGSM processed data set. As can be seen, the method of the present application has more obvious class aggregation than Vanilla KD on the t-SNE scatter plot, which indicates that the model of the present application has higher robustness.

[0186] As shown in Tables 3 and 4, the present application adopts [ResNet50, ResNet18] teacher-student architecture for robustness experiments on the Tiny-Imagenet dataset, and adopts [ResNet50, MobileNetv1] teacher-student architecture for robustness experiments on the CIFAR-10 dataset. It can be found that the robustness of the method of the present application is stronger than all the comparison methods on the CIFAR-10 and Tiny-Imagenet datasets. Therefore, the hierarchical guidance knowledge distillation proposed in the present application can improve the robustness of the student model.

[0187] According to the above, the present application can be applied in the field of automatic driving. Environment perception and decision-making: autonomous vehicles rely on numerous sensors (such as cameras, lidar, etc.) to perceive the surrounding environment and make decisions. However, sensor data is easily affected by adversarial factors such as adverse weather (such as rain, snow, fog, etc. leading to signal interference), malicious interference signals, etc. The student model learns from clean sensor data features in normal environments, which can enhance the perception and decision-making robustness under such adversarial interference, ensuring that autonomous vehicles can still drive stably and safely in complex road conditions and interference environments. High-precision map positioning and updating: precise positioning and map updating are achieved through analysis of map data and vehicle self-positioning data. If these data are tampered with or subjected to adversarial interference, it will affect the safety of vehicle driving. The student model trained by this method can learn based on accurate and undisturbed map and positioning data features, improving the robustness when the data is subjected to adversarial destruction, and ensuring the accurate positioning of the vehicle and the reliable updating of the map.

[0188] Based on the above method, the present application provides a hierarchical knowledge distillation system for enhancing the adversarial robustness of a student model, which comprises a hierarchical knowledge guidance module, a starting feature fusion learning module, a self-adaptive temperature standardization Logit module, a mixed knowledge fusion optimization module, and an output module.

[0189] Hierarchical knowledge guidance module: used for extracting and transmitting the Feature knowledge and Logit knowledge of the teacher model to the student model respectively; through hierarchical learning and decoupling, the student model learns the knowledge of the teacher model from different dimensions.

[0190] Starting feature fusion learning module: by introducing an attention mechanism to assign weights to features and convert them, combining multi-scale mean square error loss, the feature learning of the student model is optimized.

[0191] Self-adaptive temperature standardization Logit module: extracts Logit information from the teacher model, introduces a dynamic temperature parameter to adjust the temperature value, and standardizes the distribution of Logit values. The loss is calculated by KL divergence to optimize the Logit learning of the student model.

[0192] Mixed knowledge fusion optimization module: fuse Feature knowledge and Logit knowledge, balance corresponding distillation loss with hyperparameters, and optimize mixed knowledge distillation loss.

[0193] Student model output module: output the student model obtained by the final distillation after the above modules are processed.

[0194] Embodiments of the present application provide an electronic device, which includes at least one processor, at least one network interface, a user interface, a memory, and at least one communication bus.

[0195] The communication bus is used to realize the connection and communication between the components.

[0196] The user interface can include a display screen and a camera, and the optional user interface can further include a standard wired interface and a wireless interface.

[0197] The network interface can optionally include a standard wired interface and a wireless interface.

[0198] The processor can include one or more processing cores. The processor connects various parts within the server through various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one of the following hardware forms: digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor can integrate a combination of one or more of the following: central processing unit (Central Processing Unit, CPU), graphics processing unit (Graphics Processing Unit, GPU), and modem. Among them, the CPU is mainly used to process the operating system, user interface and application programs; the GPU is used to render and draw the content to be displayed on the display screen; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be realized by a separate chip.

[0199] The memory can include a random access memory (RAM) and a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets, or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the various method embodiments described above, etc.; and the data storage area can store data involved in the various method embodiments described above, etc. The memory can also optionally be at least one storage device located away from the aforementioned processor. As a computer storage medium, the memory can include an operating system, a network communication module, a user interface module, and an application program of the layered knowledge distillation method for enhancing the adversarial robustness of a student model.

[0200] In the electronic device of the embodiment, the user interface is mainly used to provide an interface for user input and obtain data input by the user; and the processor can be used to call an application program of the layered knowledge distillation method for enhancing the adversarial robustness of a student model stored in the memory, which, when executed by one or more processors, causes the electronic device to perform the method described in one or more of the above embodiments.

[0201] An electronic device readable storage medium stores instructions. When executed by one or more processors, the instructions cause the electronic device to perform the method described in one or more of the above embodiments.

[0202] The above description is only preferred embodiments of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A hierarchical knowledge distillation method for enhancing adversarial robustness of student models, characterized in that: Comprise the following steps: S1: Construct a hierarchical knowledge guidance framework, extract and transfer knowledge to the student model by decoupling semantic knowledge, and extract and transfer the characteristic knowledge and Logit knowledge of the teacher model to the student model; S2: Start the feature fusion learning mechanism, introduce the attention mechanism to assign weights to the features and perform feature conversion, optimize the feature learning of the student model based on the multi-scale mean square error loss, including the following three sub-steps: S21: Introduce the attention mechanism to assign feature weights, and convert important features of the teacher model into learnable feature knowledge of the student model; S22: Fuse the initial features of the student model with the current features; Measure the consistency between the student features and the teacher features through multi-scale mean square error loss; S23: Calculate the feature loss and optimize the student model; S3: Self-adaptive temperature standardization of Logit, extract Logit information from the teacher model, and perform distribution standardization processing, calculate the loss through KL divergence and optimize the Logit learning of the student model, including the following four sub-steps: S31: Extract Logit information from the teacher model; S32: Introduce a dynamic temperature parameter to dynamically adjust the temperature value according to the sharpness of the Logit; S33: Perform distribution standardization processing on the Logit value to make the student model focus on learning the relative relationship between Logits; S34: Calculate the probability distribution difference between the standardized Logit of the student model and the teacher model through KL divergence, and optimize the Logit learning of the student model; S4: Mixed hierarchical knowledge guidance framework, fuse the feature knowledge and Logit knowledge, and optimize the mixed knowledge distillation loss; S5: Output the final generated distilled student model.

2. The layered knowledge distillation method for enhancing student model adversarial robustness according to claim 1, wherein: In S21, the specific content of introducing the attention mechanism to assign feature weights is: Given an input picture and a teacher network ; represents the output Logit of the teacher; is divided into different parts , represents the classifier, are different stages separated by down-sampling layers; thus generating The output of is represented as: ， with the function symbol representing the nesting of functions, is represented as: ， The intermediate feature level representation of the teacher is denoted as The i-th layer feature representation of the teacher is denoted as: ， Single-level feature knowledge transferred by teacher model is expressed as: ， wherein, represents the i-th layer feature of the student, represents the i-th layer feature of the teacher; represents the important feature knowledge of the i-th layer of the student; represents the important feature knowledge of the i-th layer of the teacher; represents the distance function measuring the difference between the student network feature and the teacher network feature; represents the first layer feature of the teacher; represents the n-th layer feature of the teacher; represents the n-th stage of the teacher network T; represents the i-th stage of the teacher network T.

3. The layered knowledge distillation method for enhancing student model adversarial robustness according to claim 1, wherein: In S23, the specific content of calculating the feature loss and optimizing the student model is: The first stage feature of the network includes low-level detailed information, and the first stage feature is expressed as a starting feature; by introducing low semantic information of a fused starting feature layer, a student network can better understand high-level semantic information of a teacher network; in each stage of knowledge transmission of the teacher model, a student feature is fused with a feature and a feature loss between the student feature and the teacher feature is calculated; the loss generated by the initial feature fusion learning mechanism for each stage is represented as: ， Full distillation knowledge loss of multi-level knowledge distillation is expressed as: ， The distance sum formula between the initial feature of the student and the current student feature and the teacher feature is: ， Final characteristic loss is represented as: ， wherein, represents the important feature knowledge of the i-th layer of the student; represents the important feature knowledge of the i-th layer of the teacher; represents the i-th layer feature of the student; represents the i-th layer feature of the teacher; represents the distance function measuring the difference between the network features of the student and the network features of the teacher; n represents the sample size; represents the feature fusion module; represents the first layer feature of the student; represents the important feature knowledge of the first layer of the student.

4. The layered knowledge distillation method for enhancing student model adversarial robustness according to claim 1, wherein: In S33, the specific content is: Standardize the Logit value to zero mean and unit standard deviation to make the student model focus on learning the relative relationship between Logits, and the calculation formula of the distribution standardization function D1 is: ， where K represents the total number of categories; represents the kth component vector, x represents the original Logit value, represents the mean of the Logit vector x, represents the standard deviation of the Logit vector x, represents the adaptive temperature.

5. The layered knowledge distillation method for enhancing student model adversarial robustness according to claim 1, wherein: In S34, the specific content of calculating the probability distribution difference between the standardized Logit of the student model and the teacher model through KL divergence to optimize the Logit learning of the student model is: First, get the softmax output of the student and the teacher, specifically: ， ， wherein, represents the Logit value after the student distribution normalization function D1 processing; represents the Logit value after the teacher distribution normalization function D1 processing; represents the soft prediction of the i-th stage of the student network; represents the soft prediction of the i-th stage of the teacher network; represents the normalized exponential function; Then calculate the probability distribution difference through KL divergence, specifically: ， wherein KL denotes the KL divergence; denotes the normalized exponential function; denotes the Logit value after the student distribution standardization function D1 processing; denotes the Logit value after the teacher distribution c standardization function D1 processing; denotes the sample category number; Finally, the effect of temperature on label smoothing is considered, and the loss of Logit knowledge is eventually set is: ， wherein, represents the number of samples, KL represents the KL divergence; represents the temperature parameter of the student network; represents the temperature parameter of the teacher network; P represents the soft prediction of the student network; Q represents the soft prediction of the teacher network.

6. The layered knowledge distillation method for enhancing student model adversarial robustness according to claim 1, wherein: In S4, the specific implementation process is: S41: Fuse the feature knowledge and Logit knowledge, adjust the proportion of learning feature knowledge and Logit knowledge of the student model through decoupling; S42: Use hyperparameters to balance the feature distillation loss and the Logit distillation loss; S43: Optimize the mixed knowledge distillation loss.

7. The hierarchical knowledge distillation method for enhancing student model's adversarial robustness according to claim 6, characterized in that: In S43, the specific content of optimizing the mixed knowledge distillation loss is: ， ， wherein, represents cross-entropy loss, N represents the number of samples; C represents the number of sample categories; represents the true label of whether the ith sample belongs to category c; represents the probability that the student network predicts the ith sample to be category c; represents the mixed knowledge distillation loss; represents the final feature loss; represents the final Logit loss; , both represent hyperparameters; represents a dynamic factor.

8. A layered knowledge distillation system for enhancing adversarial robustness of a student model, implementing the layered knowledge distillation method for enhancing adversarial robustness of a student model according to any one of claims 1 to 7, characterized in that: The system comprises a hierarchical knowledge guidance module, a starting feature fusion learning module, an adaptive temperature standardization Logit module, a mixed knowledge fusion optimization module, and an output module. The hierarchical knowledge guidance module is used for extracting and delivering feature knowledge and Logit knowledge of a teacher model to a student model respectively, and letting the student model learn the knowledge of the teacher model from different dimensions through hierarchical learning and decoupling. The starting feature fusion learning module optimizes feature learning of the student model by introducing an attention mechanism to assign weights to features and convert them, and combining a multi-scale mean square error loss. The adaptive temperature standardization Logit module extracts Logit information from the teacher model, introduces a dynamic temperature parameter to adjust a temperature value, performs distribution standardization on Logit values, and optimizes Logit learning of the student model by calculating a loss through KL divergence. The mixed knowledge fusion optimization module balances corresponding distillation losses with a hyperparameter to optimize a mixed knowledge distillation loss by fusing feature knowledge and Logit knowledge. The output module outputs a student model obtained through final distillation.

9. An electronic device, comprising: The electronic device comprises a processor, a memory, a user interface, and a network interface, the memory is used for storing instructions, the user interface and the network interface are used for communicating with other devices, and the processor is used for executing the instructions stored in the memory to enable the electronic device to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer program stored in the memory can be loaded and executed by the processor to perform the method of any one of claims 1-7.

Images