Layered knowledge distillation method, system and equipment for enhancing adversarial robustness of student model and storage medium

Abstract

The invention discloses a hierarchical knowledge distillation method, system and equipment for enhancing the adversarial robustness of a student model, and a storage medium, and belongs to the field of artificial intelligence model optimization. Comprising the following steps: constructing a hierarchical knowledge guidance framework, and extracting and transmitting knowledge to a student model by using the hierarchical knowledge guidance framework; starting a feature fusion learning mechanism, introducing an attention mechanism for feature distribution weight to perform feature conversion, and optimizing feature learning of a student model; adaptive temperature standardization Logit: Logit information is extracted from the teacher model, distribution standardization processing is carried out, loss is calculated through KL divergence, and Logit learning of the student model is optimized; mixing a hierarchical knowledge guidance framework; and outputting the finally generated student model after distillation. According to the method, the two kinds of knowledge are aggregated through a decoupling method, higher robustness is achieved, and the efficient recognition rate on clean samples is reserved without introducing extra anti-disturbance data.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence model optimization, and particularly relates to a layered knowledge distillation method for enhancing the adversarial robustness of a student model, a system, a device and a storage medium. BACKGROUND

[0002] Computer vision applications are widespread, and DNN is an indispensable component. It plays a key role in various real-world tasks such as image classification, semantic segmentation, and object detection. However, completing these tasks often requires a large amount of computational power. Therefore, model distillation has emerged. Model distillation is a model compression technique aimed at transferring the knowledge of a large, complex model (teacher model) to a smaller, more efficient model (student model). Although knowledge distillation has made significant progress in improving model performance, studies have shown that even deep neural network (DNN) models obtained through model distillation can be vulnerable to adversarial perturbations. By adding tiny, imperceptible perturbations to input data, it can cause the model to output incorrect prediction results.

[0003] Therefore, when designing a model distillation method, the robustness of the student model generated by distillation is necessary to consider. Existing methods such as RSLAD (Robust Soft Label Adversarial Distillation) in defense distillation improve robustness by using robust soft labels, and MTARD (Multi-Teacher Adversarial Robust Distillation) improves model robustness by using a mixture of robust teacher models and normal teacher models for teaching. Both methods introduce additional training data and models, and have poor recognition efficiency on clean samples. Therefore, the existing technology mainly studies distillation methods that do not introduce additional adversarial data samples.

[0004] Current mainstream knowledge distillation methods include Logit distillation and Feature distillation. In the existing technology, the knowledge is first transferred through soft labels, and a temperature parameter is introduced to soften the output distribution of the teacher model, thereby capturing the similarity information between classes. Later, many methods based on Logit response have emerged. In terms of feature distillation, the FitNet distillation method is representative in proposing to use the feature map of the intermediate layer of the teacher as a "hint" to guide the student to learn the corresponding layer features. However, the current mainstream knowledge distillation methods all ignore the problem of the robustness of the student model, which is risky in the application of DNN models. SUMMARY

[0005] In view of the problems mentioned in the background art, the present application proposes a layered knowledge distillation method for enhancing the adversarial robustness of a student model, a system, a device and a storage medium, which teaches the student by decoupling the Logit knowledge and Feature knowledge of the teacher, achieving higher robustness; and without introducing additional adversarial data samples, maintaining high recognition rate on clean samples.

[0006] Technical scheme: In order to solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0007] A hierarchical knowledge distillation method for enhancing the adversarial robustness of a student model, comprising the following steps:

[0008] S1: Construct a hierarchical knowledge guidance framework, decouple semantic knowledge, and use the hierarchical knowledge guidance framework to extract and transfer knowledge to the student model;

[0009] S2: Start the feature fusion learning mechanism, introduce the attention mechanism to assign weights to the features and perform feature conversion, and based on the multi-scale mean square error loss, optimize the feature learning of the student model;

[0010] S21: Introduce the attention mechanism to assign feature weights, and convert important features of the teacher model into learnable feature knowledge for the student model;

[0011] S22: Fuse the initial features of the student model with the current features;

[0012] Measure the consistency between the student features and the teacher features through multi-scale mean square error loss;

[0013] S23: Calculate the feature loss and optimize the student model;

[0014] S3: Adaptive temperature standardization Logit, extract Logit information from the teacher model and perform distribution standardization processing, calculate the loss through KL divergence and optimize the Logit learning of the student model;

[0015] S31: Extract Logit information from the teacher model;

[0016] S32: Introduce a dynamic temperature parameter to dynamically adjust the temperature value according to the sharpness of the Logit;

[0017] S33: Perform distribution standardization processing on the Logit value to make the student model focus on learning the relative relationship between Logits;

[0018] S34: Calculate the probability distribution difference between the student model and the teacher model standardized Logit through KL divergence, and optimize the Logit learning of the student model;

[0019] S4: Mixed hierarchical knowledge guidance framework, fuse Feature knowledge and Logit knowledge, and optimize the mixed knowledge distillation loss;

[0020] S5: Output the final generated distilled student model.

[0021] As preferred, in S21, the specific content of introducing attention mechanism to assign feature weights is that:

[0022] Given an input image and a teacher network ; The output Logit of the teacher represents; Divided into different parts , The classifier represents, Different stages separated by down-sampling layers; therefore, the output of The output of

[0023] ,

[0024] The function symbol is used to represent the nesting of functions, The output is represented as:

[0025] ,

[0026] The intermediate feature level of the teacher represents The i-th layer feature of the teacher represents:

[0027] ,

[0028] The single-level feature knowledge transferred by the teacher model represents:

[0029] ,

[0030] Wherein, The i-th layer feature of the student represents, The i-th layer feature of the teacher represents; The important feature knowledge of the i-th layer of the student represents; The important feature knowledge of the i-th layer of the teacher represents; The distance function measures the difference between the feature of the student network and the feature of the teacher network; The first layer feature of the teacher represents; The n-th layer feature of the teacher represents; The n-th stage of the teacher network T represents; The i-th stage of the teacher network T represents.

[0031] As preferred, in S23, the specific content of calculating feature loss and optimizing the student model is that:

[0032] ​The first stage feature of the network includes low-level detailed information, and the first stage feature is denoted as a starting feature; by introducing low semantic information of the fused starting feature layer, the student network can better understand high-level semantic information of the teacher network; in each stage of knowledge transmission of the teacher model, the student feature is fused with the feature and the feature loss between the teacher feature is calculated;

[0033] The loss generated by the starting feature fusion learning mechanism in each stage is denoted as:

[0034] ,

[0035] The total distilled knowledge loss of the multi-level knowledge distillation is denoted as:

[0036] ,

[0037] The distance sum formula between the starting feature of the student and the current student feature and the teacher feature is denoted as the distance between the current student fusion feature and the teacher feature:

[0038] ,

[0039] The final feature loss is denoted as:

[0040] ,

[0041] wherein, represents important feature knowledge of the i-th layer of the student; represents important feature knowledge of the i-th layer of the teacher; represents the i-th layer feature of the student; represents the i-th layer feature of the teacher; represents a distance function for measuring the difference between the student network feature and the teacher network feature; n represents a sample capacity; represents a feature fusion module; represents the first layer feature of the student; represents important feature knowledge of the first layer of the student.

[0042] As a preferred, in S33, specifically:

[0043] The Logit value is standardized to zero mean and unit standard deviation, so that the student model focuses on learning the relative relationship between Logits, and the calculation formula of the distribution standardization function D1 is:

[0044]

[0045] Among them, K represents the total number of categories; represents the kth component vector, x represents the original Logit value, represents the mean of the Logit vector x, represents the standard deviation of the Logit vector x, Indicates the adaptive temperature.

[0046] Preferably, in S34, the probability distribution difference between the standardized Logit of the student model and the teacher model is calculated by KL divergence, and the specific content of optimizing the Logit learning of the student model is:

[0047] First, get the softmax output of the student and teacher, specifically:

[0048] ,

[0049] ,

[0050] in, Represents the Logit value after processing by the Student distribution standardization function D1; It represents the Logit value after the teacher distribution standardization function D1 is processed; represents the soft prediction of the i-th stage of the student network; represents the soft prediction of the teacher network at stage i; represents the normalized exponential function;

[0051] Then the probability distribution difference is calculated by KL divergence, specifically:

[0052] ,

[0053] Where KL represents KL divergence; represents the normalized exponential function; Represents the Logit value after processing by the Student distribution standardization function D1; It represents the Logit value of teacher distribution c after processing by standardization function D1; Indicates the number of sample categories;

[0054] Finally, considering the effect of temperature on label smoothing, the loss of Logit knowledge is finally set for:

[0055] ,

[0056] in, represents the number of samples, KL represents KL divergence; represents the temperature parameter of the student network; temperature parameter of the teacher network; P represents the soft prediction of the student network; Q represents the soft prediction of the teacher network.

[0057] As preferred, in S4, the specific implementation process is:

[0058] S41: Fuse Feature knowledge and Logit knowledge, and adjust the proportion of learning Feature knowledge and Logit knowledge of the student model by decoupling;

[0059] S42: Balance the feature distillation loss and the Logit distillation loss using hyperparameters;

[0060] S43: Optimize the mixed knowledge distillation loss.

[0061] As preferred, in S43, the specific content of optimizing the mixed knowledge distillation loss is:

[0062] ,

[0063] ,

[0064] wherein, represents the cross-entropy loss, N represents the number of samples; C represents the number of sample categories; represents the true label of whether the ith sample belongs to category c; represents the probability of the student network predicting the ith sample as category c; represents the mixed knowledge distillation loss; represents the final feature loss; represents the final Logit loss; , both represent hyperparameters; represents a dynamic factor.

[0065] A hierarchical knowledge distillation system for enhancing the adversarial robustness of a student model, which implements the hierarchical knowledge distillation method for enhancing the adversarial robustness of a student model described in any of the above, and the system comprises a hierarchical knowledge guidance module, a starting feature fusion learning module, a self-adaptive temperature standardization Logit module, a mixed knowledge fusion optimization module, and an output module.

[0066] The hierarchical knowledge guidance module is used to extract and deliver the Feature knowledge and Logit knowledge of the teacher model to the student model, and through hierarchical learning and decoupling, the student model learns the knowledge of the teacher model from different dimensions.

[0067] The starting feature fusion learning module optimizes the feature learning of the student model by introducing an attention mechanism to assign weights to features and convert them, and combining a multi-scale mean square error loss.

[0068] Adaptive temperature normalization Logit module: extract Logit information from the teacher model, introduce a dynamic temperature parameter to adjust the temperature value, and distribute the Logit value, and calculate the loss by KL divergence to optimize the Logit learning of the student model;

[0069] Hybrid knowledge fusion optimization module: fuse Feature knowledge and Logit knowledge, balance the corresponding distillation loss with hyperparameters, and optimize the hybrid knowledge distillation loss;

[0070] Output module: output the student model obtained by final distillation.

[0071] An electronic device includes a processor, a memory, a user interface, and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to make the electronic device execute the method described in any of the above.

[0072] A computer-readable storage medium stores a computer program capable of being loaded and executed by a processor to execute the method described in any of the above.

[0073] Advantages: Compared with the prior art, the present application has the following advantages:

[0074] (1) The knowledge distillation method of the present application creates a model trained only on clean samples, aiming to retain the high recognition rate of the model, while the model should have good robustness to resist potential adversarial perturbations.

[0075] (2) By decoupling the Logit knowledge and Feature knowledge of the teacher to teach the student, higher robustness can be achieved; and by not introducing additional adversarial data samples, the high recognition rate on clean samples is retained; and the Logit layer and the Feature layer have their unique focuses in knowledge representation, so by decoupling after hierarchical learning, the student model can learn the strong points of the teacher.

[0076] (3) By constructing two hierarchical knowledge extraction methods LMIFF to extract low-level semantic feature knowledge information, and ATNL to extract high-level semantic Logit knowledge information, the present application proves that the two kinds of knowledge have different focuses on adversarial perturbation pictures, and by extracting the knowledge focuses of the two and decoupling, the adversarial robustness is enhanced.

[0077] (4) How to extract useful information from the teacher's Logit knowledge level and Feature knowledge level and pass it to the student is a challenging problem. To solve these problems, the present application proposes a starting feature fusion learning mechanism in Feature knowledge distillation, an adaptive temperature standardization method in Logit knowledge distillation to learn useful knowledge, and finally aggregates the two kinds of knowledge through a decoupling method. The hierarchical knowledge guidance framework proposed by the present application enables the student to learn knowledge in multiple dimensions, which greatly improves the effectiveness of learning, and most importantly, it achieves higher robustness performance. BRIEF DESCRIPTION OF DRAWINGS

[0078] Figure 1 is a general flowchart of the present application;

[0079] Figure 2 is a schematic diagram of the Feature layer knowledge extraction module of the present application;

[0080] Figure 3 is a schematic diagram of the Logit layer knowledge extraction module of the present application;

[0081] Figure 4 is a t-SNE visualization comparison chart of the HIKD method and the Vanilla KD method of the present application;

[0082] Figure 5 is an effect display chart of five perturbation methods;

[0083] Figure 6 is a t-SNE visualization comparison chart of the HIKD method and the Vanilla KD method of the present application on the CIFAR-10 dataset after FGSM processing. DETAILED DESCRIPTION

[0084] The present application will be further illustrated below in combination with specific embodiments, which are implemented on the premise of the technical solutions of the present application, and it should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0085] As shown in Figure 1 , the hierarchical knowledge distillation method for enhancing the adversarial robustness of the student model provided by the present embodiment mainly includes the following steps:

[0086] S1: Construct a hierarchical knowledge guidance framework (Hierarchical Integration Knowledge Distillation, HIKD);

[0087] By decoupling low-level semantic knowledge and high-level semantic knowledge, the robustness of the student model is improved while retaining high recognition rate.

[0088] Specific operation: a framework is constructed to extract and pass the Feature knowledge and Logit knowledge of the teacher model to the student model respectively. Through hierarchical learning and decoupling, the student model learns the knowledge of the teacher model from different dimensions.

[0089] S2: Learning Mechanism of Initial Feature Fusion (LMIFF);

[0090] S21: introduce attention mechanism to assign feature weights, and convert important features of the teacher model into learnable feature knowledge of the student model;

[0091] In this embodiment, given an input picture and a teacher network ; taking the teacher network as an example, represents the output Logit of the teacher. It can be divided into different parts , wherein is a classifier, is different stages separated by down-sampling layers. Therefore, the output of is expressed as:

[0092]

[0093] represented by the function symbol as the nesting of functions, can be expressed as:

[0094]

[0095] wherein, is the output Logit of the teacher, and the intermediate feature level of the teacher is represented as , so the i-th layer feature of the teacher is represented as:

[0096]

[0097] Therefore, the single-level feature knowledge passed by the teacher model is expressed as:

[0098]

[0099] wherein, represents the i-th layer feature of the student, represents the i-th layer feature of the teacher; represents the important feature knowledge of the i-th layer of the student; represents the important feature knowledge of the i-th layer of the teacher; represents the first layer feature of the teacher; represents the n-th layer feature of the teacher; represents the n-th stage of the teacher network T; represents the i-th stage of the teacher network T. The attention mechanism is used to allocate feature weight fusion. This design is the main method of the B module. First, the input feature x is processed by the first convolutional layer Conv1. Then, the input feature x is spliced with the residual feature y which is upsampled to the same size by bilinear interpolation in the channel dimension, and the attention weight calculated by an attention convolutional layer and the learnable weight are used to dynamically adjust the contribution ratio of x and y.

[0100] represents the distance function measuring the difference between the student network feature and the teacher network feature.

[0101] The attention mechanism is used to allocate feature weight, and the important features of the teacher model are converted into learnable feature knowledge of the student model. As shown in Figure 2 represents the important features of the teacher are converted into important feature knowledge that the student can learn.

[0102] S22: Fuse the starting feature of the student model with the current feature.

[0103] The consistency between the student feature and the teacher feature is measured by the multi-scale mean square error loss (MSE).

[0104] S23: Calculate the feature loss and optimize the student model.

[0105] The first stage feature of the network mainly includes low-level detail information such as edges and textures. The first stage feature is the starting feature, which retains the most detailed information. By introducing the low semantic information of the starting feature layer, the student network can better understand the high-level semantic information of the teacher network. Low semantic information can be used as an auxiliary signal to help the student network better capture the feature representation of the teacher network. In each stage of the teacher model knowledge transfer, the student feature is fused with feature and the feature loss between the student feature and the teacher feature is calculated.

[0106] By optimizing the feature loss, the student model can better learn the low-level semantic information of the teacher model.

[0107] The loss generated by the starting feature fusion learning mechanism in each stage

[0108] , ​

[0109] All distilled knowledge losses of multi-level knowledge distillation is denoted as:

[0110] ,

[0111] The distance between the student's initial feature and the current student feature and the distance between the teacher feature are summed up respectively, denoted as the distance between the student fusion feature and the current teacher feature:

[0112] ,

[0113] Final feature loss is denoted as:

[0114] ,

[0115] wherein, denotes the important feature knowledge of the i-th layer of the student; denotes the important feature knowledge of the i-th layer of the teacher; denotes the i-th layer feature of the student; denotes the i-th layer feature of the teacher; denotes the distance function to measure the difference between the student network feature and the teacher network feature; n denotes the sample capacity; denotes the feature fusion module for fusion between two features; denotes the first layer feature of the student; denotes the important feature knowledge of the first layer of the student.

[0116] The feature fusion module first processes the input feature x through the first convolutional layer Conv1. Then the input feature x is spliced with the residual feature y which is up-sampled to the same size by bilinear interpolation in the channel dimension, and the contribution ratio of x and y is dynamically adjusted by the attention weight calculated through an attention convolutional layer and the learnable weight.

[0117] Low semantic information can be used as an auxiliary signal to help the student network better capture the learning signals contained in the current teacher network feature representation.

[0118] S3: Adaptive Temperature Normalization of Logit (ATNL);

[0119] S31: Extract Logit information from the teacher model.

[0120] These Logit information contains high-level semantic knowledge. Specifically as follows:

[0121]

[0122] in, is the predicted probability of the i-th category, is the logit value of the model corresponding to the i-th category, is the temperature parameter that controls the smoothness of the probability distribution.

[0123] S32: Introducing dynamic temperature parameters.

[0124] The temperature value is dynamically adjusted according to the sharpness (dispersion) of the Logit. When the sharpness is high, the Logit value difference is large, and the temperature parameter is also adjusted accordingly.

[0125] The temperature value settings are as follows:

[0126]

[0127] in, Indicates adaptive temperature; Indicates the logit value of the model corresponding to the i-th category, This means calculating the logarithm of the sum of the exponentials of all logit values ​​to quantify logit sharpness. The decay rate changes with the number of training rounds. The sharpness dynamically adjusts the softness and hardness of the label to help students better learn the adaptive label information. is a numerical stability constant to avoid numerical underflow / overflow.

[0128] S33: Standardize the distribution of Logit values.

[0129] like Figure 3 As shown, the Logit values ​​are standardized to zero mean and unit standard deviation, so that the student model focuses on learning the relative relationship between Logits. The details are as follows:

[0130] The calculation formula of the distribution normalization function D1 is:

[0131]

[0132] Among them, K represents the total number of categories; Indicates the component vectors, represents the original Logit value, Represents the Logit vector The mean of Represents the Logit vector The standard deviation of Indicates the above-mentioned adaptive temperature.

[0133] After such processing, the Logit vector of each sample will have a zero mean and a unit standard deviation. After processing by the standardization function D1, the logic value of the student model no longer needs to match the absolute size of the teacher model, but can be freely selected within the range suitable for its capacity.

[0134] S34: Calculate the loss by KL divergence and optimize the student model.

[0135] By optimizing the Logit loss, the student model learns better the advanced semantic information of the teacher model.

[0136] First, get the softmax output of the student and teacher:

[0137]

[0138]

[0139] where, represents the Logit value of the student distribution after processing by the standardization function D1; represents the Logit value of the teacher distribution after processing by the standardization function D1; represents the soft prediction of the i-th stage of the student network; represents the soft prediction of the i-th stage of the teacher network. represents the normalized exponential function.

[0140] Then calculate the probability distribution difference by KL divergence, specifically:

[0141]

[0142] where, represents the normalized exponential function; represents the Logit value of the student distribution after processing by the standardization function D1; represents the Logit value of the teacher distribution after processing by the standardization function D1; C represents the number of sample categories; KL represents the KL divergence.

[0143] The formula is simplified as:

[0144]

[0145] where, P represents the soft prediction of the student network; Q represents the soft prediction of the teacher network; represents the soft prediction of the i-th stage of the student network; represents the soft prediction of the i-th stage of the teacher network; C represents the number of sample categories.

[0146] Finally, considering the effect of temperature on label smoothing, the final Logit knowledge loss is set as: ​

[0147]

[0148] wherein, is the number of samples, KL is the KL divergence (Kullback-Leibler divergence); represents the temperature parameter of the student network, used to control the softening degree of the student output softmax; represents the temperature parameter of the teacher network, used to control the softening degree of the student output softmax; P represents the soft prediction of the student network; Q represents the soft prediction of the teacher network.

[0149] S4: Mixed Hierarchical Integration Framework;

[0150] S41: Fusion of Feature knowledge and Logit knowledge.

[0151] By decoupling, the proportion between learning Feature knowledge and Logit knowledge of the student model is adjusted.

[0152] S42: Use of hyperparameters to balance the losses of different knowledge.

[0153] Use hyperparameters (such as a and β) to balance the Feature distillation loss and the Logit distillation loss, and use the dynamic factor λ to balance the Feature distillation loss and the original loss.

[0154] S43: Optimization of mixed knowledge distillation loss.

[0155] The Feature distillation and the Logit distillation have different sensitivities to noise and perturbation methods, and the two distillation methods have their own strong fields when verifying pictures. By fusing the different key knowledge learned by the two distillation methods, model distillation is realized only on clean samples to improve the robustness of the student model. By optimizing the mixed knowledge distillation loss, the student model obtains high recognition rate on clean samples while improving the adversarial robustness.

[0156] The specific calculation process is as follows:

[0157]

[0158]

[0159] wherein, represents the cross-entropy loss, N represents the number of samples; C represents the number of sample categories; represents the true label of whether the ith sample belongs to category c (0 or 1, usually using one-hot encoding). represents the probability that the student network predicts the i-th sample as class c (softmax output); represents the mixed knowledge distillation loss; represents the final feature loss; represents the final Logit loss; 、 represents the hyperparameter; represents the dynamic factor.

[0160] S5: Output the final generated distilled student model.

[0161] The present application uses accuracy and robustness as evaluation criteria:

[0162] Accuracy: Accuracy is a key indicator of model performance, used to evaluate the proportion of correct classification or prediction of the model on test data. Specifically, it quantifies the accuracy of the model by calculating the proportion of model output results consistent with the true label. The higher the accuracy, the higher the correctness of the model in processing data, and the more effective the model can complete the task.

[0163] Robustness: Robustness is used to evaluate the stability of the model in the face of various disturbances and abnormal situations. It measures the performance of the model under adverse conditions such as noise, data distribution shift, and adversarial perturbation. A model with high robustness can maintain a high level of performance in complex and variable environments, even when the input data has some degree of disturbance, it can still output reliable results.

[0164] The present application uses Tiny-Imagenet and CIFAR-10 datasets. CIFAR-10 contains 50,000 training images, 5,000 training images per class, and 10,000 test images. Tiny-ImageNet is a challenging classification dataset that provides 100,000 images for training and 10,000 images for verification of 200 classes.

[0165] The results of the hierarchical integration knowledge distillation (HIKD) framework proposed in the present application are compared with a variety of existing knowledge distillation methods. Among these methods, including traditional Logit distillation methods (such as soft label method KD, decoupled knowledge distillation DKD, Wasserstein knowledge distillation WKD) and feature-based distillation methods.

[0166] For example, FitNet guides the student network to learn the feature representation of the teacher network by aligning the intermediate feature maps layer by layer. CRD introduces a contrastive learning mechanism, using positive and negative sample pairs to gradually bring the features of the teacher network and the student network closer or further apart in contrastive learning, thereby transferring knowledge. OFD sets factors such as feature position, connector composition, and loss function of various distillation algorithms to reduce information loss. RKD focuses on transferring structural relationship knowledge between the intermediate features of the teacher network and the student network. ReviewKD uses knowledge review to make each layer of the teacher model not only match the corresponding layer of the student network, but also match all previous layers. AT converts the feature map into a spatial attention map, which is used as valuable information to guide knowledge transfer. These methods cover the current mainstream knowledge distillation techniques, among which the soft label method KD is the classic Logit distillation method, which transfers knowledge by softening the output distribution of the teacher model. The FitNet and CRD methods are feature matching distillation methods that transfer knowledge by matching the intermediate features of the teacher and student models.

[0167] Training implementation details: The model generated by the algorithm of the present application is tested through different teacher-student architectures and compared with the baselines of the teacher and student to demonstrate the superiority of the algorithm. Considering the influence of different architectures on knowledge distillation, the present application constructs six different architectures, namely [ResNet34, ResNet18], [ResNet50, ResNet18], [ResNet34, MobileNetv1], [ResNet50, MobileNetv1], [ResNet101, ResNet18], and [ResNet101, ResNet50]. The present application uses an untrained teacher network as a prerequisite. The student uses SGD as the optimizer, sets the initial momentum to 0.9, and sets the weight decay to 1e-4. The initial learning rate is set to 0.1, and is halved every 30 periods during training. The present application compares multiple baselines, including KD, DKD, and WKD for Logit distillation, and CRD, AT, ReviewKD, OFD, and RKD for feature distillation.

[0168] Results on CIFAR-10: On the CIFAR-10 dataset, the present application sets [ResNet50, MobileNetv1], [ResNet34, ResNet18], and [ResNet50, ResNet18] as the comparison architecture, and the comparison method includes mainstream Logit distillation and Feature distillation.

[0169] As shown in Table 1, the method of the present application is superior to all the comparison methods in top-1 accuracy on CIFAR-10. The present application uses [ResNet101, ResNet50] as a new architecture scheme to perform visualization verification, because the gap between the model quantities tests the efficiency of the teacher to transfer knowledge. Finally, the comparison results are shown in Figure 4 The figure shows that the hierarchical knowledge distillation method based on KIKD of the present application is more separable than the baseline Vanilla KD (original knowledge distillation method), proving that the HIKD of the present application is beneficial to the distinguishability of deep features.

[0170] Table 1 sets [ResNet50, MobileNetv1], [ResNet34, ResNet18] and [ResNet50, ResNet18].

[0171] Table 1 Experimental results of three architectures on CIFAR-10 dataset

[0172]

[0173] Results on Tiny-Imagenet: The number of classes of CIFAR-10 is less, so the present application performs experiments on Tiny-Imagenet with 200 classes, and compared with the architecture method on CIFAR-10, the present application adds two architectures [ResNet34, MobileNetv1] and [ResNet101, ResNet18] to better verify the superior generalization of the present application method.

[0174] As shown in Table 2, the method of the present application is only lower than the Review KD method in top-1 (the probability that the class with the highest confidence is consistent with the real label) of [ResNet50, ResNet18] architecture and top-5 (the probability that the real label is included in the top 5 classes with the highest confidence) of [ResNet34, MobileNetv1] architecture, and is superior to the comparison method in other aspects. And the newly set [ResNet101, ResNet18] tests the efficiency of the teacher to transfer knowledge in the distillation method because of the large gap between the model quantities. Compared with other methods, the hierarchical guidance framework of the present application performs more outstandingly on the more challenging [ResNet101, ResNet18] distillation architecture. This shows that the method of the present application is excellent in the efficiency of the teacher to transfer knowledge.

[0175] Table 2 Settings [ResNet34, ResNet18], [ResNet50, ResNet18], [ResNet34, MobileNetvl], [ResNet50, MobileNetvl] and [ResNet101, ResNet18].

[0176] Table 2 Experimental results of five architectures on Tiny-Imagenet dataset

[0177]

[0178] Robustness evaluation: In order to verify the robustness of the comparative method, five kinds of perturbation methods are set in this application, which are FGSM (fast gradient sign method), PGD (projected gradient descent), Gaussian Noise Attack (Gaussian noise), Salt-and-Pepper Noise Attack (salt and pepper noise) and Poisson Noise Attack (Poisson noise).

[0179] The specific perturbation addition parameters are as follows: the perturbation amplitude of FGSM is 0.3. The perturbation amplitude of PGD is 0.3, the iteration step is 0.1, and the iteration number is 40. The standard deviation of Gaussian noise method is set to 25, and the mean is set to 0. In the Poisson noise method, the mean of the Poisson distribution is set to 0.1. In the salt and pepper noise method, the salt noise and pepper noise are 1:1, and the noise intensity is set to 0.04.

[0180] This application adopts 5 kinds of comparative perturbation methods, such as Figure 5As shown, first is PGD (projected gradient descent), which gradually adjusts the input data along the gradient direction under given constraints through iterative optimization, generating adversarial samples that can make the model prediction error. Then is FGSM (fast gradient sign method), which uses the gradient information of the model to directly add perturbations opposite to the gradient sign on the input data, quickly generating adversarial samples to test the sensitivity of the model to gradient direction perturbations. Then is Gaussian Noise (Gaussian noise), which is a random noise conforming to normal distribution, widely existing in natural environment. By adding Gaussian noise to the input data, the random interference in the actual scene can be simulated to evaluate the robustness of the model. Next is Salt-and-Pepper Noise (salt and pepper noise), which contains randomly distributed white and black points, which can simulate image sensor failure or transmission error, etc. By adding salt and pepper noise, the anti-interference ability of the model to such extreme noise can be tested. Finally, Poisson Noise (Poisson noise) is a random noise related to signal strength, conforming to Poisson distribution, commonly used to simulate noise in photon counting and other physical processes. By adding Poisson noise, the performance of the model under low signal-to-noise ratio conditions can be evaluated. And all experiments are conducted on the data sets processed by the five methods.

[0181] Table 3 Experiments on Tiny-Imagenet dataset using [ResNet50, ResNet18] teacher-student architecture

[0182]

[0183] Table 4 Experiments on CIFAR-10 dataset using [ResNet50, MobileNetv1] teacher-student architecture

[0184]

[0185] And in order to more intuitively display the method of the present application, the present application processes the training data of CIFAR-10 using FGSM and visualizes the perturbation effect of Vanilla KD (original knowledge distillation method) and the hierarchical knowledge distillation method based on HIKD of the present application through t-SNE (t-distributed stochastic neighbor embedding) for comparison, as shown in Figure 6 As shown, the HIKD method of the present application exhibits better separability effect than the Vanilla KD method on the FGSM processed data set. It can be seen that the method of the present application has more obvious class aggregation than Vanilla KD on the t-SNE scatter plot, which indicates that the model of the present application has higher robustness.

[0186] As shown in Tables 3 and 4, the present application adopts [ResNet50, ResNet18] teacher-student architecture for robustness experiments on the Tiny-Imagenet dataset, and adopts [ResNet50, MobileNetv1] teacher-student architecture for robustness experiments on the CIFAR-10 dataset. It can be found that the robustness of the method of the present application is stronger than all the comparison methods on the CIFAR-10 and Tiny-Imagenet datasets. Therefore, the hierarchical guidance knowledge distillation proposed in the present application can improve the robustness of the student model.

[0187] According to the above, the present application can be applied in the field of automatic driving. Environment perception and decision-making: autonomous vehicles rely on numerous sensors (such as cameras, lidar, etc.) to perceive the surrounding environment and make decisions. However, sensor data is easily affected by adversarial factors such as adverse weather (such as rain, snow, fog, etc. leading to signal interference), malicious interference signals, etc. The student model learns from clean sensor data features in normal environments, which can enhance the perception and decision-making robustness under such adversarial interference, ensuring that autonomous vehicles can still drive stably and safely in complex road conditions and interference environments. High-precision map positioning and updating: precise positioning and map updating are achieved through analysis of map data and vehicle self-positioning data. If these data are tampered with or subjected to adversarial interference, it will affect the safety of vehicle driving. The student model trained by this method can learn based on accurate and undisturbed map and positioning data features, improving the robustness when the data is subjected to adversarial destruction, and ensuring the accurate positioning of the vehicle and the reliable updating of the map.

[0188] Based on the above method, the present application provides a hierarchical knowledge distillation system for enhancing the adversarial robustness of a student model, which comprises a hierarchical knowledge guidance module, a starting feature fusion learning module, a self-adaptive temperature standardization Logit module, a mixed knowledge fusion optimization module, and an output module.

[0189] Hierarchical knowledge guidance module: used for extracting and transmitting the Feature knowledge and Logit knowledge of the teacher model to the student model respectively; through hierarchical learning and decoupling, the student model learns the knowledge of the teacher model from different dimensions.

[0190] Starting feature fusion learning module: by introducing an attention mechanism to assign weights to features and convert them, combining multi-scale mean square error loss, the feature learning of the student model is optimized.

[0191] Self-adaptive temperature standardization Logit module: extracts Logit information from the teacher model, introduces a dynamic temperature parameter to adjust the temperature value, and standardizes the distribution of Logit values. The loss is calculated by KL divergence to optimize the Logit learning of the student model.

[0192] Mixed knowledge fusion optimization module: fuse Feature knowledge and Logit knowledge, balance corresponding distillation loss with hyperparameters, and optimize mixed knowledge distillation loss.

[0193] Student model output module: output the student model obtained by the final distillation after the processing of the above modules.

[0194] The embodiments of the present application provide an electronic device, which comprises at least one processor, at least one network interface, a user interface, a memory and at least one communication bus.

[0195] The communication bus is used to realize the connection and communication among the components.

[0196] The user interface can comprise a display screen and a camera, and the optional user interface can further comprise a standard wired interface and a wireless interface.

[0197] The network interface can optionally comprise a standard wired interface and a wireless interface.

[0198] The processor can comprise one or more processing cores. The processor connects various parts in the server through various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory and calling data stored in the memory. Optionally, the processor can be realized in at least one of the hardware forms of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA) and programmable logic array (Programmable Logic Array, PLA). The processor can integrate a combination of one or more of central processing units (Central Processing Unit, CPU), graphics processing units (Graphics Processing Unit, GPU) and modems. Among them, the CPU is mainly used to process the operating system, user interface and application programs; the GPU is used to render and draw the content to be displayed on the display screen; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be realized by a separate chip.

[0199] The memory can include a random access memory (RAM) and a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets, or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the various method embodiments described above, etc.; and the data storage area can store data involved in the various method embodiments described above, etc. The memory can also optionally be at least one storage device located away from the aforementioned processor. As a computer storage medium, the memory can include an operating system, a network communication module, a user interface module, and an application program of the layered knowledge distillation method for enhancing the adversarial robustness of a student model.

[0200] In the electronic device of the embodiment, the user interface is mainly used to provide an interface for user input and obtain data input by the user; and the processor can be used to call an application program of the layered knowledge distillation method for enhancing the adversarial robustness of a student model stored in the memory, which, when executed by one or more processors, causes the electronic device to perform the method described in one or more of the above embodiments.

[0201] An electronic device readable storage medium stores instructions. When executed by one or more processors, the instructions cause the electronic device to perform the method described in one or more of the above embodiments.

[0202] The above description is only preferred embodiments of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A hierarchical knowledge distillation method for enhancing the adversarial robustness of student models, characterized by: The following steps are involved: S1: Construct a hierarchical knowledge guidance framework, decouple semantic knowledge, and use the hierarchical knowledge guidance framework to extract and transfer knowledge to the student model; S2: Initial feature fusion learning mechanism, introducing the attention mechanism to assign weights to features and perform feature transformation, and optimizing the feature learning of the student model based on multi-scale mean squared error loss; S21: Introduce the attention mechanism to assign feature weights, converting the important features of the teacher model into feature knowledge that can be learned by the student model; S22: Fuse the starting features of the student model with the current features; The consistency between student features and teacher features is measured by multi-scale mean squared error loss; S23: Calculate feature loss and optimize student model; S3: Adaptive temperature normalized logit, extracts logit information from the teacher model and normalizes its distribution, calculates the loss through KL divergence and optimizes the logit learning of the student model; S31: Extract Logit information from the teacher model; S32: Introducing dynamic temperature parameters to dynamically adjust the temperature value according to the sharpness of Logit; S33: Normalize the distribution of Logit values ​​so that the student model can focus on learning the relative relationship between Logits; S34: Calculate the probability distribution difference between the standardized logit of the student model and the teacher model through KL divergence to optimize the logit learning of the student model; S4: Hybrid hierarchical knowledge guidance framework, which integrates feature knowledge and logit knowledge to optimize the hybrid knowledge distillation loss; S5: Output the final generated distilled student model.

2. The hierarchical knowledge distillation method for enhancing the robustness of the student model according to claim 1, characterized in that: In S21, the specific content of introducing the attention mechanism to assign feature weights is as follows: Given an input image and teacher network ; represents the teacher's output Logit; Divided into different parts , represents a classifier, are different stages separated by downsampling layers; thus generating The output is represented as: ， Using function notation Indicates the nesting of functions. Expressed as: ， The intermediate feature level of the teacher is represented as , the teacher's i-th layer feature is expressed as: ， Single-level feature knowledge transferred by the teacher model Expressed as: ， in, represents the i-th layer feature of students, represents the i-th level characteristics of teachers; represents the important feature knowledge of the student at level i; represents the important feature knowledge of the teacher at level i; The representation distance function measures the difference between the characteristics of the student network and the teacher network; represents the first-level characteristics of teachers; represents the n-th level characteristics of teachers; represents the nth stage of the teacher network T; represents the i-th stage of the teacher network T.

3. The hierarchical knowledge distillation method for enhancing the robustness of the student model according to claim 1, characterized in that: In S23, the specific contents of calculating feature loss and optimizing the student model are: The first stage features of the network include low-level detail information, and the features of the first stage are represented as starting features. By introducing the low-level semantic information of the fusion starting feature layer, the student network can better understand the high-level semantic information of the teacher network. In each stage of the teacher model knowledge transfer, the student network is used to Features and The features are fused and calculated with the teacher Feature loss between features; The loss generated by the initial feature fusion learning mechanism at each stage Expressed as: ， The total distillation knowledge loss of multi-level knowledge distillation Expressed as: ， The distance between the student's initial features and the current student features and the teacher's features is summed up to get the distance between the student's fusion features and the current teacher's features: ， Final feature loss Expressed as: ， in, represents the important feature knowledge of the student at level i; represents the important feature knowledge of the teacher at level i; represents the i-th level features of students; represents the i-th level characteristics of teachers; represents the distance function measuring the difference between the characteristics of the student network and the teacher network; n represents the sample size; represents the feature fusion module; represents the first-level characteristics of students; Represents the important feature knowledge of students at the first level.

4. The hierarchical knowledge distillation method for enhancing the adversarial robustness of the student model according to claim 1, characterized in that: In S33, specifically: The Logit values ​​are standardized to zero mean and unit standard deviation, so that the student model focuses on learning the relative relationship between Logits. The calculation formula of the distribution normalization function D1 is: ， Among them, K represents the total number of categories; represents the kth component vector, x represents the original Logit value, represents the mean of the Logit vector x, represents the standard deviation of the Logit vector x, Indicates the adaptive temperature.

5. The hierarchical knowledge distillation method for enhancing the adversarial robustness of the student model according to claim 1, characterized in that: In S34, the probability distribution difference between the student model and the teacher model's standardized Logit is calculated by KL divergence. The specific content of optimizing the student model's Logit learning is: First, get the softmax output of the student and teacher, specifically: ， ， in, Represents the Logit value after processing by the Student distribution standardization function D1; It represents the Logit value after the teacher distribution standardization function D1 is processed; represents the soft prediction of the i-th stage of the student network; represents the soft prediction of the teacher network at stage i; represents the normalized exponential function; Then the probability distribution difference is calculated by KL divergence, specifically: ， Where KL represents KL divergence; represents the normalized exponential function; Represents the Logit value after processing by the Student distribution standardization function D1; It represents the Logit value of teacher distribution c after processing by standardization function D1; Indicates the number of sample categories; Finally, considering the effect of temperature on label smoothing, the loss of Logit knowledge is finally set for: ， in, represents the number of samples, KL represents KL divergence; represents the temperature parameter of the student network; represents the temperature parameter of the teacher network; P represents the soft prediction of the student network; Q represents the soft prediction of the teacher network.

6. The hierarchical knowledge distillation method for enhancing the adversarial robustness of the student model according to claim 1, characterized in that: In S4, the specific implementation process is: S41: Integrate feature knowledge and logit knowledge, and adjust the proportion between feature knowledge and logit knowledge learned by the student model through decoupling; S42: Use hyperparameters to balance feature distillation loss and logit distillation loss; S43: Optimizing hybrid knowledge distillation loss.

7. The hierarchical knowledge distillation method for enhancing the robustness of the student model according to claim 6, characterized in that: In S43, the specific content of optimizing the mixed knowledge distillation loss is: ， ， in, Represents cross entropy loss, N represents the number of samples; C represents the number of sample categories; Indicates whether the i-th sample belongs to the true label of category c; represents the probability that the student network predicts that the i-th sample is category c; represents the mixed knowledge distillation loss; Represents the final feature loss; Represents the final Logit loss; 、 All represent hyperparameters; Represents the dynamic factor.

8. A hierarchical knowledge distillation system for enhancing the adversarial robustness of a student model, implementing the hierarchical knowledge distillation method for enhancing the adversarial robustness of a student model as described in any one of claims 1 to 7, characterized in that: The system includes a hierarchical knowledge guidance module, an initial feature fusion learning module, an adaptive temperature normalization Logit module, a hybrid knowledge fusion optimization module, and an output module; Hierarchical knowledge guidance module: This module extracts and transfers the teacher model's feature knowledge and logit knowledge to the student model. Through hierarchical learning and decoupling, the student model learns the teacher model's knowledge from different dimensions. Initial feature fusion learning module: This module introduces an attention mechanism to assign weights to features and transforms them, and combines it with a multi-scale mean square error loss to optimize the feature learning of the student model. Adaptive Temperature Normalized Logit Module: This module extracts logit information from the teacher model, introduces a dynamic temperature parameter to adjust the temperature value, normalizes the distribution of logit values, and optimizes the logit learning of the student model through KL divergence calculation loss. Hybrid knowledge fusion optimization module: It integrates feature knowledge and logit knowledge, uses hyperparameters to balance the corresponding distillation loss, and optimizes the hybrid knowledge distillation loss; Output module: Outputs the final distilled student model.

9. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

Images