Inplanatable data-knowledge-free distillation method based on class activation graph

Abstract

The invention discloses an interpretable data-knowledge-free distillation method based on a class activation graph, and the method comprises the following steps: inputting random noise to a generator, and generating a composite image similar to the distribution of a target data set; respectively inputting the synthesized image into a teacher model and a student model which are subjected to structure adjustment; respectively generating corresponding class activation diagrams through the teacher model and the student model; inputting the teacher class activation graph and the student class activation graph into a pre-trained scale mapping network, and performing cross-scale scaling on the class activation graphs; performing regularization processing on the scaled class activation diagram to obtain a standardized teacher class activation diagram and a student class activation diagram; calculating mean square error loss based on the teacher type activation graph and the student type activation graph to realize knowledge transmission; and finally, optimizing the generator in combination with information entropy loss, one-time heat loss and activation loss. According to the method, effective knowledge distillation can be realized under the condition of no real data, and the interpretability and stability of the distillation process are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and deep learning technology, specifically to an interpretable data-free knowledge distillation method based on class activation graphs. Background Technology

[0002] Knowledge distillation has garnered widespread attention due to its universality and effectiveness. Its basic idea is to train a lightweight student model to mimic the behavior of a high-performance teacher model, thereby improving inference efficiency while maintaining accuracy. Traditional knowledge distillation methods generally rely on raw training data. In data-constrained scenarios, such as those involving privacy protection, data compliance (e.g., GDPR, HIPAA), trade secrets, or data loss, obtaining raw data is often impractical. This real-world problem has spurred the research direction of data-free knowledge distillation. The core idea of ​​data-free knowledge distillation is to use generative models to synthesize pseudo-samples, using feedback from the teacher model to guide the training of the student model, thus achieving knowledge transfer in the absence of real data. Typical methods include DAFL, DFAD, and RDSKD, which improve the quality and coverage of pseudo-data by introducing mechanisms such as adversarial generation, feature matching, and diversity constraints.

[0003] While knowledge distillation without data offers a viable solution for scenarios with missing data, existing methods primarily rely on the teacher model's logits or feature distribution for supervision, neglecting explicit modeling of the teacher's discriminative regions. This leads to situations where, although the student model may approximate the teacher's predictions, deviations may occur in decision logic and regions of interest, resulting in insufficient interpretability. Furthermore, the limited stability of knowledge transfer means that pseudo-data often lacks naturalness in semantics and structure, and the lack of clear visualization verification methods makes the distillation process difficult to interpret and optimize. Additionally, when the teacher and student models belong to different architecture families (e.g., ResNet → MobileNet), traditional logits alignment methods cannot guarantee the effectiveness of knowledge transfer and exhibit poor cross-architecture adaptability. Therefore, there is an urgent need for a knowledge distillation method that can be performed without real data, balancing performance and interpretability. Summary of the Invention

[0004] The purpose of this invention is to provide an interpretable, data-free knowledge distillation method based on class activation graphs. By using the class activation graph of the teacher model as the main knowledge carrier, a generator synthesizes pseudo-samples, guides the generator and student model through the class activation graph, and achieves cross-scale alignment of the class activation graph through a pre-trained scale mapping network, robust and interpretable knowledge transfer can be completed without accessing real data, thereby solving the problems mentioned in the background art.

[0005] This invention provides the following technical solution: an interpretability-free knowledge distillation method based on class activation graphs, comprising the following steps:

[0006] Step S1: Input random noise into the generator to generate a synthetic image with a distribution similar to the target dataset;

[0007] Preferably, generating synthetic images with a distribution similar to the target dataset includes:

[0008] S11: Sample the noise vector z from a standard Gaussian or uniform distribution as the input to the generator;

[0009] S12: The generator uses a multi-layer convolution and deconvolution structure, combined with batch normalization and non-linear activation functions. The Tanh function is used in the last layer of convolution to normalize the output pixels to the range of [-1,1].

[0010] S13: Synchronously set the resolution of the output composite image according to the target task.

[0011] Step S2: Input the synthesized image into the structure-adjusted teacher model and student model respectively;

[0012] Preferably, the synthetic images are input into the structure-adjusted teacher model and student model respectively, specifically including:

[0013] Step S21: Modify the structure of the teacher model and the student model by replacing the original fully connected layers with 1×1 convolutional layers. The teacher model keeps its parameters frozen during training, while the student model continuously updates its parameters during distillation training.

[0014] Step S22: Replace the fully connected layers of the teacher model and the student model with one-dimensional convolutional layers respectively, and add a global average pooling operation after the output of the convolutional layer to support the direct extraction of class activation maps;

[0015] Step S23: Input the generated synthetic image into the teacher model and student model respectively, and obtain the intermediate features and prediction results of the teacher model and student model respectively.

[0016] Step S3: Generate corresponding class activation maps for the teacher and student models after structural adjustment, and use the class activation maps to reflect the areas of interest of the model in the spatial dimension;

[0017] Preferably, generating a class activation graph includes:

[0018] Step S31: Calculate the class activation maps of the teacher model and the student model based on the convolutional feature maps and the weights of the classification convolutional kernels;

[0019] Step S32: Perform bilinear interpolation or deconvolution upsampling on the class activation map to maintain the same resolution as the input image;

[0020] Step S33: Normalize the obtained class activation map to highlight the main regions of interest.

[0021] Step S4: Input the teacher class activation graph and the student class activation graph into the scale mapping network to perform cross-scale scaling on the class activation graph;

[0022] Preferably, cross-scale scaling includes:

[0023] Step S41: Input the teacher class activation map into the pre-trained scale mapping network and map it to a fixed scale representation. The scale mapping network adopts a lightweight convolution-deconvolution symmetric structure.

[0024] Step S42: Input the student class activation graph into the same network to maintain consistency in scale between teachers and students;

[0025] Step S43: Output the cross-scale aligned class activation map;

[0026] Step S44: The scale mapping network is trained in a separate pre-training phase. The training objective is to minimize the difference between activation maps of different resolution classes and to calculate the scale mapping loss function.

[0027] Step S5: Regularize the class activation graphs scaled by the scale mapping network to obtain standardized teacher and student class activation graphs.

[0028] Preferably, the regularization process includes:

[0029] Step S51: Normalize the teacher activation graph and the student activation graph to the L2 norm, and normalize them to the [0,1] interval;

[0030] Step S52: Standardize the normalized class activation map so that its mean is 0 and its variance is 1. Use L2 regularization to enhance robustness and suppress interference from noisy regions.

[0031] Step S53: Output the standardized activation graphs for the teacher and student classes to ensure that the outputs of different models are comparable at the same scale.

[0032] Step S6: Calculate the mean squared error loss based on the standardized teacher class activation graph and student class activation graph, and calculate the output cross-entropy loss between teachers and students to perform knowledge transfer from the teacher model to the student model;

[0033] Preferably, the implementation method of step S6 specifically includes:

[0034] Step S61: Align the class activation maps of teachers and students, and calculate the difference pixel by pixel;

[0035] Step S62: Square the difference and calculate the mean to obtain the mean squared error loss;

[0036] Step S63: Calculate the output cross-entropy loss between the teacher model and the student model;

[0037] Step S64: Integrate the distillation loss of the student model;

[0038] Step S65: Use the student model distillation loss to update the parameters of the student model through backpropagation.

[0039] Step S7: Optimize the generator by combining information entropy loss, one-hot loss, and activation loss.

[0040] Preferably, optimizing the generator includes:

[0041] Step S71: Calculate the unique heat loss. The unique heat loss is used to maintain the class discriminativeness of the samples and ensure that the generated images can be correctly recognized by the teacher model.

[0042] Step S72: Calculate the activation loss, which is used to enhance the activation strength of the generated samples in the feature space, making them closer to the real data distribution;

[0043] Step S73: Define information entropy loss. Information entropy loss is used to avoid the generated sample distribution being too homogeneous and to improve class coverage.

[0044] Step S74: Synthesize the generator loss function;

[0045] Step S75: Output the optimization objective of the final total loss function.

[0046] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0047] (1) The method of the present invention enables efficient knowledge transfer even in scenarios without real training data, reducing the cost of data access and privacy compliance.

[0048] (2) The method of the present invention uses class activation graph as the main supervision signal, so that the distillation process not only transmits the prediction ability, but also transmits the inherent discrimination logic of the "model interest region", thereby improving the interpretability and robustness of the student model.

[0049] (3) The method of the present invention introduces a scale mapping network as a cross-scale mapping module, which effectively replaces simple pooling / interpolation operations, reduces spatial information loss, and improves the accuracy and stability of class activation graph alignment.

[0050] (4) The method of the present invention has strong compatibility with the architecture of teachers and students (supports ResNet, WideresNet, MobileNet, ShuffleNet, etc.) and can maintain good performance in cross-structure distillation scenarios. Attached Figure Description

[0051] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0052] Figure 1 This is a flowchart of the network architecture provided in an embodiment of the present invention.

[0053] Figure 2 This is a visual comparison diagram of class activation graphs provided in an embodiment of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Example 1:

[0056] In this embodiment, combined with Figure 1 As shown, the present invention provides the following technical solution: an interpretable data-free knowledge distillation method based on class activation graphs, comprising the following operational steps:

[0057] Step S1: Input random noise into the generator to generate a synthetic image with a distribution similar to the target dataset, providing pseudo-sample input for the subsequent distillation process.

[0058] In this embodiment, generating a synthetic image with a distribution similar to the target dataset specifically includes:

[0059] S11: Sample the noise vector z from the standard Gaussian distribution or uniform distribution as the input to the generator. The dimension of z is generally set to 100 to 512 dimensions to ensure that the latent space has sufficient expressive power.

[0060] S12: The generator employs a multi-layer convolutional and deconvolutional structure, combined with batch normalization (BatchNorm) and non-linear activation functions (such as ReLU and LeakyReLU) to improve training stability. Its overall structure can be referenced from DCGAN (DeepConvolutionalGAN), but the Tanh function is used in the last layer to normalize the output pixels to the [-1,1] interval;

[0061] For example, the generator includes: a fully connected layer, several convolutional blocks, two upsampling operations, and a convolutional output block. The fully connected layer maps the input noise vector to an initial feature map. The convolutional blocks consist of convolutional layers, batch normalization layers, and LeakyReLU activation functions. The two upsampling operations are used to progressively increase the spatial resolution of the feature map. The convolutional output block contains convolutional layers, Tanh activation functions, and normalization layers, used to generate a synthetic image.

[0062] S13: The resolution of the output synthesized image can be set according to the target task, such as 28×28 (MNIST) or 32×32 (CIFAR). The generator output not only needs to be visually reasonable, but also needs to have discriminative features that can be recognized by the teacher model.

[0063] Step S2: Input the synthesized image into the structure-adjusted teacher model and student model respectively.

[0064] In this embodiment, the synthetic images are input into the structure-adjusted teacher and student models, respectively, specifically including:

[0065] Step S21: Modify the structure of the teacher model and the student model by replacing the original fully connected layers with 1×1 convolutional layers. The teacher model keeps its parameters frozen during training, while the student model continuously updates its parameters during distillation training. The teacher model can choose high-performance architectures such as ResNet and WideResNet, while the student model can choose lightweight architectures such as MobileNet and ShuffleNet to improve deployment efficiency.

[0066] In this embodiment, the teacher model is a deep convolutional neural network pre-trained on the target dataset, and the student model is a lightweight convolutional neural network.

[0067] Step S22: Replace the fully connected layers of the teacher model and the student model with one-dimensional convolutional layers respectively, and add a global average pooling operation after the output of the convolutional layer to support the direct extraction of class activation maps;

[0068] Step S23: Input the synthesized image generated in step S1 into the teacher model and the student model respectively to obtain their respective intermediate features and prediction results.

[0069] Step S3: Generate corresponding class activation maps for the teacher and student models after structural adjustment. The class activation maps reflect the areas of interest of the model in the spatial dimension.

[0070] In this embodiment, the specific steps for generating the class activation graph are as follows:

[0071] Step S31: Calculate the class activation maps for the teacher and student models based on the convolutional feature maps and the classification convolutional kernel weights. Optionally, class activation maps for all categories can be generated to support multi-class alignment, not just the top-1 category. In visualization analysis, the class activation maps for teachers and students can be compared to demonstrate the interpretability of knowledge transfer.

[0072] Step S32: Perform bilinear interpolation or deconvolution upsampling on the class activation map to maintain the same resolution as the input image.

[0073] Step S33: Normalize the obtained class activation map to highlight the main regions of interest.

[0074] Step S4: Input the teacher class activation map and the student class activation map into the scale mapping network, and perform cross-scale scaling of the class activation map to achieve a unified representation under different resolutions.

[0075] In this embodiment, cross-scale scaling specifically includes:

[0076] Step S41: Input the teacher class activation map into the pre-trained scale mapping network and map it to a fixed scale representation. The scale mapping network adopts a lightweight convolution-deconvolution symmetric structure, similar to a sub-module of U-Net, to realize the mapping of feature maps between multiple resolutions. The input is a CAM of different sizes, and the output is uniformly fixed resolution (such as 32×32 or 64×64).

[0077] In this embodiment, the scale mapping network includes: an input convolutional layer, several downsampling and upsampling convolutional layers, and an output convolutional layer. The input convolutional layer receives class activation map features, the several downsampling and upsampling convolutional layers perform cross-scale mapping on the class activation maps, and the output convolutional layer maps class activation maps of different scales to a uniform spatial resolution.

[0078] Step S42: Input the student class activation map into the same network to maintain scale consistency between teachers and students. The input class activation map is first compressed by a 1×1 convolution to reduce computation. It is then unified to the target resolution through upsampling or downsampling operations (such as bilinear interpolation or deconvolution). The final output is processed by a normalization layer (such as Sigmoid) to ensure numerical stability.

[0079] Step S43: Output the cross-scale aligned class activation map to provide input for subsequent distillation loss calculation.

[0080] Step S44: The scale mapping network can be trained in a separate pre-training stage, with the training objective being to minimize the differences between activation maps of different resolutions.

[0081]

[0082] Among them, L SDM SDM(·) represents the scale-mapping loss function, and SDM(·) represents the mapping operation of the scale-mapping network, used to map low-resolution class activation maps to high-resolution spaces. CAM low CAM represents a low-resolution class activation map. high Represents a high-resolution class activation map. The L2 norm represents the squared difference during the distillation stage, used to measure the Euclidean distance between two class activation maps.

[0083] The parameters of the frozen scale mapping network are used only as a scale alignment module to ensure that the class activation maps of teachers and students are comparable at the same spatial scale.

[0084] Step S5: Perform regularization on the class activation graphs scaled by the scale mapping network to obtain standardized teacher and student class activation graphs.

[0085] In this embodiment, the regularization process includes:

[0086] Step S51: Perform L2 norm normalization on the activation graphs of the teacher and student classes respectively, normalizing them to the [0,1] interval. A small constant is used to prevent division by zero errors during normalization while maintaining numerical stability.

[0087] Step S52: Standardize the normalized class activation map so that its mean is 0 and its variance is 1. L2 regularization can be used to enhance robustness and suppress interference from noisy regions.

[0088] Step S53: Obtain the standardized activation graphs of the teacher class and the student class to ensure that the outputs of different models can be compared at the same scale.

[0089] Step S6: Calculate the mean squared error loss based on the standardized teacher class activation graph and student class activation graph, and calculate the output cross-entropy loss between teachers and students to realize knowledge transfer from the teacher model to the student model.

[0090] In this embodiment, the specific implementation steps are as follows:

[0091] Step S61: Align the class activation maps of teachers and students, and calculate the difference pixel by pixel.

[0092] In this embodiment, for the input sample x, the teacher model output class activation map is as follows:

[0093]

[0094] Among them, A T (x) represents the set of class activation maps generated by the teacher model on the input image x, where C is the number of classes, W represents the width of the class activation map, and H represents the height of the class activation map. That is, each input image will correspond to C two-dimensional class activation maps, and each map corresponds to a discrimination region of a class.

[0095] The activation graph of the student model output class is as follows:

[0096]

[0097] Among them, A S (x) represents the set of class activation maps generated by the student model on the input image x.

[0098] Step S62: Square the difference and calculate the mean to obtain the mean squared error loss.

[0099] In this embodiment, the CAM alignment loss is defined as:

[0100]

[0101] Among them, L CAM This represents the class activation graph alignment loss, where c represents the class index, and its value ranges from 1 to c to C. Let represent the class activation graph of class c generated by the student model on input x. Let φ(·) represent the class activation map of class c generated by the teacher model on input x, and let φ(·) represent the spatial downsampling function, which measures the difference between the teacher and the student in spatial attention distribution.

[0102] Step S63: Calculate the output cross-entropy loss between the teacher and the student.

[0103] In this embodiment, a cross-entropy loss term L based on pseudo-labels can also be introduced. CE The calculation method is as follows:

[0104] L CE =H CE (S(x),c * )

[0105] Among them, L CE It is the cross-entropy loss output by the teacher model and the student model, H. CE (·,·) represents the standard cross-entropy loss, c *The pseudo-label is the class predicted by the teacher model, i.e., the class with the highest probability value is taken as the pseudo-label. * =argmaxT(x) c , T(x) c S(x) is the teacher model's predicted probability for category c, the student model's predicted output for input x, and a softmax probability distribution vector.

[0106] In this embodiment, the core knowledge distillation process of the present invention can independently achieve effective knowledge transfer and interpretability enhancement based on class activation graph loss, while the cross-entropy loss is only an optional supplement, which can further improve the classification accuracy of the student model in some embodiments.

[0107] Step S64: Integrate the student model distillation loss.

[0108] In this embodiment, the loss of the final student model distillation is:

[0109] L S =λL CAM

[0110] Among them, L S Let λ represent the loss function of the student model, and λ be the class activation graph loss L. CAM The weight hyperparameter.

[0111] As an optional embodiment, in step S6, a pseudo-label cross-entropy loss can be introduced on top of the class activation graph loss. In this case, the total distillation loss of the student model is:

[0112] L S =L CE +λL CAM

[0113] Classification performance can be improved by introducing pseudo-label cross-entropy loss.

[0114] Step S65: Use the loss to update the parameters of the student model through backpropagation, thereby gradually approximating the teacher's attention pattern.

[0115] Step S7: Optimize the generator by combining information entropy loss, one-hot loss, and activation loss.

[0116] In this embodiment, optimizing the generator includes:

[0117] Step S71: Calculate the unique heat loss.

[0118] In this embodiment, based on the teacher model's prediction results for the synthesized image, a pseudo-label t = argmaxT(x) is constructed, where T(x) is the teacher model's prediction for image x. The cross-entropy loss between the prediction distribution and the pseudo-label is calculated.

[0119]

[0120] Among them, L oh Let T(x) be the unique heat loss function. i ) is the softmax output of the i-th image in the teacher model, t i is the one-hot pseudo-label of the i-th image in the teacher model, and n represents the number of generated samples in a batch. This loss constrains the generator output image to have high confidence and clear class attribution.

[0121] Step S72: Calculate the activation loss.

[0122] In this embodiment, the feature vector before the last convolutional layer of the teacher model is extracted, and its average l1 norm is calculated:

[0123]

[0124] Among them, L a To activate the loss function, Let ||·||1 represent the high-level feature representation of the teacher model on the i-th generated sample, and let ||·||1 represent the l1 norm of the vector. This loss enables the image synthesized by the generator to stimulate the high-level semantic features of the teacher network, and constrains the feature representation of the generated sample to be consistent with the distribution of real data, thereby improving interpretability.

[0125] Step S73: Define information entropy loss.

[0126] In this embodiment, The teacher model generates images based on these images. The output probability vector is the softmax prediction for C categories, and the batch average distribution of the teacher's output is calculated.

[0127]

[0128] Define the information entropy loss as:

[0129]

[0130] Among them, L ie This represents the information entropy loss function. Let H(·) represent the component of the average probability vector in class c, and let H(·) represent the Shannon entropy function, which is used to maintain class balance and prevent the generator from collapsing into a minority of classes.

[0131] Step S74: Synthesize the generator loss function.

[0132] In this embodiment, the optimization objective of the final generator is:

[0133] LG =L oh +αL a +βL ie

[0134] Among them, L G The generator loss function is represented by α and β, which are weighting coefficients used to balance the importance of different loss terms. The parameters α and β can be determined through cross-validation, or by using preset weights or dynamically adjusting the weights. The combination of the three loss functions ensures the discriminativeness, semantic richness, and class diversity of the generated samples.

[0135] Step S75: Output the optimization objective of the final total loss function.

[0136] When introducing a pseudo-label-based cross-entropy loss term, the optimization objective of the final total loss function is:

[0137] L total =L G +L S =(L oh +αL a +βL ie )+λL CAM

[0138] Among them, L total This represents the overall loss function.

[0139] As an optional embodiment, in step S6, a pseudo-label cross-entropy loss can be introduced on top of the class activation graph loss. The optimization objective of the total loss function is then as follows:

[0140] L total =L G +L S =(L oh +αL a +βL ie )+L CE +λL CAM

[0141] For example, the experimental results of this embodiment are compared with those of the prior art as follows:

[0142] 1. The class activation graph distribution effects of the teacher model, the student model distilled by the method of this invention, and the comparative student model were compared, such as... Figure 2 As shown, the class activation graph distributions of the teacher model, the distilled student model, and the control student model are significantly different. The student model is able to gradually learn the attention regions of the teacher model, proving the interpretability of class activation graph distillation.

[0143] 2. The accuracy of the method of this invention, DAFL, RDSKD, and DFAD on basic datasets such as MNIST, SVHN, and CIFAR-10 was tested. The specific data results are shown in the table below.

[0144]

[0145]

[0146] 3. The accuracy of the method of this invention, DAFL, RDSKD, and DFAD on the Caltech101 dataset was tested, comparing the teacher and student models under the same and different architectures. Detailed results are shown in the table below.

[0147]

[0148] In this embodiment, the method of the present invention combines data-free knowledge distillation with interpretability based on class activation graphs. Furthermore, a separate scaling network is used to scale the acquired class activation graphs. This solves the problem of whether class activation graphs can be used in data-free knowledge distillation to guide student models to learn the teacher model's ability to classify input images.

[0149] Meanwhile, the knowledge transferred between the teacher and student models in this invention differs from existing methods. Knowledge distillation is achieved by having the student model learn the teacher model's spatial ability to identify objects in images by identifying specific regions. Furthermore, in terms of interpretability based on class activation graphs, this invention, to fully utilize spatial information, does not employ average pooling as in traditional methods. Instead, it pre-trains a scale mapping network to ensure that the parameters during the scaling process of the class activation graph are updated accordingly. This emphasizes using different knowledge as guidance in the data-free knowledge distillation process, verifying the feasibility of transferring class activation graphs as knowledge transfer in data-free knowledge distillation.

[0150] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0151] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An interpretable data-free knowledge distillation method based on class activation map, characterized in that: The method comprises the following steps: Step S1: inputting random noise into a generator to generate a synthetic image similar to the distribution of a target data set; Step S2: inputting the synthetic image into a structurally adjusted teacher model and a student model respectively; Step S3: generating corresponding class activation maps through the structurally adjusted teacher model and student model respectively, and using the class activation maps to reflect the attention area of the model in the spatial dimension; Step S4: inputting the teacher class activation map and the student class activation map into a scale mapping network to perform cross-scale scaling on the class activation maps; Step S5: performing regularization processing on the class activation maps scaled by the scale mapping network to obtain standardized teacher class activation maps and student class activation maps; Step S6: calculating the mean square error loss based on the standardized teacher class activation maps and student class activation maps, and calculating the output cross-entropy loss between the teacher and the student to perform knowledge transfer from the teacher model to the student model; Step S7: combining the information entropy loss, the one-hot loss and the activation loss to optimize the generator.

2. The activation graph-based explainability data-free knowledge distillation method according to claim 1, characterized in that: The generation of a synthetic image similar to the distribution of a target data set comprises: S11: sampling a noise vector z from a standard Gaussian distribution or a uniform distribution as the input of the generator; The generator comprises a fully connected layer, a plurality of convolutional blocks, two upsampling operations and a convolutional output block; S12: the generator adopts a multi-layer convolution and deconvolution structure, combines batch normalization and a nonlinear activation function, uses a Tanh function in the last convolutional layer to normalize the output pixels to the interval [-1, 1], and synchronously sets the resolution of the output synthetic image according to the target task setting. The input of the synthetic image into the structurally adjusted teacher model and student model comprises:

3. The activation graph-based explainable data-free knowledge distillation method according to claim 2, characterized in that: Step S21: structurally modifying the teacher model and the student model, replacing the original fully connected layer with a 1x1 convolutional layer, keeping the parameters of the teacher model frozen during the training process, and constantly updating the parameters of the student model during the distillation training; The teacher model is a deep convolutional neural network pre-trained on a target data set; The student model is a lightweight convolutional neural network; Step S22: replacing the fully connected layer of the teacher model and the student model with a one-dimensional convolutional layer, and adding a global average pooling operation after the convolutional layer output to support direct extraction of the class activation map; Step S23: inputting the generated synthetic image into the teacher model and the student model respectively to obtain the intermediate features and the prediction results of the teacher model and the student model. The generation of the class activation map comprises:

4. The activation graph-based explainable data-free knowledge distillation method according to claim 3, characterized in that: Step S31: calculating the class activation maps of the teacher model and the student model according to the convolutional feature map and the classification convolutional kernel weight; Step S32: performing bilinear interpolation or deconvolution upsampling on the class activation map to make it have the same resolution as the input image; Step S33: normalizing the obtained class activation map to highlight the main attention area. The cross-scale scaling comprises:

5. The activation graph-based explainability data-free knowledge distillation method according to claim 4, characterized in that: Step S41: inputting the teacher class activation map into a pre-trained scale mapping network to map it to a fixed scale representation, and the scale mapping network adopts a lightweight convolution-deconvolution symmetric structure; The scale mapping network comprises an input convolutional layer, a plurality of down-sampling and up-sampling convolutional layers, and an output convolutional layer; ​ Step S42: input the student class activation map into the same network to keep the consistency of the teacher and student in the scale; Step S43: output the class activation map after cross-scale alignment; Step S44: the scale mapping network is trained in a separate pre-training stage, and the training target is to minimize the difference between different resolution class activation maps, and the scale mapping loss function is calculated.

6. The activation graph-based explainable data-free knowledge distillation method according to claim 5, characterized in that: The scale mapping loss function calculation formula is as follows: wherein L SDM denotes the scale mapping loss function, SDM(·) denotes the mapping operation of the scale mapping network, which is used to map the low-resolution class activation map to the high-resolution space, CAM low denotes the low-resolution class activation map, CAM high denotes the high-resolution class activation map, denotes the two-norm of squaring the difference value in the distillation stage, which is used to measure the Euclidean distance between the two class activation maps.

7. The activation graph-based explainability data-free knowledge distillation method according to claim 6, characterized in that: The regularization processing includes: Step S51: the teacher class activation map and the student class activation map are respectively normalized by L2 norm, and are normalized to the [0, 1] interval; Step S52: the normalized class activation map is standardized to have a mean of 0 and a variance of 1, and L2 regularization is used to enhance robustness and suppress noise region interference; Step S53: output the standardized teacher class activation map and student class activation map to ensure that the outputs of different models are comparable at the same scale.

8. The activation graph-based explainable data-free knowledge distillation method according to claim 7, characterized in that: Step S6 specifically includes: Step S61: align the class activation maps of the teacher and the student, and calculate the difference pixel by pixel; Step S62: take the square of the difference and calculate the mean to obtain the mean square error loss; The mean square error loss is calculated pixel by pixel, which constrains the student class activation map to be consistent with the teacher class activation map in spatial distribution; Step S63: calculate the output cross-entropy loss between the teacher model and the student model; Step S64: integrate the student model distillation loss; Step S65: update the parameters of the student model by back propagation using the student model distillation loss.

9. The activation graph-based explainable data-free knowledge distillation method according to claim 8, characterized in that: The integrated student model distillation loss includes: introducing a pseudo-label cross-entropy loss based on the class activation map loss.

10. The activation graph-based explainable data-free knowledge distillation method according to claim 9, characterized in that: The optimization of the generator includes: Step S71: calculate the one-hot loss; the one-hot loss is achieved by maximizing the cross-entropy of the teacher predicted class, which is used to maintain the class distinguishability of the generated image; Step S72: calculate the activation loss; the activation loss is achieved by minimizing the mean of the absolute value of the teacher class activation map, which is used to constrain the effectiveness of the generated sample in the model feature space; Step S73: define the information entropy loss; the information entropy loss is achieved by averaging the class distribution output by the teacher model and then calculating the entropy value; Step S74: integrate the generator loss function; Step S75: output the optimization target of the final total loss function.

Images