A model perturbation-based knowledge distillation method and storage medium
By employing a strategy of randomly discarding features and injecting noise into class activation graphs, multi-teacher knowledge distillation is achieved based on a single teacher model. This solves the problem of multi-teacher knowledge distillation relying on multiple models, reduces costs, and improves model performance on resource-constrained platforms.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TOWER CO LTD
- Filing Date
- 2025-11-05
- Publication Date
- 2026-07-21
AI Technical Summary
Multi-teacher knowledge distillation methods rely on multiple independent pre-trained teacher models, resulting in high computational and storage costs and making them difficult to deploy effectively on resource-constrained platforms.
We employ a knowledge distillation method based on model perturbation, which simulates the diverse outputs of a multi-teacher model by randomly discarding features and injecting noise into the class activation graph, and then uses a single pre-trained teacher model for knowledge transfer.
It significantly reduces computing and storage costs, improves resource utilization, enhances generalization ability and classification accuracy under complex data distributions, and ensures the stability of knowledge transfer and model performance.
Smart Images
Figure CN121683875B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning technology, and in particular to a knowledge distillation method and storage medium based on model perturbation. Background Technology
[0002] Deep learning, as a core technology in the field of artificial intelligence, has not only achieved success in tasks such as object detection and text understanding, but has also brought new solutions to engineering needs such as information processing and environmental perception. However, these high-performance deep learning models have also brought about increasingly large parameter numbers and computing power requirements. In order to meet the deployment requirements of resource-constrained platforms, such as mobile terminals and satellites, various model compression methods have emerged.
[0003] Knowledge distillation, as an effective model compression strategy, transfers knowledge from a large-scale pre-trained "teacher model" to a lightweight "student model," significantly reducing model size, accelerating inference speed, and remarkably improving generalization ability on specific tasks. Most knowledge distillation methods achieve knowledge transfer by aligning the outputs or intermediate features of the teacher and student models; however, such methods are often limited by the single perspective of the teacher model, making it difficult to cover the potential diversity in real-world data distributions. To alleviate this problem, multi-teacher knowledge distillation proposes integrating the outputs of multiple pre-trained teacher models and designing a knowledge fusion module to eliminate individual biases and effectively extract diverse knowledge. Although multi-teacher knowledge distillation has achieved good results, its reliance on multiple independent pre-trained teacher models limits its practical application.
[0004] Therefore, how to alleviate the dependence of multi-teacher knowledge distillation on multiple independent pre-trained teacher models, thereby improving resource utilization, is an urgent problem to be solved by those skilled in the art. Summary of the Invention
[0005] The purpose of this invention is to provide a knowledge distillation method and storage medium based on model perturbation, which can alleviate the dependence of multi-teacher knowledge distillation on multiple independent pre-trained teacher models and improve resource utilization.
[0006] To achieve the above objectives, the present invention provides the following technical solution: A knowledge distillation method based on model perturbation includes: Step S1: Obtain the training set and test set, as well as the pre-trained teacher model and the students to be distilled. The model inputs sample data from the training set into the backbone networks of the teacher and student models respectively, and extracts teacher feature maps. F T Student Feature Map F S ; Step S2: Copy the teacher feature map F T , to obtain the total Z +1 teacher profile F T Among them Z Teacher Feature Chart F T Perform a feature random discarding operation at the channel level to obtain Z initial perturbed teacher feature maps. And an undisturbed teacher profile F T ; Step S3, the... Z Initial perturbation teacher feature map The unperturbed teacher feature map F T Input the classification header of the teacher model respectively, and obtain the mapping. Z Initial perturbation teacher class activation graph L T1 ~ L TZ And the activation graph of the non-disruptive teacher class L T0 The student feature map F S Input the class header of the student model to obtain the student class activation map. L S ; Step S4, for the Z Initial perturbation teacher class activation graph L T1 ~ L TZ Inject constrained Gaussian noise separately to generate Z Disturbance teacher-induced noisy activation graph ; Step S5, Calculation Z +1 teacher-class activation image and the aforementioned student-class activation image L S Distillation loss between L KD , wherein Z +1 teacher-related activation image, including a non-disruptive teacher-related activation image. L T0 and Z Disturbance teacher-induced noisy activation graph ; Step S6: Activate the graph of the student class. L S Perform global average pooling to output logits Value, the logits Value after softmaxAfter processing by the function, the classification prediction result is obtained, and the cross-entropy loss between the classification prediction result and the true label is calculated. L cls ; Step S7: Utilize the distillation loss L KD and the cross-entropy loss L cls Weighted sum of total loss L sum And by minimizing the total loss L sum To update the parameters of the student model; Step S8: Repeat steps S1-S7 until the training set has been traversed, and use the test set to evaluate the performance of the student model and obtain the optimal student model parameters.
[0007] Based on the same inventive concept, the present invention also provides a computer storage medium storing computer-executable instructions, which, when executed, implement the aforementioned knowledge distillation method based on model perturbation.
[0008] The technical effects and advantages of this invention are as follows: (1) By using two cooperative perturbation strategies, namely feature random discarding strategy and class activation graph noise injection, only a single pre-trained teacher model is needed to simulate the diverse output distribution of multi-teacher models, thereby alleviating the dependence of traditional multi-teacher knowledge distillation on multiple independent pre-trained teacher models, significantly reducing computational and storage costs and deployment threshold, making this method have outstanding practical value in resource-constrained scenarios such as satellite networks and mobile terminals. (2) Traditional knowledge distillation methods only use a single teacher model output as supervision. In this method, the distillation loss considers both the original output of the teacher model (providing accurate anchor points) and multiple perturbation versions of the output (providing diverse knowledge), ensuring that the student model learns more robust feature representations and effectively improving the generalization ability and final classification accuracy under complex data distributions. (3) The student model performs knowledge distillation on the class activation layer of the teacher model, enabling it to deeply learn the teacher model's attention mechanism and discriminative features for key image regions. Simultaneously, it preserves the channel's performance through constrained noise injection. logits The value remains unchanged, ensuring the stability of the knowledge transfer process and avoiding the introduction of erroneous signals due to disturbances, thus ensuring model performance while compressing the model.
[0009] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 This is a schematic diagram illustrating the general principle of the knowledge distillation method. Figure 2 This is a schematic diagram illustrating the principle of the knowledge distillation method based on model perturbation in this invention. Figure 3 This is a flowchart illustrating the knowledge distillation method based on model perturbation implemented in this invention. Detailed Implementation
[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0013] Figure 1 The diagram illustrates the general principle of the knowledge distillation method, showcasing the general process of a knowledge distillation method based on class activation graphs. Figure 1 First, the backbone networks of the teacher model and the student model extract teacher feature maps respectively. F T Student Feature Map F S Then, the teacher class activation graph is generated through their respective class heads. L T Student-related activation graph L S Knowledge distillation is performed, and its core lies in designing a loss function to force the student activation map to approximate the teacher activation map in terms of spatial detail and class discriminability, rather than simply aligning the final output probabilities. Finally, the student activation map is transformed into output probabilities through global average pooling and compared with the true labels. Calculate cross-entropy loss L clsDuring training, the distillation loss of the teacher-student class activation graph is jointly optimized. L KD With cross-entropy loss L cls This enables the effective transfer of knowledge from the teacher model to the student model.
[0014] Building upon the above, to alleviate the dependence of multi-teacher knowledge distillation on multiple independent pre-trained teacher models, such as... Figure 2 and Figure 3 As shown, this invention provides a knowledge distillation method based on model perturbation, which includes the following steps: Step S1: Obtain the training set and test set, as well as the pre-trained teacher model and the student model to be distilled. Input the training sample data, i.e., the sample data of the training set, into the backbone network of the teacher model and the student model respectively, and extract the teacher feature map. F T Student Feature Map F S ; Step S2: Copy the teacher feature map F T , to obtain the total Z +1 teacher profile F T Among them Z Teacher Feature Chart F T Perform a feature random discarding operation at the channel level to obtain Z initial perturbed teacher feature maps. And an undisturbed teacher profile F T ; Step S3, the... Z Initial perturbation teacher feature map The unperturbed teacher feature map F T Input the classification header of the teacher model respectively, and obtain the mapping. Z Initial perturbation teacher class activation graph L T1 ~ L TZ And the activation graph of the non-disruptive teacher class L T0 The student feature map F S Input the class header of the student model to obtain the student class activation map. L S ; Step S4, for the Z Initial perturbation teacher class activation graph L T1 ~ LTZ Inject constrained Gaussian noise separately to generate Z Disturbance teacher-induced noisy activation graph ; Step S5, Calculation Z +1 teacher-class activation image and the aforementioned student-class activation image L S Distillation loss between L KD , wherein Z +1 teacher-related activation image, including a non-disruptive teacher-related activation image. L T0 and Z Disturbance teacher-induced noisy activation graph ; Step S6: Activate the graph of the student class. L S Perform global average pooling to output logits Value, the logits Value after softmax After processing by the function, the classification prediction result is obtained, and the cross-entropy loss between the classification prediction result and the true label is calculated. L cls ; Step S7: Utilize the distillation loss L KD and the cross-entropy loss L cls Weighted sum of total loss L sum And by minimizing the total loss L sum To update the parameters of the student model; Step S8: Repeat steps S1-S7 until the training set has been traversed, and use the test set to evaluate the performance of the student model and obtain the optimal student model parameters.
[0015] This invention proposes a model perturbation strategy based on random feature discarding. After obtaining the feature map of the teacher model, it performs multiple random discarding operations to simulate a random sampling process and mine the feature map distribution of multiple sub-models. It also proposes a model perturbation strategy based on class activation map noise injection. During the class activation map generation stage of the teacher model, Gaussian noise is injected into the teacher class activation map, while simultaneously constraining the pooled... logits While keeping the values unchanged, the diversity of activation graph knowledge in sub-model classes is enhanced. The two model perturbation strategies complement each other, ultimately forming a knowledge distillation method based on model perturbation, which alleviates the dependence of multi-teacher knowledge distillation on multiple independent pre-trained teacher models.
[0016] In step S1, the pre-trained teacher model and the student model to be distilled are convolutional neural network models, both of which contain a backbone network and a classification head. The backbone network is responsible for extracting feature representations, and the classification head is responsible for mapping the feature representations to the category space.
[0017] In step S1, the training sample data is input into the backbone networks of the teacher model and the student model respectively to extract the teacher feature map. F T Student Feature Map F S Specifically, it includes: Step S11: Input the training sample data into the backbone network of the teacher model, and extract the feature map of the last layer of the teacher model as the teacher feature map. F T :
[0018] Teacher Feature Map F T It is a three-dimensional tensor, in which, C T This represents the total number of channels in the teacher feature map. H T , W T Let these represent the height and width of the teacher feature map, respectively, and the first and second dimensions of the teacher feature map. i Each channel is located in The eigenvalues are represented as ,exist middle, , , ; Step S12: Input the training sample data into the backbone network of the student model, and extract the feature map of the last layer of the student model as the student feature map. F S :
[0019] in, C S Student feature map F S The number of channels, H S , W S Let these represent the height and width of the student feature map, respectively, and the first and second dimensions of the student feature map. i Each channel is located in The eigenvalues are represented as ,exist middle, , , .
[0020] According to a preferred embodiment, in step S2, for Z Teacher Feature Chart F T Perform a feature random discard operation at the channel level to obtain Z Initial perturbation teacher feature map Specifically, it includes: right Z Teacher Feature Chart F T Bernoulli distribution was used to analyze the feature maps of each teacher. F T Each channel is sampled independently using a binary mask to obtain a set of binary masks. , No. i Binary mask for each channel m i It conforms to the Bernoulli distribution. , p This represents the channel drop probability.
[0021] The perturbed teacher feature map obtained after the feature random discarding operation Represented as:
[0022] in, Represents the teacher feature map F T Perform successive multiplication operations with the binary mask set M, where, Teacher feature map F T The i Feature map of each channel.
[0023] By employing the aforementioned feature random discarding strategy, the process of randomly sampling the teacher model parameters is simulated, enabling the model to obtain the output feature map of the sub-model in a single forward propagation. .
[0024] According to a preferred embodiment, step S3 specifically includes: Step S31, the... Z Initial perturbation teacher feature map The classification headers of the teacher model are input sequentially, and the mapping is obtained. Z Initial perturbation teacher class activation graph L T1 ~ L TZ ; Step S32: Transfer the unperturbed teacher feature map F TInput the classifier header of the teacher model to obtain an unperturbed teacher class activation graph. L T0 ;
[0025] L Tk Indicates the first k There are perturbation versions of the teacher class activation graph, where N is the number of categories in the teacher model, and each channel corresponds to the activation graph of one category. , k =0 indicates no disturbance to the teacher class activation graph. L T0 ; No. k A disturbed version of the teacher class activation image L Tk The i Each channel is located at ( x , y The activation value of ) is represented as ,exist middle, , , ; Step S33: Transfer the student feature map F S Input the class header of the student model to obtain the student class activation map. L S ;
[0026] Where N is the number of classification categories in the student model, and each channel corresponds to the activation map of one category; The student class activation graph L S The i Each channel is located at ( x , y The activation value of ) is represented as ,exist middle, , , .
[0027] According to a preferred embodiment, step S4 specifically includes: Step S41: Select a graph with the same dimensions as the perturbation teacher class activation graph, a mean of 0, and a variance of . Gaussian noise As noise to be injected; Step S42: Inject Gaussian noise into the respective... Z Initial perturbation teacher class activation graphL T1 ~ L TZ In the middle, we obtained Z Disturbance teacher-induced noisy activation graph ;
[0028] in, Indicates the first k A perturbed version of the teacher model with noisy class activation graphs, and .
[0029] According to a preferred embodiment, step S4 further includes: Step S43: Gaussian noise to be injected Apply constraints.
[0030] Among them, Gaussian noise The i Each channel is at a random location. noise value Replace it with the negative sum of other noise values in the same channel, so that the sum of the noise values in the entire channel is zero. This ensures that the noise activation map of the perturbed teacher is properly represented in each channel after global average pooling. logits The value remains unchanged;
[0031] in, The first Gaussian noise i In each channel, except for position The noise value at any other location outside of the specified range.
[0032] In deep learning, logits The value refers to the raw output value of the last layer (usually a fully connected layer) when the model is performing a classification task. softmax or sigmoid Used before normalization.
[0033] By employing the aforementioned class activation graph noise injection strategy, we can further explore the diversity of knowledge in the sub-model class activation graphs, thereby forcing the student model to maintain prediction stability within the neighborhood (noise perturbation range) of the teacher's class activation graph.
[0034] To further enhance the predictive consistency of the teacher-student model's class activation graph neighborhood, the feature random discarding operation and Gaussian noise injection into the class activation graph involved in this embodiment of the invention are random. In a single inference, the perturbation strategy is applied multiple times to the teacher model feature graph to be perturbed, resulting in multiple perturbed teacher class activation graphs. According to a preferred embodiment, in step S5, the... Z+1 teacher-class activation map, including the aforementioned noisy teacher-class activation map. L T0 and stated Z Disturbance teacher-induced noisy activation graph , respectively with the student class activation graph L S Substitute into the distillation loss function to calculate the distillation loss. L KD .
[0035] Distillation loss L KD The function calculates the KL divergence between the probability distributions of the activation graph of the teacher-student model class at each spatial location, and then sums the KL divergence values at all locations.
[0036] The distillation loss function is as follows:
[0037] Among them, the student class activation graph L S N channels at position ( x , y ) activation value ,go through After function normalization, the probability distribution of student model categories is obtained. It is used to represent the student class activation graph. L S exist( x , y The characteristics of the position belong to the first i The probability of each category;
[0038] for Z +1 teacher-type activation image, and select the first one from it. k Activation diagram for teacher category N channels at position ( x , y ) activation value ,go through After function normalization, we obtain the first... k Activation diagram for teacher category Category probability distribution It is used to represent the first k Activation diagram for teacher category At position ( x , y The characteristics of ) belong to the first i The probability of each category, where, k =0 indicates no disturbance to the teacher class activation graph. LT0 ;
[0039] According to a preferred embodiment, step S6 specifically includes: Step S61: Activate the graph of the student class. L S For each channel, the average activation value at all spatial locations is calculated to obtain... logits vector ;
[0040]
[0041] in, Represents the activation graph of the student class L S The i The average activation value in each of the channels is used as the value of the first channel. i Each channel logits value, H S , W S These represent the activation graphs of the student class. L S Height and width; Step S62, using function pairs logits vector Normalization is performed to obtain the classification prediction results. q It is used to represent logits vector The probability of belonging to each category;
[0042] Classification prediction results , express logits vector Belongs to the i The probability of each category; Step S63: Calculate the cross-entropy loss L cls as follows:
[0043] in, It is a real label Corresponding category c The probability of.
[0044] According to a preferred embodiment, in step S7, the total loss of the construction is... Lsum The function is as follows:
[0045] in, , These are the loss weights, used to represent the cross-entropy loss. L cls and distillation loss L KD The importance of.
[0046] It should be noted that the order of steps described in the above embodiments is merely a specific implementation method and is not intended to limit the present invention. Those skilled in the art, based on their understanding of the core ideas of the present invention, can appropriately adjust the order of steps or perform some steps in parallel; all such modifications should be considered to fall within the protection scope of the present invention.
[0047] In this embodiment of the invention, by employing two collaborative perturbation strategies—a feature random discarding strategy and class activation map noise injection—only a single pre-trained teacher model is needed to simulate the diverse output distributions of multi-teacher models. This alleviates the dependence of traditional multi-teacher knowledge distillation on multiple independent pre-trained teacher models, significantly reducing computational and storage costs and deployment barriers. This makes the method highly practical in resource-constrained scenarios such as satellite networks and mobile terminals. The distillation loss in this method considers both the original output of the teacher model (providing accurate anchor points) and multiple perturbation versions of the output (providing diverse knowledge), ensuring that the student model learns more robust feature representations and effectively improving generalization ability and final classification accuracy under complex data distributions. The student model performs knowledge distillation on the class activation map of the teacher model, enabling it to deeply learn the teacher model's attention mechanism and discriminative features for key image regions. Simultaneously, by constraining noise injection, the channel's... logits The value remains unchanged, ensuring the stability of the knowledge transfer process and avoiding the introduction of erroneous signals due to disturbances, thus ensuring model performance while compressing the model.
[0048] The effectiveness of the present invention will be verified below through specific examples applied to particular scenarios.
[0049] Example: A knowledge distillation method based on model perturbation is used to distill the knowledge of a large teacher model into a lightweight student model.
[0050] 1. Experimental Environment and Dataset Configuration Hardware environment: Model training was performed using a server equipped with an NVIDIA GeForce RTX 3090 GPU.
[0051] Software environment: The operating system is Ubuntu 20.04, and the deep learning framework is PyTorch 1.11.0.
[0052] Dataset: CIFAR-100 dataset.
[0053] The dataset contains 60,000 32×32 pixel color images, divided into 100 categories, with 50,000 images used as the training set and 10,000 as the test set. Images were standardized before training and conventional data augmentation strategies were employed, including random horizontal flipping and random cropping.
[0054] 2. Model Configuration Teacher Model: ResNet-32x4 was selected as the teacher model, which has a large capacity and high baseline accuracy. It was first pre-trained on the CIFAR-100 training set to achieve its optimal performance.
[0055] Student model: ResNet-8x4 was chosen as the student model. This model has a lighter structure and far fewer parameters and computational cost than the teacher model. Its parameters are randomly initialized before distillation training begins.
[0056] 3. Training parameters and key parameter configuration of this invention Number of training cycles: 240 Batch package size: 64 Optimizer: SGD optimizer with Nesterov momentum (0.9) is used.
[0057] Learning rate: The initial learning rate is set to 0.05, and a step decay strategy is adopted. The learning rate is adjusted to 0.1 times the original value at the 150th, 180th, and 210th training cycles.
[0058] Number of perturbations: 5, meaning that each iteration generates 5 perturbation versions of the teacher model output.
[0059] Channel drop probability: p= 0.05, in the feature random discard operation, each channel has a 5% probability of being set to zero.
[0060] Noise variance: =0.01, meaning the standard deviation of the Gaussian noise injected into the class activation map is 0.01.
[0061] Loss weights: Cross-entropy loss L cls weight =1.0, distillation loss L KD weight =2.0, giving a higher weight to distillation loss to emphasize the importance of knowledge transfer.
[0062] 4. Specific Implementation Steps Step 1: Load the pre-trained ResNet-32x4 teacher model and the randomly initialized ResNet-8x4 student model; Step 2: Prepare the data loader according to the above configuration; Step 3: Begin cycle training (240 cycles in total): a. Input a batch of 64 images into the backbone networks of the teacher and student models respectively to obtain the teacher feature map. F T Student Feature Map F S .
[0063] b. 6 copies were obtained. F T (1 original) F T +5 units for perturbation F T Five of the execution features were randomly discarded to obtain five initial perturbed teacher feature maps. .
[0064] c. The original F T and 5 portions Input the classification header of the teacher model to obtain 6 class activation maps (1 original teacher class activation map). L T0 5 initial perturbation teacher class activation graphs L T1 ~ L T5 ).
[0065] d. To L T1 ~ L T5 Injecting Gaussian noise that meets the constraints yields 5 noisy activation maps of the perturbed teacher classes. .
[0066] e. will F S The student class activation graph is obtained through the classification head of the student model. L S .
[0067] f. Calculate distillation loss L KD :Will L T0 and These six teacher-related activation images are respectively related to L SCalculate and sum the KL divergence at all spatial locations.
[0068] g. Activation graph for student classes L S Global average pooling is performed to obtain logits reuse softmax The function calculates the prediction result and then calculates the cross-entropy loss between the prediction result and the true label. L cls .
[0069] h. Calculate the total loss The parameters of the student model and the optimizer are updated iteratively through the backpropagation algorithm.
[0070] Step 4: After each training cycle, evaluate the classification accuracy of the current student model on the test set.
[0071] Step 5: After training, select the student model parameters that have the highest accuracy on the test set as the final model.
[0072] 5. Performance Comparison Results To verify the effectiveness of the method of the present invention, it was compared with several mainstream knowledge distillation methods under the same settings. The results are shown in Table 1 (Top-1 accuracy of the test set, %).
[0073] Table 1
[0074] 6. Conclusion: As can be seen from the comparison results in Table 1, under the conditions set in this embodiment, the method proposed in this invention based on... The knowledge distillation method based on model perturbation achieves optimal distillation performance, with an accuracy significantly higher than the student model baseline and other mainstream distillation methods. This fully demonstrates that the present invention can simulate the diverse knowledge of multiple teachers through single-teacher model perturbation, and can effectively alleviate the dependence of multi-teacher knowledge distillation on multiple independent pre-trained teacher models, thereby improving resource utilization.
[0075] Accordingly, embodiments of the present invention also protect a system for implementing the above-described knowledge distillation method based on model perturbation. Those skilled in the art will understand that the system should include functional modules corresponding to the steps of the above method. Specifically, the system should at least include functional units configured to perform the steps of the above method. Regarding the system in this embodiment, the specific manner in which each unit module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0076] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including: a memory and a processor, wherein the processor is used to read and execute a computer program stored in the memory to implement the aforementioned knowledge distillation method based on model perturbation.
[0077] Based on the same inventive concept, embodiments of the present invention also provide a computer storage medium storing computer-executable instructions, which, when executed, implement the aforementioned knowledge distillation method based on model perturbation.
[0078] In the several embodiments provided in this application, it should be understood that the disclosed apparatus, system, and method can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.
[0079] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0080] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0081] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.
[0082] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail in a particular embodiment can be found in the relevant descriptions of other embodiments. Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A knowledge distillation method based on model perturbation, characterized in that, The method includes: Step S1: Obtain the training set and test set, as well as the pre-trained teacher model and the student model to be distilled. The training set and test set are color images. Input the sample data in the training set into the backbone network of the teacher model and the student model respectively, and extract the teacher feature map. F T Student Feature Map F S ; Step S2: Copy the teacher feature map F T , to obtain the total Z +1 teacher profile F T Among them Z Teacher Feature Chart F T Perform a feature random discarding operation at the channel level to obtain Z initial perturbed teacher feature maps. And an undisturbed teacher profile F T ; Step S3, the... Z Initial perturbation teacher feature map The unperturbed teacher feature map F T Input the classification header of the teacher model respectively, and obtain the mapping. Z Initial perturbation teacher class activation graph L T1 ~ L TZ And the activation graph of the non-disruptive teacher class L T0 The student feature map F S Input the class header of the student model to obtain the student class activation map. L S ; Step S4, for the Z Initial perturbation teacher class activation graph L T1 ~ L TZ Inject constrained Gaussian noise separately to generate Z Noisy teacher activation graph ; Step S5, Calculation Z +1 teacher-class activation image and the aforementioned student-class activation image L S Distillation loss between L KD , wherein Z +1 teacher-related activation image, including a non-disruptive teacher-related activation image. L T0 and Z Noisy teacher activation graph ; Step S6: Activate the graph of the student class. L S Perform global average pooling to output logits Value, the logits Value after softmax After processing by the function, the classification prediction result is obtained, and the cross-entropy loss between the classification prediction result and the true label is calculated. L cls ; Step S7: Utilize the distillation loss L KD and the cross-entropy loss L cls Weighted sum of total loss L sum And by minimizing the total loss L sum To update the parameters of the student model; Step S8: Repeat steps S1-S7 until the training set has been traversed, and use the test set to evaluate the performance of the student model and obtain the optimal student model parameters.
2. The method according to claim 1, characterized in that, In step S1, the pre-trained teacher model and the student model to be distilled are convolutional neural network models, both of which contain a backbone network and a classification head. The backbone network is responsible for extracting feature representations, and the classification head is responsible for mapping the feature representations to the category space. The training sample data is input into the backbone networks of the teacher model and the student model respectively, and the teacher feature map is extracted. F T Student Feature Map F S ,include: The training sample data is input into the backbone network of the teacher model to extract teacher feature maps. F T : Teacher Feature Map F T It is a three-dimensional tensor, in which, C T This represents the total number of channels in the teacher feature map. H T , W T Let these represent the height and width of the teacher feature map, respectively, and the first and second dimensions of the teacher feature map. i Each channel is located in The eigenvalues are represented as ,exist middle, , , ; The training sample data is input into the backbone network of the student model to extract student feature maps. F S : in, C S Student feature map F S The number of channels, H S , W S Let these represent the height and width of the student feature map, respectively, and the first and second dimensions of the student feature map. i Each channel is located in The eigenvalues are represented as ,exist middle, , , .
3. The method according to claim 2, characterized in that, In step S2, for Z Teacher Feature Chart F T Perform a feature random discard operation at the channel level to obtain Z Initial perturbation teacher feature map ,include: right Z Teacher Feature Chart F T Bernoulli distribution was used to analyze the feature maps of each teacher. F T Each channel is sampled independently using a binary mask to obtain a set of binary masks. , No. i Binary mask for each channel m i It conforms to the Bernoulli distribution. , p This represents the probability of channel dropping. The perturbed teacher feature map obtained after the feature random discarding operation Represented as: in, Represents the teacher feature map F T Perform successive multiplication operations with the binary mask set M, where, Teacher feature map F T The i Feature map of each channel.
4. The method according to claim 3, characterized in that, Step S3 specifically includes: The Z Initial perturbation teacher feature map The classification headers of the teacher model are input sequentially, and the mapping is obtained. Z Initial perturbation teacher class activation graph L T1 ~ L TZ ; The non-disturbing teacher feature map F T Input the classifier header of the teacher model to obtain an unperturbed teacher class activation graph. L T0 ; L Tk Indicates the first k There are perturbation versions of the teacher class activation graph, where N is the number of categories in the teacher model, and each channel corresponds to the activation graph of one category. , k =0 indicates no disturbance to the teacher class activation graph. L T0 ; No. k A disturbed version of the teacher class activation image L Tk The i Each channel is located at ( x , y The activation value of ) is represented as ,exist middle, , , ; The student feature map F S Input the class header of the student model to obtain the student class activation map. L S ; Where N is the number of classification categories in the student model, and each channel corresponds to the activation map of one category; The student class activation graph L S The i Each channel is located at ( x , y The activation value of ) is represented as ,exist middle, , , .
5. The method according to claim 4, characterized in that, Step S4 specifically includes: The selected dimensions are the same as those of the perturbation teacher class activation graph, with a mean of 0 and a variance of . Gaussian noise As noise to be injected; Gaussian noise was injected into the respective... Z Initial perturbation teacher class activation graph L T1 ~ L TZ In the middle, we get Z Noisy teacher activation graph ; in, Indicates the first k A perturbed version of the teacher model with noisy class activation graphs, and .
6. The method according to claim 5, characterized in that, Step S4 also includes: Injected Gaussian noise Apply constraints; Among them, Gaussian noise The i Each channel is at a random location. noise value Replace it with the negative sum of other noise values in the same channel, so that the sum of the noise values in the entire channel is zero. This ensures that the noise activation map of the perturbed teacher is properly represented in each channel after global average pooling. logits The value remains unchanged; in, The first Gaussian noise i In each channel, except for position The noise value at any other location outside of the specified range.
7. The method according to claim 5 or 6, characterized in that, In step S5, Z +1 teacher-class activation map, including the aforementioned noisy teacher-class activation map. L T0 and stated Z Noisy teacher activation graph , respectively with the student class activation graph L S Substitute into the distillation loss function to calculate the distillation loss. L KD It calculates the KL divergence between the class probability distributions of the teacher-student model class activation graph at each spatial location, and then sums the KL divergence values at all locations. The distillation loss function is as follows: Among them, the student class activation graph L S N channels at position ( x , y ) activation value ,go through After function normalization, the probability distribution of student model categories is obtained. It is used to represent the student class activation graph. L S exist( x , y The characteristics of the position belong to the first i The probability of each category; for Z +1 teacher-type activation image, and select the first one from it. k Activation diagram for teacher category N channels at position ( x , y ) activation value ,go through After function normalization, we obtain the first... k Activation diagram for teacher category Category probability distribution It is used to represent the first k Activation diagram for teacher category At position ( x , y The characteristics of ) belong to the first i The probability of each category, where, k =0 indicates no disturbance to the teacher class activation graph. L T0 ; 。 8. The method according to claim 7, characterized in that, Step S6 specifically includes: Activation graph of the student class L S For each channel, the average activation value at all spatial locations is calculated to obtain... logits vector ; in, Represents the activation graph of the student class L S The i The average activation value in each of the channels is used as the value of the first channel. i Each channel logits value, H S , W S These represent the activation graphs of the student class. L S Height and width; use function pairs logits vector Normalization is performed to obtain the classification prediction results. q It is used to represent logits vector The probability of belonging to each category; Classification prediction results , express logits vector Belongs to the i The probability of each category; Calculate cross-entropy loss L cls as follows: in, The category corresponding to the real label c The probability of.
9. The method according to claim 8, characterized in that, In step S7, the total loss is constructed. L sum The function is as follows: in, , These are weight parameters, used to represent the cross-entropy loss. L cls and distillation loss L KD The importance of.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed, implement the method described in any one of claims 1-9.