Deep neural network pruning method based on dynamic contrast mask and knowledge distillation

Abstract

The application is suitable for the technical field of model pruning, and provides a deep neural network pruning method based on dynamic contrastive mask and knowledge distillation.The application proposes a dynamic pruning framework based on dynamic contrastive mask learning and mutual knowledge distillation to compress a deep neural network model.Dynamic contrastive mask learning aggregates similar masks through contrastive learning and distinguishes them in a feature space to generate a binary mask;combined with a sample complexity adaptive factor, a sample subnetwork is adaptively pruned without manually setting a pruning threshold.Mutual knowledge distillation realizes knowledge transfer and feature enhancement through mutual learning between subnetworks to alleviate feature information loss in the pruning process.The framework can reduce model size and computational overhead, improve inference speed, maintain high accuracy, and improve model performance and robustness, and is suitable for scenarios with limited computing resources, can realize efficient channel pruning under the premise of ensuring deep neural network accuracy, and further improve model generalization ability.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of model pruning, and particularly relates to a deep neural network pruning method based on dynamic contrast mask and knowledge distillation. BACKGROUND

[0002] In recent years, deep neural networks (DNNs) have made significant progress in image classification, image segmentation and object detection. However, in pursuit of higher performance, the network structure is constantly expanding in depth and width, leading to a sharp increase in computing overhead and storage requirements. This growing resource demand limits the application of deep neural networks on resource-constrained platforms such as smartphones and Internet of Things (IoT). To address this challenge, model compression and acceleration technology has emerged, aiming to maintain high accuracy and improve inference speed while minimizing the size and computational overhead of deep neural network models. Existing methods mainly include knowledge distillation, network pruning, parameter quantization and lightweight model design, among which network pruning and knowledge distillation are the mainstream technologies for deep neural network model compression due to their relatively simple implementation and significant compression effect.

[0003] Traditional network pruning methods typically apply a uniform pruning strategy to all samples, generating a fixed, compact network for the entire dataset; these methods are known as static pruning. However, in real-world applications, different samples exhibit significantly different levels of network redundancy. For example, samples with complex backgrounds or occlusions often require more channels to capture detailed features, exhibiting lower redundancy; while simpler samples usually have higher redundancy and can be pruned more aggressively while maintaining accuracy. Static pruning cannot fully exploit network redundancy based on sample differences. To more fully exploit network redundancy, researchers have proposed dynamic pruning methods to compress deep neural networks. These methods can dynamically adjust the network structure based on the complexity of the input samples, generating tailored sub-networks for different samples. Although dynamic pruning shows certain performance advantages, it still has the following drawbacks: Firstly, its pruning threshold usually relies on expert experience to set through multiple trials, which can easily lead to inaccurate removal of redundant channels and unpredictable performance degradation; secondly, these methods often only focus on the performance of a single optimal sub-network, neglecting the potential mutual learning and knowledge complementarity between multiple sub-networks, limiting the overall learning ability and robustness of the pruning model. To mitigate the uncertainty of manual configuration, recent research has explored automatic pruning methods to compress deep neural network models, enabling adaptive evaluation and selection of channel importance. For example, some methods utilize reinforcement learning or neural architecture search to automatically find compact neural network models that meet specific criteria, while others generate binary masks by imposing constraints during mask learning, thus safely removing channels with a mask value of 0 without requiring pruning. Despite significant progress in automatic pruning, problems remain, including low search efficiency, unstable training, and complex pruning processes.

[0004] In view of the shortcomings of the existing pruning techniques, this invention proposes a deep neural network pruning method based on dynamic contrast masking and knowledge distillation. Summary of the Invention

[0005] The purpose of this invention is to provide a deep neural network pruning method based on dynamic contrast masking and knowledge distillation, which aims to solve the problems mentioned in the background art.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] The deep neural network pruning method based on dynamic contrastive masking and knowledge distillation includes the following steps:

[0008] To construct a proxy task to guide mask learning for input samples, specifically, the training process is modeled as a time series, and positive and negative examples are constructed by combining the mask of the current training period sample with the mask of the previous training period sample.

[0009] A dynamic contrast mask loss function containing a sample complexity adaptive factor is designed to generate a binary mask matching the complexity of the input sample, thereby achieving differential pruning of different input samples.

[0010] Based on the similarity of the feature extraction capabilities of the subnetworks for input samples, a mutual knowledge distillation mechanism is constructed to measure the consistency of the prediction distribution of the subnetworks for input samples through KL divergence, thereby achieving knowledge transfer and collaborative learning between subnetworks and ensuring the performance of the pruned model.

[0011] Further, in the step of constructing a proxy task, for the first channel, the sample is represented by the mask obtained in the first cycle of training as , and the mask obtained in the second cycle of training as , which form a positive mask pair, and the remaining 2K-1 masks are considered as negative masks, where K represents the number of channels of the entire network.

[0012] Further, the base target of the dynamic contrast mask loss function is optimized by maximizing the average log-likelihood of positive samples, and the formula is:

[0013] ;

[0014] wherein is the mask of the current training cycle; is the mask of the previous training cycle; is the probability that the mask of the previous training cycle is a positive example of the mask of the current training cycle; represents a positive example; K represents the number of channels of the entire network; is the first channel; represents the cosine similarity distance; is the temperature coefficient; k is the first channel.

[0015] Further, the calculation formula of the sample complexity adaptive factor is:

[0016] ;

[0017] wherein is the sample complexity adaptive factor; is the reverse transformation of the cross-entropy loss ; and represent the maximum and minimum values of , respectively.

[0018] Further, the dynamic contrast mask loss function introduces an L1 regularization term, and the mask contrast loss is obtained:

[0019] ;

[0020] wherein, is the mask contrast loss function; N represents the total number of samples; i represents the i-th sample; is the weight balance coefficient; is the mask obtained in the e-th training period.

[0021] Further, the formula for obtaining the predicted class probability from the subnetwork is:

[0022] ;

[0023] wherein, is the subnetwork predicting the probability of class y; represents the output of the Softmax layer of the subnetwork ; and represent the class and the total number of classes, respectively.

[0024] Further, the formula for measuring the consistency of the prediction distribution of the two subnetworks by KL divergence is:

[0025] ;

[0026] wherein, is the KL divergence of the prediction probability distribution of the subnetwork and the prediction probability distribution of the subnetwork ; i represents the i-th sample; N represents the total number of samples; and represent the class and the total number of classes, respectively; represents the subnetwork

[0027] predicting the input sample to be class ; represents the subnetwork predicting the input sample to be class .

[0028] Further, the mutual distillation loss formula of the mutual knowledge distillation mechanism is:

[0029] wherein, Loss function for mutual distillation between any two sub-networks in N sub-networks; N is the total number of sub-networks; i is the i-th sub-network; is the i-th sub-network; is the i-th sub-network; is the predicted probability distribution of the i-th sub-network is the predicted probability distribution of the i-th sub-network is the predicted probability distribution of the i-th sub-network is the KL divergence of the predicted probability distribution of the i-th sub-network

[0030] A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the deep neural network pruning method based on dynamic contrastive mask and knowledge distillation as described above

[0031] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the deep neural network pruning method based on dynamic contrastive mask and knowledge distillation as described above when executing the computer program.

[0032] Compared with the prior art, the beneficial effects of the present application are:

[0033] The present application proposes a dynamic pruning framework based on dynamic contrastive mask learning and mutual knowledge distillation for compressing deep neural network models. In terms of dynamic contrastive mask learning, similar masks are aggregated together (i.e., the masks of redundant channels and important channels are aggregated into two clusters, respectively) and distinguished in the feature space through contrastive learning to generate binary masks; meanwhile, a sample complexity adaptive factor is introduced to dynamically evaluate sample complexity and adjust the polarization degree of the mask, without the need for manual setting of the pruning threshold, so as to realize adaptive automatic pruning of sub-networks for different samples. In terms of mutual knowledge distillation, knowledge transfer and feature enhancement are realized through mutual learning between different sub-networks to promote knowledge complementarity and information sharing of sub-networks, effectively alleviating the loss of feature information during the pruning process. This framework can reduce model size and computational overhead, improve inference speed, maintain high accuracy, and improve model performance and robustness, and is particularly suitable for scenarios with limited computing resources. It can automatically realize efficient channel pruning while ensuring the accuracy of deep neural networks, further improving the generalization ability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a flowchart of the method of the present application. DETAILED DESCRIPTION

[0035] In order to have a clearer understanding of the technical features, objectives and beneficial effects of the present application, the technical solutions of the present application will be described in detail below, but it should not be understood as limiting the scope of the present application. ​

[0036] The specific implementation of the present application is described in detail below in combination with specific embodiments.

[0037] The present application provides a deep neural network pruning method based on dynamic contrastive mask and knowledge distillation. The core is to combine the advantages of dynamic pruning and knowledge distillation, fully tap the redundancy of each sample network, and ensure the high performance of the pruned model. The flow chart is as shown in Figure 1 The specific technical solutions are as follows:

[0038] I. Dynamic contrastive mask learning: the key to realizing dynamic contrastive mask learning lies in designing an effective proxy task and a dynamic contrastive mask loss function, so as to automatically generate the corresponding binary mask according to the complexity of different samples and complete automatic pruning. The specific process is as follows:

[0039] 1. Construct a proxy task to guide the learning process of the mask. In computer vision tasks, the proxy task usually relies on various data augmentation (for example, random cropping, color distortion, Gaussian blur), and the mask learned in channel pruning is usually low-dimensional and cannot directly benefit from such augmentation strategies. Therefore, designing a reasonable proxy task is crucial to effectively guide the mask learning process. Specifically, the entire training process is modeled as a time series, and each training cycle is regarded as a time point. For any sample, the mask generated in the current training cycle is regarded as an anchor point, and the mask generated in the previous training cycle is used to construct positive and negative examples to guide the learning of the current mask. For example, for the i-th channel, the sample The mask obtained in the training of the i-th cycle is denoted as , and the mask obtained in the training of the i-1-th cycle is denoted as , which constitutes a positive mask pair, and the remaining 2K-1 masks are regarded as negative masks (where K represents the number of channels of the entire network). 2. By designing a dynamic contrastive mask loss function, a sample complexity adaptive factor is introduced to enable the model to adaptively learn a binary mask for each sample. Firstly, the present application proposes a dynamic contrastive mask loss function, which uses the characteristics of contrastive learning to narrow the distance between similar masks in the feature space and push away the distance between dissimilar masks. The basic goal of this loss function is to maximize the lower bound of mutual information between the mask

[0040] obtained in the current training cycle and the mask obtained in the previous training cycle. Since it is difficult to directly optimize the lower bound of mutual information, an approximate strategy is adopted, that is, the average log-likelihood of positive samples is maximized to optimize, and the formula is as follows:

[0041]

[0042] ​​​;

[0043] wherein, is the probability that the previous training cycle mask is a positive example of the current training cycle mask; represents a positive example; K represents the number of channels of the entire network; is the first channel; represents a cosine similarity distance; is a temperature coefficient; k is the first channel.

[0044] Secondly, the dynamic contrast mask loss can perceive the sample complexity to automatically learn the binary mask for each sample. In order to dynamically adjust the network structure according to the complexity of each sample and generate the corresponding subnetwork, the present application further introduces an L1 regularization term on the basis of the above objective function, and designs a sample complexity adaptive factor , which can dynamically generate a binary mask matching the complexity of the sample by automatically perceiving the sample complexity, not only realizing the adaptive mining of network redundancy by adjusting the binary distribution of the mask, but also generating a subnetwork matching the complexity of the sample according to the mask, and finally realizing the differentiated pruning of samples with different complexities. The calculation formula is as follows:

[0045] ;

[0046] Here, the cross-entropy loss (Cross-Entropy Loss) for the classification task is abbreviated as . Since the complexity of the sample is positively correlated with , an inverse transformation is needed, denoted as , , and represent the maximum and minimum values, respectively. The numerical value of is positively correlated with the sample complexity, and the complexity sample corresponds to a larger , and the mask value tends to be close to 1, so as to retain more channels and form a more complex subnetwork ; while the simple sample corresponds to a smaller , and the mask value tends to 0, realizing more aggressive pruning and forming a more streamlined subnetwork . Therefore, without manually setting the pruning threshold, the channels corresponding to 0 can be safely removed, and then multiple subnetworks with different structures can be obtained. Finally, the final expression of the mask contrastive loss (Mask Contrastive Loss, MC Loss) proposed in the present application is as follows:

[0047] ;

[0048] wherein, is the mask contrast loss function; denotes the total number of samples; i is the i-th sample; is the weight balance coefficient, used to control the size of the pruning rate; is the mask obtained in the e-th training period.

[0049] II. Mutual knowledge distillation: dynamic pruning according to the complexity of each sample in the dataset to generate a corresponding set of subnetworks with different structures . The structural complexity of the subnetworks is positively correlated with the complexity of the samples, but they are derived from different subsets of the same large network, so they have high similarity in feature extraction ability. Based on this feature, these subnetworks can learn from and supplement each other, thereby forming a natural collaborative learning mechanism to obtain more robust representations. Therefore, a mutual knowledge distillation mechanism is constructed to make up for the loss of key information caused by pruning. The specific process is as follows:

[0050] Suppose two subnetworks and (simplified as and ) learn from each other, first get the probability of the predicted class from the subnetworks, the formula is:

[0051] ;

[0052] wherein, is the probability of the subnetwork predicting the y-th class; denotes the output of the Softmax layer of the subnetwork ; and denote the class and the total number of classes, respectively.

[0053] Traditional supervised loss is used to train the network to ensure that it can correctly predict the labels of the training samples. However, in order to further improve the generalization and robustness of the network on the test samples, the present invention introduces another peer network , and uses its posterior probability as additional training experience. By conducting discriminative learning in the feature space, each subnetwork can obtain supplementary representation information from other subnetworks. Subsequently, the subnetwork is used as the teacher network, and knowledge distillation loss is used to realize knowledge transfer between subnetworks. Specifically, the Kullback-Leibler (KL) divergence is used to measure the difference between the predicted distributions and of the two networks and consistency, thereby facilitating collaborative learning and knowledge transfer between different subnetworks. The specific calculation formula is as follows:

[0054] ;

[0055] wherein, is the KL divergence of the predicted probability distribution of the subnetwork and the predicted probability distribution of the subnetwork ; i is the i-th sample; N is the total number of samples; and respectively represent the class and the total number of classes; represents the probability of the subnetwork predicting the input sample to be the class ; represents the probability of the subnetwork predicting the input sample to be the class .

[0056] In order to embody the mutual learning ability between subnetworks, not only the subnetwork transfers knowledge to , but also transfers knowledge to , forming a mutual distillation structure. Therefore, the final mutual distillation loss (MD Loss) is as follows:

[0057] ;

[0058] wherein, is the loss function of mutual distillation between any two subnetworks in N subnetworks; N is the total number of subnetworks; i is the i-th subnetwork; is the i-th subnetwork; is the KL divergence of the predicted probability distribution of the subnetwork and the predicted probability distribution of the subnetwork . is the KL divergence of the predicted probability distribution of the subnetwork and the predicted probability distribution of the subnetwork .

[0059] Example 1: Performance verification of the method of the present application;

[0060] ​​​​​​​To verify the effectiveness of the deep neural network pruning method based on dynamic contrast mask and knowledge distillation, the pruning experiments are carried out on two different scale datasets of CIFAR-10 and ImageNet, for three mainstream deep neural network structures of simple architecture VGGNet (such as VGG16), residual structure ResNets (such as ResNet56, ResNet18 and ResNet34) and lightweight design MobileNetV2. The experimental results are shown in Tables 1 and 2.

[0061] Table 1: Comparison of pruning results of VGG16, ResNet56 and MobileNetV2 on CIFAR-10 dataset

[0062]

[0063] Table 2: Comparison of pruning results of ResNet18 and ResNet34 on ImageNet dataset.

[0064]

[0065] Note: In Tables 1 and 2, "DY" represents the dynamic pruning method, "Y" indicates that the method is dynamic pruning, and "N" indicates that the method is non-dynamic pruning; "Baseline Acc" represents the baseline model accuracy; "Pruned Acc" represents the model accuracy after pruning; "Acc↓" represents the decline rate of the model accuracy after pruning compared with the baseline model accuracy (negative value represents the accuracy increase); "FLOPs↓" represents the decline rate of the model floating-point operation (Floating-point operations) compared with the baseline model; "Param↓" represents the decline rate of the model parameter (Parameters) compared with the baseline model; "Pruned Top-1" represents the Top-1 accuracy of the model after pruning (i.e. the accuracy rate of the class with the highest probability in the prediction result consistent with the true class); "Top-1↓" represents the decline rate of the Top-1 accuracy of the model after pruning compared with the baseline model (negative value represents the accuracy increase); "Pruned Top-5" represents the Top-5 accuracy of the model after pruning (i.e. the accuracy rate of the top 5 classes containing the true class in the prediction result); "Top-5↓" represents the decline rate of the Top-5 accuracy of the model after pruning compared with the baseline model (negative value represents the accuracy increase); "-" represents that this value is not reported in the original paper; the bold values in the table are the best results of the corresponding indicators.

[0066] It can be seen from the experimental results that the performance of the method (Ours) is better. For example, after pruning ResNet56 on the CIFAR-10 dataset, the floating point operation (FLOPs) and parameter quantity (Param) of the model are reduced by 64.4% and 60.7% respectively, and the model accuracy is improved by 0.18% compared with the baseline. On the large-scale dataset ImageNet, when pruning ResNet34, a FLOPs reduction of 51.3% is achieved, the Top-1 accuracy only decreases by 0.01%, and the Top-5 accuracy increases by 0.08%. These results show that the present application can fully exploit the network redundancy of each sample, automatically identify and remove redundant filters, and effectively compensate for the loss of representation information during the pruning process through the mutual distillation mechanism, thereby improving the overall performance of the pruned model.

[0067] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which should also be considered as the protection scope of the present application, and these will not affect the effect and practicality of the present application.

Claims

1. A deep neural network pruning method based on dynamic contrastive mask and knowledge distillation, characterized in that, The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: ； wherein, is a current training cycle mask; is a previous training cycle mask; is a probability that the previous training cycle mask is a positive example of the current training cycle mask; denotes a positive example; K denotes the number of channels of the entire network; is the channel; is the channel; denotes a cosine similarity distance; is a temperature coefficient; k is the channel; is the channel; In the step of constructing the agent task, for the first... One channel, sample In training The mask obtained from each period is represented as follows: , and in the The mask obtained in each period These form positive mask pairs, while the remaining 2K-1 masks are considered negative masks, where K represents the number of channels in the entire network; The method comprises the following steps: ； wherein, is a sample complexity adaptation factor; is a cross-entropy loss is a reverse transformation of; and denote the maximum and minimum, respectively, of the maximum and minimum, respectively, of The method comprises the following steps: ； wherein, is a mask contrast loss function; denotes the total number of samples; i is the i-th sample; is a weight balancing coefficient; is a mask obtained in the e-th training cycle.

2. The deep neural network pruning method based on dynamic contrast mask and knowledge distillation according to claim 1, characterized in that, The method comprises the following steps: ； wherein, is a subnetwork predicts a probability of being a class y; denotes an output of a Softmax layer of a subnetwork and denote a class and a total number of classes, respectively.​ 3. The deep neural network pruning method based on dynamic contrast mask and knowledge distillation according to claim 2, characterized in that, The method comprises the following steps: ； wherein, is the predicted probability distribution of the subnetwork is the predicted probability distribution of the subnetwork is the KL divergence of the predicted probability distribution of the subnetwork and denote the class and the total number of classes, respectively; denotes the predicted probability of the input sample by the subnetwork is the probability of the class denotes the predicted probability of the input sample by the subnetwork is the probability of the class ​​​​ 4. The deep neural network pruning method based on dynamic contrast mask and knowledge distillation according to claim 3, characterized in that, The method comprises the following steps: ； wherein, is the loss function for mutual distillation between any two subnetworks in N subnetworks; N is the total number of subnetworks; i is the i-th subnetwork; is the i-th subnetwork; is the predicted probability distribution of the i-th subnetwork is the KL divergence of the predicted probability distribution of the i-th subnetwork​​​​ 5. A computer-readable storage medium having stored thereon a computer program, characterized in that, The method comprises the following steps:

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises

Images