Deep neural network pruning method based on dynamic contrast mask and knowledge distillation

Abstract

The invention is suitable for the technical field of model pruning, and provides a deep neural network pruning method based on dynamic contrast masks and knowledge distillation. The invention provides a dynamic pruning framework based on dynamic contrast mask learning and mutual knowledge distillation to compress a deep neural network model. According to dynamic comparison mask learning, similar masks are aggregated through comparison learning and are distinguished in a feature space, and binary masks are generated; and in combination with a sample complexity adaptive factor, a pruning threshold does not need to be manually set, and adaptive pruning of the sample sub-network is realized. Mutual knowledge distillation realizes knowledge migration and feature enhancement by utilizing mutual learning among sub-networks, and feature information loss in the pruning process is relieved. The framework can maintain high precision and improve the model performance and robustness while reducing the model size and calculation overhead and improving the reasoning speed, is suitable for a scene with limited calculation resources, can realize efficient channel pruning on the premise of ensuring the precision of the deep neural network, and further improves the generalization ability of the model.

Description

Technical Field

[0001] This invention belongs to the field of model pruning technology, and particularly relates to a deep neural network pruning method based on dynamic contrast masking and knowledge distillation. Background Technology

[0002] In recent years, deep neural networks (DNNs) have made significant progress in image classification, image segmentation, and object detection. However, in pursuit of higher performance, network structures have continuously expanded in depth and width, leading to a sharp increase in computational overhead and storage requirements. This growing resource demand limits the application of deep neural networks on resource-constrained platforms such as smartphones and the Internet of Things (IoT). To address this challenge, model compression and acceleration techniques have emerged, aiming to minimize the size and computational overhead of deep neural network models while maintaining high accuracy and improving inference speed. Existing methods mainly include knowledge distillation, network pruning, parameter quantization, and lightweight model design. Among these, network pruning and knowledge distillation have become mainstream techniques for deep neural network model compression due to their relatively simple implementation and significant compression effects.

[0003] Traditional network pruning methods typically apply a uniform pruning strategy to all samples, generating a fixed, compact network for the entire dataset; these methods are known as static pruning. However, in real-world applications, different samples exhibit significantly different levels of network redundancy. For example, samples with complex backgrounds or occlusions often require more channels to capture detailed features, exhibiting lower redundancy; while simpler samples usually have higher redundancy and can be pruned more aggressively while maintaining accuracy. Static pruning cannot fully exploit network redundancy based on sample differences. To more fully exploit network redundancy, researchers have proposed dynamic pruning methods to compress deep neural networks. These methods can dynamically adjust the network structure based on the complexity of the input samples, generating tailored sub-networks for different samples. Although dynamic pruning shows certain performance advantages, it still has the following drawbacks: Firstly, its pruning threshold usually relies on expert experience to set through multiple trials, which can easily lead to inaccurate removal of redundant channels and unpredictable performance degradation; secondly, these methods often only focus on the performance of a single optimal sub-network, neglecting the potential mutual learning and knowledge complementarity between multiple sub-networks, limiting the overall learning ability and robustness of the pruning model. To mitigate the uncertainty of manual configuration, recent research has explored automatic pruning methods to compress deep neural network models, enabling adaptive evaluation and selection of channel importance. For example, some methods utilize reinforcement learning or neural architecture search to automatically find compact neural network models that meet specific criteria, while others generate binary masks by imposing constraints during mask learning, thus safely removing channels with a mask value of 0 without requiring pruning. Despite significant progress in automatic pruning, problems remain, including low search efficiency, unstable training, and complex pruning processes.

[0004] In view of the shortcomings of the existing pruning techniques, this invention proposes a deep neural network pruning method based on dynamic contrast masking and knowledge distillation. Summary of the Invention

[0005] The purpose of this invention is to provide a deep neural network pruning method based on dynamic contrast masking and knowledge distillation, which aims to solve the problems mentioned in the background art.

[0006] The objective of this invention is achieved through the following technical solution: The deep neural network pruning method based on dynamic contrastive masking and knowledge distillation includes the following steps: To construct a proxy task to guide mask learning for input samples, specifically, the training process is modeled as a time series, and positive and negative examples are constructed by combining the mask of the current training period sample with the mask of the previous training period sample. Design a dynamic contrastive mask loss function with an adaptive factor for sample complexity to generate a binary mask that matches the complexity of the input samples, thereby achieving differentiated pruning for different input samples; Based on the similarity of the sub-networks' feature extraction capabilities for input samples, a mutual knowledge distillation mechanism is constructed. The KL divergence is used to measure the consistency of the sub-networks' predicted distribution of input samples, thereby realizing knowledge transfer and collaborative learning among sub-networks and ensuring the performance of the model after pruning.

[0007] Furthermore, in the step of constructing the agent task, for the first... One channel, sample In training The mask obtained from each period is represented as follows: , and in the The mask obtained in each period Form a positive mask pair, the remaining 2 K A mask of -1 is considered a negative mask, where K This represents the total number of channels in the entire network.

[0008] Furthermore, the fundamental objective of the dynamic contrastive mask loss function is optimized by maximizing the average log-likelihood of positive samples, as shown in the formula: ; in, This is the mask for the current training cycle; The mask for the previous training cycle; This represents the probability that the mask from the previous training cycle is a positive example of the mask from the current training cycle. represents a positive example; K This represents the total number of channels in the entire network. For the first One channel; Represents the cosine similarity distance; Temperature coefficient; k For the first One channel.

[0009] Furthermore, the formula for calculating the sample complexity adaptive factor is as follows: ; in, This is an adaptive factor for sample complexity. Cross-entropy loss The reverse transformation; and They represent The maximum and minimum values.

[0010] Furthermore, after introducing an L1 regularization term into the dynamic contrast mask loss function, the mask contrast loss is obtained as follows: ; in, For mask contrast loss function; Indicates the total number of samples; i For the first i One sample; This is the weighting balance coefficient; For the first e The mask obtained from each training cycle.

[0011] Furthermore, the formula for obtaining the predicted class probability from the sub-network is as follows: ; in, For sub-networks Predicted as y The probability of the category; Subnetwork The output of the Softmax layer; and These represent the category and the total number of categories, respectively.

[0012] Furthermore, the formula for measuring the consistency of the predicted distributions of the two sub-networks using the KL divergence metric is as follows: ; in, For sub-networks Predicted probability distribution Hezi Network Predicted probability distribution KL divergence; i For the first i One sample; N The total number of samples; and These represent the category and the total number of categories, respectively. Subnetwork Predict input samples For category The probability of; Subnetwork Predict input samples For category The probability of.

[0013] Furthermore, the formula for the mutual distillation loss of the mutual knowledge distillation mechanism is as follows: ; in, In order to be in N The loss function for mutual distillation of any two subnetworks in a subnetwork; N The total number of sub-networks; i For the firsti Subnetwork; For the first Subnetwork; For sub-networks Predicted probability distribution Hezi Network Predicted probability distribution The KL divergence.

[0014] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the deep neural network pruning method based on dynamic contrast masking and knowledge distillation as described above.

[0015] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the deep neural network pruning method based on dynamic contrast masking and knowledge distillation as described above.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention proposes a dynamic pruning framework based on dynamic contrastive mask learning and mutual knowledge distillation for compressing deep neural network models. In dynamic contrastive mask learning, similar masks are clustered together (redundant and important channel masks are grouped into two separate clusters) and distinguished in the feature space to generate binary masks. Simultaneously, an adaptive sample complexity factor is introduced to dynamically evaluate sample complexity and adjust mask polarization, achieving adaptive automatic pruning for different sample sub-networks without the need for manually setting pruning thresholds. In mutual knowledge distillation, knowledge transfer and feature enhancement are achieved through mutual learning between different sub-networks, promoting knowledge complementarity and information sharing among sub-networks and effectively mitigating feature information loss during pruning. This framework can reduce model size and computational overhead, improve inference speed, maintain high accuracy, and enhance model performance and robustness. It is particularly suitable for scenarios with limited computational resources, automatically achieving efficient channel pruning while ensuring the accuracy of deep neural networks, further improving model generalization ability. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0018] In order to provide a clearer understanding of the technical features, objectives and beneficial effects of the present invention, the technical solution of the present invention will now be described in detail below, but it should not be construed as limiting the scope of implementation of the present invention.

[0019] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0020] This invention provides a deep neural network pruning method based on dynamic contrastive masking and knowledge distillation. The core idea is to combine the advantages of dynamic pruning and knowledge distillation to fully exploit network redundancy in each sample while ensuring high performance of the pruned model. Its flowchart is as follows: Figure 1 As shown, the specific technical solution is as follows: I. Dynamic Contrast Mask Learning: The key to achieving dynamic contrast mask learning lies in designing an effective proxy task and a dynamic contrast mask loss function. This allows for the automatic generation of corresponding binary masks based on the complexity of different samples, thus achieving automated pruning. The specific process is as follows: 1. Constructing a surrogate task to guide the mask learning process. In computer vision tasks, surrogate tasks often rely on various data augmentations (e.g., random cropping, color distortion, Gaussian blur). Masks learned through channel pruning are typically low-dimensional and cannot directly benefit from such augmentation strategies. Therefore, designing a suitable surrogate task is crucial for effectively guiding the mask learning process. Specifically, the entire training process is modeled as a time series, with each training epoch considered a point in time. For any sample, the mask generated in the current training epoch is considered the anchor point, while the mask generated in the previous training epoch is used to construct positive and negative examples to guide the learning of the current mask. For example, for the One channel, sample In training The mask obtained from each period is represented as follows: , and in the The mask obtained in each period Form a positive mask pair, the remaining 2 K A mask of -1 is considered a negative mask (where...) K (This represents the total number of channels in the entire network).

[0021] 2. By designing a dynamic contrast mask loss function and introducing an adaptive factor for sample complexity, the model can adaptively learn a binary mask for each sample.

[0022] First, this invention proposes a dynamic contrastive mask loss function that leverages the characteristics of contrastive learning to narrow the distance between similar masks in the feature space and widen the distance between dissimilar masks. The fundamental objective of this loss function is to maximize the mask size in the current training cycle. Mask of the previous training cycle The lower bound of mutual information is determined. Since directly optimizing the lower bound of mutual information is difficult, an approximation strategy is adopted, namely, optimizing it by maximizing the average log-likelihood of positive samples, as shown in the following formula: ; in, This represents the probability that the mask from the previous training cycle is a positive example of the mask from the current training cycle. represents a positive example; K This represents the total number of channels in the entire network. For the first One channel; Represents the cosine similarity distance; Temperature coefficient; k For the first One channel.

[0023] Secondly, by dynamically contrasting the mask loss, the system can perceive sample complexity and automatically learn a binary mask for each sample. To dynamically adjust the network structure and generate corresponding sub-networks based on the complexity of each sample, this invention further introduces an L1 regularization term and designs a sample complexity adaptive factor based on the aforementioned objective function. This factor can dynamically generate a binary mask that matches the complexity of a sample by automatically sensing its complexity. It adaptively mines network redundancy by adjusting the binary distribution of the mask and generates sub-networks that match the sample complexity, ultimately achieving differentiated pruning for samples of different complexities. Its calculation formula is shown below: ; Here, the cross-entropy loss used for classification tasks will be abbreviated as... Due to the complexity of the samples and There is a positive correlation, therefore it is necessary to Perform a reverse transformation, denoted as , and These represent its maximum and minimum values, respectively. The numerical value is positively correlated with the sample complexity; more complex samples correspond to larger values. The mask value tends to be closer to 1 in order to retain more channels and form more complex subnetworks. Simpler samples correspond to smaller... With the mask value approaching 0, more aggressive pruning is achieved, resulting in a more streamlined subnetwork structure. Therefore, channels corresponding to 0 can be safely removed without manually setting a pruning threshold, thus obtaining multiple sub-networks with different structures. Finally, the final expression for the Mask Contrastive Loss (MC Loss) proposed in this invention is as follows: ; in, For mask contrast loss function; Indicates the total number of samples; i For the firsti One sample; This is the weighting balance coefficient, used to control the pruning rate. For the first e The mask obtained from each training cycle.

[0024] II. Mutual Knowledge Distillation: Dynamic Pruning Based on Dataset The complexity of each sample generates a set of subnetworks with different structures. The structural complexity of subnetworks is positively correlated with the sample complexity, but they all originate from different subsets of the same large network, thus exhibiting high similarity in feature extraction capabilities. Based on this characteristic, these subnetworks can learn from and complement each other, forming a natural collaborative learning mechanism to obtain more robust representations. Therefore, a mutual knowledge distillation mechanism is constructed to compensate for the loss of key information caused by pruning. The specific process is as follows: Assuming two subnetworks and (abbreviated as) and To facilitate mutual learning, the probability of the predicted class is first obtained from each sub-network, using the following formula: ; in, For sub-networks Predicted as y The probability of the category; Subnetwork The output of the Softmax layer; and These represent the category and the total number of categories, respectively.

[0025] Traditional supervised loss is used to train the network to ensure it can correctly predict the labels of training samples. However, to further improve the network's generalization and robustness on test samples, this invention introduces another peer network. and its posterior probability As additional training experience, discriminative learning is conducted in the feature space, enabling each sub-network to acquire supplementary representational information from other sub-networks. Subsequently, the sub-networks... For a teacher network, knowledge transfer between subnetworks is achieved through knowledge distillation loss. Specifically, Kullback–Leibler (KL) divergence is used to measure this. and Two network prediction distributions and This consistency promotes collaborative learning and knowledge transfer among different subnetworks. The specific calculation formula is shown below: ; in, For sub-networks Predicted probability distribution Hezi Network Predicted probability distribution KL divergence; i For the first i One sample; N The total number of samples; and These represent the category and the total number of categories, respectively. Subnetwork Predict input samples For category The probability of; Subnetwork Predict input samples For category The probability of.

[0026] To demonstrate the mutual learning capabilities between subnetworks, not only should the subnetworks... Towards Transferring knowledge, while also allowing Towards Knowledge is transferred, forming a mutual distillation structure. Therefore, the final mutual distillation loss (MD Loss) is as follows: ; in, In order to be in N The loss function for mutual distillation of any two subnetworks in a subnetwork; N The total number of sub-networks; i For the first i Subnetwork; For the first Subnetwork; For sub-networks Predicted probability distribution Hezi Network Predicted probability distribution KL divergence; For sub-networks Predicted probability distribution Hezi Network Predicted probability distribution The KL divergence.

[0027] Example 1: Performance verification of the method of the present invention; To verify the effectiveness of the proposed deep neural network pruning method based on dynamic contrastive masking and knowledge distillation, this invention conducted pruning experiments on two datasets of different sizes, CIFAR-10 and ImageNet, targeting three mainstream deep neural network architectures: simple VGGNet (e.g., VGG16), residual ResNets (e.g., ResNet56, ResNet18, ResNet34), and the lightweight MobileNetV2. The experimental results are shown in Tables 1 and 2.

[0028] Table 1: Comparison of pruning results for VGG16, ResNet56, and MobileNetV2 on the CIFAR-10 dataset.

[0029] Table 2: Comparison of the results of pruning ResNet18 and ResNet34 rows on the ImageNet dataset.

[0030]

[0031] Note: In Tables 1 and 2, "DY" represents the dynamic pruning method, "Y" indicates that the method is dynamic pruning, and "N" indicates that the method is non-dynamic pruning; "Baseline Acc" represents the baseline model accuracy; "Pruned Acc" represents the pruned model accuracy; "Acc↓" represents the rate of decrease in pruned model accuracy compared to the baseline model accuracy (negative values ​​indicate an increase in accuracy); "FLOPs↓" represents the rate of decrease in floating-point operations of the pruned model compared to the baseline model; "Param↓" represents the rate of decrease in the number of parameters of the pruned model compared to the baseline model; "Pruned "Top-1" indicates the Top-1 accuracy of the pruned model (i.e., the accuracy rate at which the predicted class with the highest probability matches the true class); "Top-1↓" indicates the rate of decrease in Top-1 accuracy of the pruned model compared to the baseline model (negative values ​​indicate an increase in accuracy); "PrunedTop-5" indicates the Top-5 accuracy of the pruned model (i.e., the accuracy rate at which the top 5 predicted classes contain the true class); "Top-5↓" indicates the rate of decrease in Top-5 accuracy of the pruned model compared to the baseline model (negative values ​​indicate an increase in accuracy); "–" indicates that this value was not reported in the original paper; the bold values ​​in the table represent the best results for the corresponding metrics.

[0032] Experimental results demonstrate that the method presented in this invention outperforms state-of-the-art pruning techniques. For example, pruning ResNet56 on the CIFAR-10 dataset reduces floating-point operations (FLOPs) and parameters by 64.4% and 60.7%, respectively, while improving model accuracy by 0.18% compared to the baseline. On the large-scale ImageNet dataset, pruning ResNet34 achieves a 51.3% reduction in FLOPs, with only a 0.01% decrease in Top-1 accuracy and a 0.08% increase in Top-5 accuracy. These results indicate that this invention effectively exploits network redundancy in each sample, automatically identifies and removes redundant filters, and effectively compensates for the loss of representational information during pruning through a mutual distillation mechanism, thereby improving the overall performance of the pruned model.

[0033] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.

Claims

1. A deep neural network pruning method based on dynamic contrast masking and knowledge distillation, characterized in that, Includes the following steps: To construct a proxy task to guide mask learning for input samples, specifically, the training process is modeled as a time series, and positive and negative examples are constructed by combining the mask of the current training period sample with the mask of the previous training period sample. Design a dynamic contrastive mask loss function with an adaptive factor for sample complexity to generate a binary mask that matches the complexity of the input samples, thereby achieving differentiated pruning for different input samples; Based on the similarity of the sub-networks' feature extraction capabilities for input samples, a mutual knowledge distillation mechanism is constructed. The KL divergence is used to measure the consistency of the sub-networks' predicted distribution of input samples, thereby realizing knowledge transfer and collaborative learning among sub-networks and ensuring the performance of the model after pruning.

2. The deep neural network pruning method based on dynamic contrast masking and knowledge distillation according to claim 1, characterized in that, In the step of constructing the agent task, for the first... One channel, sample In training The mask obtained from each period is represented as follows: , and in the The mask obtained in each period Form a positive mask pair, the remaining 2 K A mask of -1 is considered a negative mask, where K This represents the total number of channels in the entire network.

3. The deep neural network pruning method based on dynamic contrast masking and knowledge distillation according to claim 1, characterized in that, The fundamental objective of the dynamic contrastive mask loss function is optimized by maximizing the average log-likelihood of positive samples, as shown in the formula: ； in, This is the mask for the current training cycle; The mask for the previous training cycle; This represents the probability that the mask from the previous training cycle is a positive example of the mask from the current training cycle. represents a positive example; K This represents the total number of channels in the entire network. For the first One channel; Represents the cosine similarity distance; Temperature coefficient; k For the first One channel.

4. The deep neural network pruning method based on dynamic contrast masking and knowledge distillation according to claim 3, characterized in that, The formula for calculating the sample complexity adaptive factor is as follows: ； in, This is an adaptive factor for sample complexity. Cross-entropy loss The reverse transformation; and They represent The maximum and minimum values.

5. The deep neural network pruning method based on dynamic contrast masking and knowledge distillation according to claim 4, characterized in that, After introducing an L1 regularization term, the dynamic contrast masking loss function yields the masking contrast loss: ； in, For mask contrast loss function; Indicates the total number of samples; i For the first i One sample; This is the weighting balance coefficient; For the first e The mask obtained from each training cycle.

6. The deep neural network pruning method based on dynamic contrast masking and knowledge distillation according to claim 1, characterized in that, The formula for obtaining the predicted class probability from the sub-network is: ； in, For sub-networks Predicted as y The probability of the category; Subnetwork The output of the Softmax layer; and These represent the category and the total number of categories, respectively.

7. The deep neural network pruning method based on dynamic contrast masking and knowledge distillation according to claim 6, characterized in that, The formula for measuring the consistency of the predicted distributions of two sub-networks using the KL divergence is as follows: ； in, For sub-networks Predicted probability distribution Hezi Network Predicted probability distribution KL divergence; i For the first i One sample; N The total number of samples; and These represent the category and the total number of categories, respectively. Subnetwork Predict input samples For category The probability of; Subnetwork Predict input samples For category The probability of.

8. The deep neural network pruning method based on dynamic contrast masking and knowledge distillation according to claim 7, characterized in that, The formula for the mutual distillation loss of the mutual knowledge distillation mechanism is as follows: ； in, In order to be in N The loss function for mutual distillation of any two subnetworks in a subnetwork; N The total number of sub-networks; i For the first i Subnetwork; For the first Subnetwork; For sub-networks Predicted probability distribution Hezi Network Predicted probability distribution The KL divergence.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the deep neural network pruning method based on dynamic contrast masking and knowledge distillation as described in any one of claims 1-8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the deep neural network pruning method based on dynamic contrast masking and knowledge distillation as described in any one of claims 1-8.

Images