Cnn structured sparsification method and system combining knowledge distillation and kernel similarity
By combining knowledge distillation and kernel similarity methods, and integrating Group Lasso loss and kernel similarity loss, a balance between sparsity and accuracy in CNN models is achieved. This solves the problem that increasing sparsity leads to a decrease in accuracy in existing technologies, and is suitable for neural network deployments on resource-constrained devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ANHUI UNIV
- Filing Date
- 2024-03-27
- Publication Date
- 2026-04-24
AI Technical Summary
Existing CNN structured sparsity methods improve model sparsity but lead to a significant drop in model accuracy, resulting in an imbalance between accuracy and sparsity.
We employ a joint knowledge distillation and kernel similarity approach, where Group Lasso loss and kernel similarity loss work together on the model. This is combined with collaborative learning between teacher and student models, and multiple rounds of training are conducted to achieve the optimal balance between sparsity and accuracy. This approach combines weakly structured sparsity, standard structured sparsity, and knowledge distillation.
While maintaining model accuracy, it achieves structured sparsity, enabling the deployment of neural networks on devices with limited hardware resources and ensuring the performance of the neural networks.
Smart Images
Figure CN118364871B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of neural network technology, and more specifically, to: 1. a CNN structured sparse method that combines knowledge distillation and kernel similarity, and 2. a CNN structured sparse system that combines knowledge distillation and kernel similarity. Background Technology
[0002] CNNs (Convolutional Neural Networks), as one of the most popular methods used in deep learning, play a leading role in the field of computer vision and have achieved a qualitative leap compared to traditional algorithms. CNNs have excellent information extraction capabilities and have shone brightly in applications ranging from basic image recognition and object classification to challenging computer vision tasks such as object detection, face tracking, and action recognition.
[0003] However, the high computational resource requirements severely hinder the deployment of CNNs on resource-constrained devices. To reduce the number of model parameters, model compression methods for CNNs have gained favor among experts. Currently, there are roughly four methods that can achieve similar effects: neuron pruning, low-rank decomposition, structured pruning, and structured sparsity.
[0004] Regarding structured sparsity, Wen Wei et al. ("Learning Structured Sparsity in Deep Neural Networks." ArXiv abs / 1608.03665(2016):n.pag.) proposed a structured sparse learning method (SSL) to regularize the filters, channels, filter shapes, and depth structure in deep neural networks. Further applying different Lasso penalties to the model can achieve structured sparsity of the network model. Their theory can force deep neural networks to dynamically learn more compact structures without incurring devastating accuracy losses. However, regrettably, in the verification process using CIFAR-10, the inventors found that using readily available libraries, the SSL method achieved acceleration of AlexNet convolutional layer computation relative to both CPU and GPU, approximately twice the acceleration of unstructured sparsity. But as sparsity increased, the model accuracy decreased by about 30%.
[0005] In other words, while existing SSL methods improve model sparsity, they often cause a significant drop in model accuracy, indicating an imbalance between accuracy and sparsity. Summary of the Invention
[0006] Therefore, it is necessary to address the imbalance between model accuracy and sparsity in existing SSL methods by providing a CNN structured sparsity method and system that combines knowledge distillation and kernel similarity.
[0007] This invention is achieved using the following technical solution:
[0008] In a first aspect, the present invention discloses a CNN structured sparsity method that combines knowledge distillation and kernel similarity, comprising the following steps:
[0009] Step 1: Obtain the sample dataset and divide it into training and test sets;
[0010] Step 2: Use the sample dataset to pre-train the original CNN model to obtain the pre-trained CNN model;
[0011] Step 3: Based on the pre-trained CNN model, perform multiple rounds of formal training using the training set until the model's accuracy and sparsity reach the optimal balance, thus obtaining the final lightweight model.
[0012] The nth round of formal training includes:
[0013] First, use the training set to perform the nth weakly structured sparsity on the teacher model after the (n-1)th round of sparsity to obtain the teacher model after the nth round of sparsity.
[0014] Then, place the teacher model after the nth round of sparseness in the teacher position and the student model after the (n-1)th round of distillation in the student position. Use the training set to perform superimposed standard structured sparse knowledge distillation to obtain the student model after the nth round of distillation.
[0015] Calculate the sparsity of the student model after the nth round of distillation, and use the test set to verify the accuracy of the student model after the nth round of distillation;
[0016] When n≥1 and n=1, the pre-trained CNN model is used as the teacher model after the 0th round of sparse training and the student model after the 0th round of distillation.
[0017] The strengths of both weakly structured sparsity and standard structured sparsity are based on the combined effect of Group Lasso loss and kernel similarity loss on the model; the strength of weakly structured sparsity is less than that of standard structured sparsity.
[0018] This joint knowledge distillation and kernel similarity CNN structured sparse method implements the method or process according to embodiments of this disclosure.
[0019] Secondly, this invention discloses a CNN structured sparse system based on joint knowledge distillation and kernel similarity, which uses the CNN structured sparse method based on joint knowledge distillation and kernel similarity from the first aspect.
[0020] The CNN structured sparse system that combines knowledge distillation and kernel similarity includes: a data acquisition module, a pre-training module, and a formal training module.
[0021] The data acquisition module acquires a sample dataset and divides it into training and test sets. The pre-training module uses the sample dataset to pre-train the original CNN model, resulting in a pre-trained CNN model. The formal training module performs multiple rounds of formal training on the pre-trained CNN model using the training set until the sparsity of the model and its accuracy on the test set reach an optimal balance, resulting in the final lightweight model.
[0022] This joint knowledge distillation and kernel similarity CNN structured sparse system implements the method or process according to embodiments of this disclosure.
[0023] Thirdly, the present invention discloses a readable storage medium. This readable storage medium stores computer program instructions that, when read and executed by a processor, perform the steps of the CNN structured sparse method based on joint knowledge distillation and kernel similarity as described in the first aspect.
[0024] Compared with the prior art, the present invention has the following beneficial effects:
[0025] This invention introduces knowledge distillation and, based on it, performs sparsification on the teacher and student models to different degrees. Furthermore, the sparsified loss function adds a function term constructed by kernel similarity, which can better maintain the accuracy performance of the original model while obtaining a sufficiently sparsity model. This not only facilitates the deployment of neural networks on devices with limited hardware resources, but also ensures the performance of the neural network. Attached Figure Description
[0026] Figure 1 This is a flowchart of the CNN structured sparse method that combines knowledge distillation and kernel similarity in Embodiment 1 of the present invention;
[0027] Figure 2 This is an information flow diagram of the CNN structured sparse method that combines knowledge distillation and kernel similarity in Embodiment 1 of the present invention;
[0028] Figure 3 This is a comparison chart of the accuracy of the four methods in Embodiment 3 of the present invention on the CIFAR-10 dataset;
[0029] Figure 4 This is a comparison chart of the sparsity of the four methods in Embodiment 3 of the present invention on the CIFAR-10 dataset. Detailed Implementation
[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0031] It should be noted that when a component is said to be "installed on" another component, it can be directly on the other component or it may be in a component that is centered on it. When a component is said to be "set on" another component, it can be directly set on the other component or it may also be in a component that is centered on it. When a component is said to be "fixed to" another component, it can be directly fixed to the other component or it may also be in a component that is centered on it.
[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.
[0033] Example 1
[0034] Please see Figure 1 , Figure 1 This is a simplified flowchart of a CNN structured sparse method that combines knowledge distillation and kernel similarity, as disclosed in Embodiment 1.
[0035] In summary, this method employs a combination of Group Lasso loss and kernel similarity loss to effectively remove redundant convolutional kernels in convolutional layers, achieving structured sparsity. Specifically, the Group Lasso construction function is used to set the weights in certain groups to zero to achieve sparsity; the kernel similarity construction function is used to constrain the similarity between different convolutional kernels within the same layer. Furthermore, to address the issue of excessive accuracy degradation with increasing sparsity, this method introduces knowledge distillation while maintaining the aforementioned structured sparsity, and designs a teacher-student co-learning approach. During training, the student model undergoes knowledge distillation with superimposed standard structured sparsity, while the teacher model is subjected to weak structured sparsity, guiding the student model to bridge the gap with the teacher model as much as possible.
[0036] For details, please refer to Figure 2 The CNN structured sparse method combining knowledge distillation and kernel similarity includes the following steps:
[0037] Step 1: Obtain the sample dataset and divide it into training and test sets.
[0038] As shown in the background section, existing SSL methods suffer from excessively reduced accuracy on CIFAR-10. Therefore, to verify the effectiveness of this method, the sample dataset can be CIFAR-10.
[0039] For ease of explanation, in this Example 1, the sample dataset uses CIFAR-10, which contains images of 10 types ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', and 'truck'). Each type contains 5000 training images and 1000 test images, and each image is a 32×32 three-channel color RGB image. Therefore, CIFAR-10 can be divided into training and test sets in a 5:1 ratio.
[0040] Of course, since this method is a general design, other public data sources can also be used for the sample dataset, such as CIFAR-100 and ImageNet. Furthermore, the ratio of training to test sets varies across different public datasets; the appropriate ratio should be chosen based on the specific circumstances.
[0041] Step 2: Use the sample dataset to pre-train the original CNN model to obtain the pre-trained CNN model.
[0042] Specifically, pre-training methods include:
[0043] The weights of the original CNN model are randomly initialized, and multiple rounds of training are performed on the training set until the accuracy of the model on the test set no longer increases, thus obtaining the pre-trained CNN model.
[0044] It is important to emphasize that the pre-training process does not include any structural sparsity operations, and the loss function L used for training is... pre It is built based on cross-entropy.
[0045] Step 3: Based on the pre-trained CNN model, perform multiple rounds of formal training using the training set until the sparsity of the model and its accuracy on the test set reach the optimal balance, thus obtaining the final lightweight model.
[0046] Step three is the core of this method, which combines joint knowledge distillation and kernel similarity.
[0047] Since the operations are the same for each round of formal training, we will take the nth round of formal training as an example for explanation (n≥1):
[0048] S31, first use the training set to perform the nth weakly structured sparsity on the teacher model after the (n-1)th round of sparsity, and obtain the teacher model after the nth round of sparsity;
[0049] S32, then place the teacher model after the nth round of sparseness in the teacher position and the student model after the (n-1)th round of distillation in the student position, and use the training set to perform superimposed standard structured sparse knowledge distillation to obtain the student model after the nth round of distillation;
[0050] S33. Calculate the sparsity of the student model after the nth round of distillation, and use the test set to verify the accuracy of the student model after the nth round of distillation.
[0051] It should be noted that when n=1 (i.e., the first round of formal training), the pre-trained CNN model is used as the teacher model after sparse training in round 0 and the student model after distillation in round 0.
[0052] The weak structured sparsity and standard structured sparsity mentioned in the training process are structured sparsity treatments with different intensities. The former is less than the latter, but both are based on the combined effect of Group Lasso loss and kernel similarity loss on the model.
[0053] For standard structured sparsity, its loss function L s for:
[0054]
[0055] In the formula, W represents the weight set, and L D (W) represents the cross-entropy loss on the training set, R(·) represents the unstructured regularization term of the weights, and R g (·) denotes a structured regularization term, k, k g S represents the penalty coefficient; k (W) represents the kernel similarity loss; k g The coefficient of the structured regularization term in the weights, ||·|| g W represents Group Lasso loss. (l) nl,:,:,: denoted as the nl-th convolutional kernel of the l-th layer; Nl represents the total number of convolutional kernels in the l-th layer; L represents the total number of layers; k represents the coefficient of the unstructured regularization term in the weights; Cl represents the total number of channels in the l-th convolutional layer; H represents the width and height of the convolutional kernel; cosine(·) represents the cosine similarity function.
[0056] L s It includes a Group Lasso loss function and a kernel similarity loss function, which can generate tendons and remove redundant convolution kernels and channels to achieve sparsity.
[0057] For weakly structured sparsity, its strength is less than that of standard structured sparsity, and its loss function is based on L. s Construct. Then, the loss function for the nth weakly structured sparse method. for:
[0058]
[0059] In the formula, L s The loss function represents the standard structured sparsity; α represents the intensity adjustment coefficient, α < 1; P n-1 Let α represent the teacher model after the (n-1)th round of sparsification. In other words, α is used to constrain the structured sparsity strength of the teacher model to reduce the learning difficulty of knowledge distillation and avoid situations where knowledge distillation cannot be effectively achieved when the performance gap between the student model and the teacher model is too large.
[0060] For knowledge distillation based on superimposed standard structured sparse knowledge, its loss function L... Q Backpropagation leads to knowledge distillation.
[0061] L Q This includes not only the Kullback-Leibler divergence (also known as relative entropy), which measures the similarity between the teacher and student models, but also the standard structured sparsity of the student model. Therefore, the loss function for the nth superposition of standard structured sparsity knowledge distillation is... for:
[0062]
[0063] In the formula, P n Let Q represent the teacher model after the nth round of sparse development. n-1 L represents the student model after the (n-1)th round of distillation; s L represents the loss function for standard structured sparsity; kd (P n Q n-1 ) represents P n and Q n-1 The Kullback-Leibler divergence; x represents the input feature of the model.
[0064] Based on the above During the nth round of training, both structured sparsity and knowledge distillation can be achieved simultaneously, and the overall loss function is...
[0065] After obtaining the student model after the nth round of distillation, the sparsity of the student model after the nth round of distillation is calculated, and the accuracy of the student model after the nth round of distillation is verified using the test set, so as to measure whether the student model after the nth round of distillation has achieved the optimal balance between sparsity and accuracy.
[0066] The methods for sparsity calculation include:
[0067] The total number of convolutional kernels in the student model after the nth round of distillation is:
[0068] Count the number of sparse convolution kernels in the student model after the nth round of distillation. In the sparse convolution kernel, all parameters are 0;
[0069] Calculate the sparsity of the student model after the nth round of distillation.
[0070] The optimal balance between sparsity and accuracy refers to achieving a balance between the model's usability and structural complexity. In this Example 1, the optimal balance means that the sum of sparsity and accuracy is maximized. Therefore, in step three, the sum of sparsity and accuracy of the student model after the nth round of distillation is calculated. If the sum of sparsity and accuracy of the student model after the nth round of distillation is maximized, it indicates that the optimal balance has been achieved, and the student model after the nth round of distillation can be used as the final lightweight model.
[0071] Of course, the optimal balance can also be achieved by using a weighted sum of sparsity and accuracy, taking into account different application scenarios and assigning appropriate weights.
[0072] This method achieves significant accuracy gains with relatively small sparsity loss, resulting in a lightweight model with high precision. It is of great significance in applications requiring high recognition accuracy, such as disease detection, disease investigation, and construction flaw detection; it also helps improve the performance of embedded devices that typically deploy neural networks.
[0073] Example 2
[0074] This embodiment 2 discloses a CNN structured sparse system that combines knowledge distillation and kernel similarity, which uses the CNN structured sparse method of combining knowledge distillation and kernel similarity from embodiment 1.
[0075] The CNN structured sparse system that combines knowledge distillation and kernel similarity includes: a data acquisition module, a pre-training module, and a formal training module.
[0076] The data acquisition module is configured to acquire a sample dataset and divide it into a training set and a test set. The pre-training module is configured to pre-train the original CNN model using the sample dataset to obtain a pre-trained CNN model. The formal training module is configured to perform multiple rounds of formal training on the pre-trained CNN model using the training set until the sparsity of the model and its accuracy on the test set reach an optimal balance, thus obtaining the final lightweight model.
[0077] Since this CNN structured sparse system based on joint knowledge distillation and kernel similarity uses the CNN structured sparse method based on joint knowledge distillation and kernel similarity in Example 1, it also has the same effect as Example 1, and will not be repeated here.
[0078] Example 3
[0079] Example 3 presents a simulation verification and comparison of the four methods on the CIFAR-10 dataset:
[0080] Method 1 uses only the pre-trained CNN model;
[0081] Method 2 employs the SSL method mentioned in the background section;
[0082] Method 3 involves adding a kernel similarity loss term to the loss function of the SSL method;
[0083] Method 4 adopts the method of Example 1.
[0084] See results Figure 3 , Figure 4 Therefore, we can conclude that:
[0085] Method 1 has the highest accuracy at 88%, but the lowest sparsity, close to 0, indicating that the pre-trained model has almost no effect on structured sparsity.
[0086] Method 2 improves sparsity to 6%, but its accuracy drops significantly to only 78.4%, reflecting the problem mentioned in the background art.
[0087] Method 3 improved sparsity to 11.5% and accuracy to 79.2%, indicating that kernel similarity loss can effectively constrain the similarity between convolution kernels, further improve sparsity, and also improve accuracy.
[0088] The accuracy of Method 4 increased from 79.2% to 84%—an improvement of about 5%, while the sparsity only decreased by about 1%, indicating that Method 4 can achieve a greater improvement in accuracy with a smaller loss of sparsity.
[0089] In summary, the method in Example 1 can not only effectively achieve structured sparsity, but also ensure the accuracy of the model.
[0090] Example 4
[0091] This embodiment 4 discloses a readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the steps of the CNN structured sparse method of joint knowledge distillation and kernel similarity in embodiment 1 are performed.
[0092] When applying the method of Example 1, it can be applied in the form of software, such as by designing it as a program that can run independently on a computer-readable storage medium. The computer-readable storage medium can be a USB flash drive, designed as a USB security token, and the program can be designed to start the entire method through an external trigger.
[0093] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0094] This embodiment 4 also provides a computer terminal, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the CNN structured sparse method based on joint knowledge distillation and kernel similarity as described in Embodiment 1.
[0095] The method in Example 1 can be applied in software form, such as by designing it as a standalone program and installing it on a computer terminal, which can be a computer, smartphone, control system, or other IoT device. Alternatively, the method in Example 1 can be designed as an embedded program and installed on a computer terminal, such as on a microcontroller.
[0096] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.
Claims
1. A CNN structured sparse method combining knowledge distillation and kernel similarity, characterized in that, Includes the following steps: Step 1: Obtain the sample dataset and divide it into a training set and a test set; the sample dataset can be any one of CIFAR-10, CIFAR-100, or ImageNet. Step 2: Use the sample dataset to pre-train the original CNN model to obtain the pre-trained CNN model; Step 3: Based on the pre-trained CNN model, perform multiple rounds of formal training using the training set until the sparsity of the model and its accuracy on the test set reach the optimal balance, thus obtaining the final lightweight model. Among them, the n The formal training round includes: First use the training set on the first n -1 rounds of sparse teacher model for the first round n The second weakly structured sparsity yields the first... n The teacher model after round sparsification; Then the first n The teacher model after round sparsening is placed in the teacher position, and the first... n The student model after the -1st round of distillation is placed in the student position, and the training set is used for the -th round of distillation. n The first superposition of standard structured sparse knowledge distillation yields the th n Student model after round distillation; Calculate the first n The sparsity of the student model after round distillation is evaluated, and the test set is used to verify the sparsity of the first round. n The accuracy of the student model after round distillation; n ≥1; and n When =1, the pre-trained CNN model is used as the teacher model after 0th round of sparse generation and the student model after 0th round of distillation. The strengths of both weakly structured sparsity and standard structured sparsity are based on the combined effect of Group Lasso loss and kernel similarity loss on the model; the strength of weakly structured sparsity is less than that of standard structured sparsity. No. n Second-weak structured sparse loss function for: ； In the formula, L s The loss function representing standard structured sparsity; α Indicates the strength adjustment factor. α <1; P n-1 Indicates the first n -1 rounds of sparse teacher model; No. n Loss function of superimposed standard structured sparse knowledge distillation for: ； In the formula, P n Indicates the first n The teacher model after round sparsening, Q n-1 Indicates the first n Student model after -1 round of distillation; L s The loss function representing standard structured sparsity; express P n and Q n-1 Kullback-Leibler divergence; x Represents the input features of the model; Standard structured sparse loss function L s for: ； In the formula, W Represents the set of weights. L D ( W ) represents the cross-entropy loss on the training set. R (.) denotes the unstructured regularization term for the weights. R g (.) represents a structured regularization term. k , k g Indicates the coefficient of the penalty term; S k ( W ) represents the loss of kernel similarity; k g This represents the coefficient of the structured regularization term in the weights. Indicates Group Lasso loss. Indicates the first l The first layer nl One convolutional kernel; Nl Indicates the first l Total number of convolutional kernels per layer; L Indicates the total number of floors; k This represents the coefficient of the unstructured regularization term in the weights. Cl Indicates the first l The total number of channels in a convolutional layer; H Indicates the length and width of the convolution kernel; cosine (.) represents the cosine similarity function.
2. The CNN structured sparse method based on joint knowledge distillation and kernel similarity according to claim 1, characterized in that, In step two, the pre-training methods include: The weights of the original CNN model are randomly initialized, and multiple rounds of training are performed on the training set until the accuracy of the model on the test set no longer increases, thus obtaining the pre-trained CNN model.
3. The CNN structured sparse method based on joint knowledge distillation and kernel similarity according to claim 1, characterized in that, Methods for sparsity calculation include: Statistics n The total number of convolutional kernels in the student model after round distillation is ; Statistics n The number of sparse convolution kernels in the student model after round distillation In sparse convolution kernels, all parameters are 0. Calculate the first n Sparsity of the student model after round distillation .
4. The CNN structured sparse method based on joint knowledge distillation and kernel similarity according to claim 3, characterized in that, In step three, calculate the first... n The sum of sparsity and accuracy of the student model after round distillation; if the first... n The optimal balance is achieved when the sum of sparsity and accuracy of the student model after round distillation reaches its maximum.
5. A CNN-structured sparse system that combines knowledge distillation and kernel similarity, characterized in that, It uses the CNN structured sparsity method of joint knowledge distillation and kernel similarity as described in any of claims 1-4; The CNN structured sparse system based on joint knowledge distillation and kernel similarity includes: The data acquisition module is used to acquire sample datasets and divide them into training and test sets; The pre-training module is used to pre-train the original CNN model using a sample dataset to obtain a pre-trained CNN model. as well as The formal training module, based on the pre-trained CNN model, uses the training set to perform multiple rounds of formal training until the sparsity of the model and its accuracy on the test set reach the optimal balance, thus obtaining the final lightweight model.
6. A readable storage medium, characterized in that, The readable storage medium stores computer program instructions that are read and executed by a processor to perform the steps of the CNN structured sparse method for joint knowledge distillation and kernel similarity as described in any of claims 1-4.