A model pruning method and system based on model structural redundancy

Abstract

This invention discloses a model pruning method and system based on model structural redundancy, relating to the fields of computer vision and neural network technology. The method includes: S1: training an image classification model as a sample model; S2: extracting model feature representations to obtain intermediate representations of the sample model, and setting a model structural redundancy threshold; S3: calculating the model structural redundancy of adjacent layers; S4: pruning the sample model based on the calculated model structural redundancy and the set threshold to obtain a pruned model; S5: applying the sample model and the pruned model to a subsequent classification task and comparing their classification accuracy. This invention proposes a new testing metric for deep learning models, providing a quantitative assessment of model structural redundancy, and designs a novel layer pruning method to eliminate structural redundancy while obtaining a pruned model with high accuracy.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and neural network technology, and more specifically to a model pruning method and system based on model structural redundancy. Background Technology

[0002] A neural network is a mathematical or computational model that mimics the structure and function of biological neural networks. It is a non-linear statistical data modeling tool composed of a large number of interconnected artificial neurons performing computations. Neural networks are often used to model complex relationships between inputs and outputs or to explore patterns in data. Deep neural networks have achieved great success in many applications, including image understanding, object detection, and autonomous vehicles. However, this success comes at a cost, as training and using them requires significant computational and storage resources, making them very expensive.

[0003] To reduce the cost of deep learning applications, an intuitive approach is to reduce model size. Ideally, the model structure should be as simple as possible to meet practical requirements. To achieve this, the machine learning community primarily employs model pruning methods. Model pruning removes redundant parts from the model structure based on the contribution of different levels to model inference. A key challenge in reducing model size is effectively identifying redundant parts in the model structure. Existing model pruning techniques can only measure model performance from different perspectives, but cannot quantitatively measure the remaining redundancy. In other words, they cannot describe our progress towards obtaining a model with minimal redundancy.

[0004] This invention focuses on the problem that previous pruning models could not accurately quantify redundancy. CN202211264295.4 describes a convolutional neural network pruning method based on distribution differences. This method extracts the feature matrix of the intermediate layers of the model, then calculates the distribution difference between the feature matrix output by the convolutional layer and the label features by combining the L1 norm of the convolutional kernel weights, obtaining a score, and pruning convolutional kernels below a threshold. CN202210177728.6 describes a deep convolutional neural network pruning method and system based on co-evolution. This method uses an evolutionary algorithm to optimize the network structure to be pruned in parallel within each group. After completing the intra-group evolution, the pruning results corresponding to the best individuals in the last generation of each group are concatenated and retrained until the maximum number of iterations is reached. CN202110818395.6 describes an automated convolutional neural network pruning method based on average rank importance ranking, which... Images from the training dataset are pre-trained to obtain the average rank of the output feature maps of each convolutional kernel and sort them. The sorting result is used as the basis for the judgment of the sensitive layer. Automated model pruning is achieved through reinforcement learning. CN202010066747.2 A convolutional neural network pruning method based on feature map sparsity adds L1 or L2 regularization to the feature maps after the activation layer in the loss function, so that the corresponding feature map channels have different sparsities. The convolutional kernels corresponding to the corresponding channels are pruned according to the sparsity. The accuracy of the pruned network is restored and the pruning rate is adjusted. Multiple iterations are performed to achieve maximum pruning. Current pruning techniques obtain the feature representation of the intermediate layers of the pre-trained model and calculate different parameters on the feature maps to achieve pruning. During the training process, the model to be pruned needs to be repeatedly trained and iterated.

[0005] It is evident that existing model pruning methods are complex in structure, cumbersome in steps, time-consuming in training, and require high computing power. Therefore, designing a model pruning technique that is simple in steps and computationally efficient is a key and challenging research area. Summary of the Invention

[0006] In view of this, the present invention provides a model pruning method and system based on model structural redundancy. It uses a new test metric to reflect the structural redundancy of deep learning models, provides a quantitative assessment of the degree of model structural redundancy, and solves the problem of large computational and storage costs required for training and using deep neural networks.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A model pruning method based on model structural redundancy includes the following steps:

[0009] S1: Train the image classification model as a sample model;

[0010] S2: Extract the model feature representation to obtain the intermediate representation of the sample model, and set the model structure redundancy threshold;

[0011] S3: Calculate the redundancy of the model structure in adjacent layers;

[0012] S4: Prune the sample model based on the calculated model structure redundancy and the set model structure redundancy threshold to obtain the pruned model.

[0013] Optionally, in S1, training the image classification model specifically includes the following steps:

[0014] S11: Select four image datasets, including CIFAR10, CIFAR100, ImageNet, and ImageNet16-120;

[0015] S12: Design the model parameters using stochastic gradient descent as the optimization model;

[0016] S13: Input the obtained set of samples into the optimization model to obtain image classification models VGGs and ResNets.

[0017] Optional, design model parameters, specifically:

[0018] The initial learning rate was 0.01, the batch size was 256, the weight decay was 0.0005, and the momentum was 0.9.

[0019] Optionally, the specific steps of S2 include:

[0020] S21: In an optimized DNN with fixed sample input parameters, the feature representation of the neural network is as follows:

[0021]

[0022] Where N represents the sample batch size, L is the number of layers, and C... i W is the number of channels in the i-th layer. i and H i Let F be the width and height of the feature map of the i-th layer, respectively; i This represents the feature representation of the i-th layer of the neural network;

[0023] Dimensionality reduction using mapping functions:

[0024]

[0025] Where M i =C i ×W i ×H i To obtain a set of feature representations F = {F1, F2, ..., F...} L};fθ This represents the mapping function used for dimensionality reduction;

[0026] S22: Set the redundancy threshold μ for the model structure, where μ is a manually set parameter.

[0027] Optionally, in S3, the redundancy of the model structure in adjacent layers is calculated, specifically including the following steps:

[0028] S31: Calculate the Gram matrix S i S j :

[0029]

[0030]

[0031] j = i + 1 (5);

[0032] Where F T S represents the transpose of matrix F, where i and j are the layer numbers of the features, and S i S j Let S be the Gram matrix of features from adjacent layers, where S ∈ R N×N The size of the matrix is ​​determined solely by the sample batch size N;

[0033] S32: Calculate the unbiased Hilbert-Schmidt independence criterion HSIC1(S i ,S j The formula is as follows:

[0034]

[0035] Where tr represents the trace of the matrix. By S i S j The matrix is ​​obtained by setting the diagonal elements to 0, where 1 represents an N×N matrix of all 1s.

[0036] S33: Calculate the redundancy of the model structure using unbiased CKA. unbiased (S i ,S j The formula is as follows:

[0037]

[0038] Among them, CKA unbiased (S i ,S j The value of ) is in the range of 0 to 1.

[0039] Optionally, in S4, the sample model is pruned based on the calculated model structure redundancy and the set model structure redundancy threshold, specifically including the following steps:

[0040] S41: Calculate the model structure redundancy of adjacent layers in the order of i from 1 to L-1. For layers i and j whose model structure redundancy is less than the set model structure redundancy threshold μ, retain the shallower layer i and delete layer j.

[0041] S42: For the retained i-th layer, add it to the pruned model;

[0042] S43: Output the pruned model.

[0043] Optionally, the method further includes S5: applying the sample model and the pruned model to a subsequent classification task, and comparing their classification accuracy, specifically:

[0044] S51: Use four image datasets, including CIFAR10, CIFAR100, ImageNet, and ImageNet16-120;

[0045] S52: Input the obtained image dataset into the sample model and the pruned model, obtain the model prediction accuracy respectively, and compare and evaluate the pruning accuracy.

[0046] A system for implementing the model pruning method based on model structure redundancy described above includes a sample model acquisition module, a pruning model input acquisition module, a model structure redundancy calculation module, a redundancy layer identification and pruning module, and an image classification module connected in sequence.

[0047] The sample model acquisition module is used to train the image classification model and use it as a sample model.

[0048] The pruning model input acquisition module is used to extract model feature representations, obtain intermediate representations of sample models, and set model structure redundancy thresholds.

[0049] The model structure redundancy calculation module is used to calculate the model structure redundancy of adjacent layers;

[0050] The redundancy layer identification and pruning module is used to prune the sample model based on the calculated model structure redundancy and the set model structure redundancy threshold to obtain the pruned model.

[0051] The image classification module is used to apply the sample model and the pruned model to subsequent classification tasks and compare their classification accuracy.

[0052] As can be seen from the above technical solution, compared with the prior art, the present invention provides a model pruning method and system based on model structural redundancy, which has the following beneficial effects:

[0053] (1) This invention quantifies the redundancy of the model structure, prunes the redundant layers of the model based on the calculated redundancy, and finally evaluates the pruning accuracy. This method can greatly improve pruning efficiency and reduce the pruning process. At the same time, the model can improve the efficiency of image classification and the classification results are more accurate.

[0054] (2) This invention proposes a new test metric for deep learning models, called model structure redundancy, which provides a quantitative assessment of the degree of model structure redundancy;

[0055] (3) This invention utilizes model structure redundancy to propose a redundancy testing framework for deep learning developers. It provides an end-to-end solution for optimizing model structure and allows developers to intuitively understand the degree of redundancy optimization.

[0056] (4) This invention uses model structure redundancy to guide layer pruning, and a large number of experiments have shown that layer pruning guided by model structure redundancy can achieve a large speedup ratio with minimal impact on performance. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0058] Figure 1 A flowchart of the model pruning method based on model structural redundancy provided by the present invention;

[0059] Figure 2 The diagram shows the structure of the model pruning system based on model redundancy provided by this invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] This invention aims to address the problem of high computational and storage costs required for training and using deep neural networks. Existing pruning models can only measure model performance from different perspectives, but cannot quantitatively measure remaining redundancy. This invention proposes a new test metric to reflect the structural redundancy of deep learning models, called Model Structural Redundancy Ratio (MSRS), which provides a quantitative assessment of the degree of model structural redundancy.

[0062] This invention proposes a redundancy testing framework for deep learning developers, providing an end-to-end solution for optimizing model structure. For pruning large-scale models, this invention proposes a novel layer pruning method based on model structural redundancy to eliminate structural redundancy. This method prunes the model according to specific numerical values, eliminating the need for subsequent accuracy recovery processes and directly obtaining a high-accuracy classifier model.

[0063] Specifically, this invention prunes redundant layers of the model based on the introduced model structural redundancy score and the calculated model structural redundancy, and finally evaluates the pruning accuracy. The model using this method can improve classification efficiency and achieve more accurate classification results.

[0064] Reference Figure 1 This invention discloses a model pruning method based on model structural redundancy, specifically including the following steps:

[0065] S1: Train the image classification model as a sample model;

[0066] S2: Extract model feature representations to obtain intermediate representations (IRs) of the sample model, and set the model structure redundancy threshold;

[0067] S3: Calculate the redundancy of the model structure in adjacent layers;

[0068] S4: Prune the sample model based on the calculated model structure redundancy and the set model structure redundancy threshold to obtain the pruned model.

[0069] Furthermore, in S1, training the image classification model specifically includes the following steps:

[0070] S11: Select four image datasets, including CIFAR10, CIFAR100, ImageNet, and ImageNet16-120;

[0071] S12: Design the model parameters using stochastic gradient descent as the optimization model;

[0072] S13: Input the obtained set of samples into the optimization model to obtain image classification models VGGs (Visual Graphics Generator) and ResNets (Residual Networks).

[0073] Furthermore, the model parameters are designed as follows:

[0074] The initial learning rate was 0.01, the batch size was 256, the weight decay was 0.0005, and the momentum was 0.9.

[0075] Furthermore, the specific steps of S2 include:

[0076] S21: In optimized DNNs (Deep Neural Networks) with fixed input parameters for the acquired samples, the feature representation of the neural network is as follows:

[0077]

[0078] Where N represents the sample batch size, L is the number of layers, and C... i W is the number of channels in the i-th layer. i and H i Let F be the width and height of the feature map of the i-th layer, respectively; i This represents the feature representation of the i-th layer of the neural network;

[0079] Dimensionality reduction using mapping functions:

[0080]

[0081] Where M i =C i ×W i ×H i To obtain a set of feature representations F = {F1, F2, ..., F...} L};f θ This represents the mapping function used for dimensionality reduction;

[0082] S22: Set the redundancy threshold μ for the model structure, where μ is a manually set parameter.

[0083] Furthermore, in S3, the redundancy of the model structure in adjacent layers is calculated, specifically including the following steps:

[0084] S31: Calculate the Gram matrix S i S j :

[0085]

[0086]

[0087] j = i + 1 (5);

[0088] Where F T S represents the transpose of matrix F, where i and j are the layer numbers of the features, and S i S j Let S be the Gram matrix of features from adjacent layers, where S ∈ R N×N The size of the matrix is ​​determined solely by the sample batch size N;

[0089] S32: Calculate the unbiased Hilbert-Schmidt independence criterion HSIC1(S i ,S j The formula is as follows:

[0090]

[0091] Where tr represents the trace of the matrix. By S i S j The matrix is ​​obtained by setting the diagonal elements to 0, where 1 represents an N×N matrix of all 1s.

[0092] S33: Calculate the redundancy of the model structure using unbiased CKA. unbiased (S i ,S j The formula is as follows:

[0093]

[0094] Among them, CKA unbiased (S i ,S j The value of ) is in the range of 0 to 1.

[0095] Furthermore, in S4, the sample model is pruned based on the calculated model structure redundancy and the set model structure redundancy threshold, specifically including the following steps:

[0096] S41: Calculate the model structure redundancy of adjacent layers in the order of i from 1 to L-1. For layers i and j with a model structure redundancy less than the set model structure redundancy threshold μ, retain the shallower layer i and delete layer j to ensure the accuracy of the prediction as much as possible.

[0097] S42: For the retained i-th layer, add it to the pruned model;

[0098] S43: Output the pruned model.

[0099] Furthermore, the method also includes S5: applying the sample model and the pruned model to a subsequent classification task, and comparing their classification accuracy, specifically:

[0100] S51: Use four image datasets, including CIFAR10, CIFAR100, ImageNet, and ImageNet16-120;

[0101] S52: Input the obtained image dataset into the sample model and the pruned model, obtain the model prediction accuracy respectively, and compare and evaluate the pruning accuracy.

[0102] and Figure 1 Corresponding to the method described above, this embodiment of the invention also provides a model pruning system based on model structural redundancy, used for... Figure 1 The specific implementation of the method, a model pruning system based on model structural redundancy provided in this embodiment of the invention, can be applied to computer terminals or various mobile devices, and its structural schematic diagram is shown below. Figure 2 As shown, it specifically includes a sample model acquisition module, a pruning model input acquisition module, a model structure redundancy calculation module, a redundancy layer identification and pruning module, and an image classification module connected in sequence.

[0103] The sample model acquisition module is used to train the image classification model and use it as a sample model.

[0104] The pruning model input acquisition module is used to extract model feature representations, obtain intermediate representations of sample models, and set model structure redundancy thresholds.

[0105] The model structure redundancy calculation module is used to calculate the model structure redundancy of adjacent layers;

[0106] The redundancy layer identification and pruning module is used to prune the sample model based on the calculated model structure redundancy and the set model structure redundancy threshold to obtain the pruned model.

[0107] The image classification module is used to apply the sample model and the pruned model to subsequent classification tasks and compare their classification accuracy.

[0108] This embodiment quantifies the redundancy of the model structure, prunes redundant layers based on the calculated redundancy, and finally evaluates the pruning accuracy. This method can significantly improve pruning efficiency and reduce the pruning process. At the same time, it can improve the efficiency of image classification and make the classification results more accurate.

[0109] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0110] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model pruning method based on model structural redundancy, characterized in that, Includes the following steps: S1: Train the image classification model as a sample model; S2: Extract the model feature representation to obtain the intermediate representation of the sample model, and set the model structure redundancy threshold; S3: Calculate the redundancy of the model structure between adjacent layers, which includes the following steps: S31: Calculate the Gram matrix S i S j : j = i + 1 (5); Where F T S represents the transpose of matrix F, where i and j are the layer numbers of the features, and S i S j Let S be the Gram matrix of features from adjacent layers, where S ∈ R N×N The size of the matrix is ​​determined solely by the sample batch size N; S32: Calculate the unbiased Hilbert-Schmidt independence criterion HSIC1(S i ,S j The formula is as follows: Where tr represents the trace of the matrix. By S i S j The matrix is ​​obtained by setting the diagonal elements to 0, where 1 represents an N×N matrix of all 1s. S33: Calculate the redundancy of the model structure using unbiased CKA. unbiased (S i ,S j The formula is as follows: Among them, CKA unbiased (S i ,S j The value of ) is in the range of 0 to 1; S4: Prune the sample model based on the calculated model structure redundancy and the set model structure redundancy threshold to obtain the pruned model.

2. The model pruning method based on model structural redundancy according to claim 1, characterized in that, In S1, training the image classification model specifically includes the following steps: S11: Select four image datasets, including CIFAR10, CIFAR100, ImageNet, and ImageNet16-120; S12: Design the model parameters using stochastic gradient descent as the optimization model; S13: Input the obtained set of samples into the optimization model to obtain image classification models VGGs and ResNets.

3. The model pruning method based on model structural redundancy according to claim 2, characterized in that, The design model parameters are as follows: The initial learning rate was 0.01, the batch size was 256, the weight decay was 0.0005, and the momentum was 0.

9.

4. The model pruning method based on model structural redundancy according to claim 1, characterized in that, The specific steps of S2 include: S21: In an optimized DNN with fixed sample input parameters, the feature representation of the neural network is as follows: Where N represents the sample batch size, L is the number of layers, and C... i W is the number of channels in the i-th layer. i and H i Let F be the width and height of the feature map of the i-th layer, respectively; i This represents the feature representation of the i-th layer of the neural network; Dimensionality reduction using mapping functions: Where M i =C i ×W i ×H i To obtain a set of feature representations F = {F1, F2, ..., F...} L };f θ This represents the mapping function used for dimensionality reduction; S22: Set the redundancy threshold μ for the model structure, where μ is a manually set parameter.

5. The model pruning method based on model structural redundancy according to claim 1, characterized in that, In S4, the sample model is pruned based on the calculated model structure redundancy and the set model structure redundancy threshold. This includes the following steps: S41: Calculate the model structure redundancy of adjacent layers in the order of i from 1 to L-1. For layers i and j whose model structure redundancy is less than the set model structure redundancy threshold μ, retain the shallower layer i and delete layer j. S42: For the retained i-th layer, add it to the pruned model; S43: Output the pruned model.

6. The model pruning method based on model structural redundancy according to claim 1, characterized in that, The method further includes S5: applying the sample model and the pruned model to subsequent classification tasks, and comparing their classification accuracy, specifically: S51: Use four image datasets, including CIFAR10, CIFAR100, ImageNet, and ImageNet16-120; S52: Input the obtained image dataset into the sample model and the pruned model, obtain the model prediction accuracy respectively, and compare and evaluate the pruning accuracy.

7. A system for implementing the model pruning method based on model structural redundancy as described in any one of claims 1-6, characterized in that, It includes a sample model acquisition module, a pruning model input acquisition module, a model structure redundancy calculation module, a redundancy layer identification and pruning module, and an image classification module connected in sequence. The sample model acquisition module is used to train the image classification model and use it as a sample model. The pruning model input acquisition module is used to extract model feature representations, obtain intermediate representations of sample models, and set model structure redundancy thresholds. The model structure redundancy calculation module is used to calculate the model structure redundancy of adjacent layers; The redundancy layer identification and pruning module is used to prune the sample model based on the calculated model structure redundancy and the set model structure redundancy threshold to obtain the pruned model. The image classification module is used to apply the sample model and the pruned model to subsequent classification tasks and compare their classification accuracy.

Images