A model compression method based on linear dependence of feature maps

Abstract

The application discloses a model compression method based on feature map linear dependence, comprising the following steps: counting the number of convolution layers in a neural network model and the number of channels in each convolution layer; obtaining the feature map output by each channel by inputting a small amount of data sets; calculating the linear dependence independence of each channel in the convolution layer to which the channel belongs based on the information entropy of the feature map, and taking the linear dependence independence as local importance; based on the genetic evolution algorithm, calculating the global scale coefficient and the global deviation coefficient of each convolution layer, and converting the local importance into global importance; according to a set target compression rate, calculating a corresponding threshold value, and removing the channels with global importance less than the threshold value, so as to obtain an optimal subnetwork; and performing adaptive weighted multi-network joint parallel training on the obtained optimal subnetwork, and restoring the performance of the network model.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a model compression method based on linear dependence of feature maps. Background Technology

[0002] With the development of deep learning, convolutional neural networks have been widely used in computer vision tasks such as image segmentation, object detection, and image super-resolution. However, this has also brought some problems. While pursuing superior performance, researchers have overlooked the excessively large number of parameters and floating-point operations required for these models. Due to their large parameter count and computational complexity, these deep learning network models are difficult to deploy on micro-devices with limited computing power and memory, restricting their application in daily life and industrial production. To accelerate the practical deployment of the latest high-performance network models, researchers have begun to focus on neural network model compression. Related research shows that for these over-parameterized network models, removing redundant parameters does not significantly degrade performance. Currently, many model compression methods have been proposed. Commonly used methods mainly include network pruning, low-rank decomposition, knowledge distillation, quantization, and designing compact networks. Among these, network pruning is a widely studied and popular model compression technique in both industry and academia.

[0003] Network pruning, from the perspective of pruning granularity, can be broadly divided into weight pruning and channel pruning. Weight pruning has a larger granularity than channel pruning. Weight pruning focuses on removing unimportant parameters from the weights. Although this method can achieve high compression rates, it requires additional hardware or software assistance, which often fails to meet practical requirements. Channel pruning, on the other hand, uses the channels of each layer as the smallest granularity, calculates the importance of each channel based on corresponding judgment criteria, and removes those unimportant channels from the network model.

[0004] Determining the importance of channels is the focus of channel pruning research. Currently, most methods assess channel importance from the perspective of the channel itself, such as calculating the l-norm of the channel parameters. However, the useful information extracted from a channel may differ depending on the input. Therefore, simply judging channel importance from the perspective of model parameters is inaccurate. Most current model pruning algorithms are based on prior experience, pre-setting the pruning rate for each layer, but the resulting subnetwork model is not necessarily optimal. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a neural network model compression method based on linear dependence of feature maps with high pruning accuracy.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] This invention is a method for compressing neural network models based on linear dependence of feature maps, comprising the following steps:

[0008] Step 1: Set up a GPU-based PyTorch deep learning framework runtime environment;

[0009] Step 2: For the trained neural network model to be compressed, obtain the number of convolutional layers in the model and the number of channels in each convolutional layer. Randomly sample a small number of samples from its training dataset and input them into the network model. Extract the feature map output by each channel in the network model.

[0010] Step 3: Based on linear dependency independence, process the extracted feature maps and use the average value as the independence of the corresponding channels;

[0011] Step 4: Treat the independence of channels as the local importance of channels. Based on the genetic evolution algorithm, the global scale coefficient and global bias coefficient of each convolutional layer in the network model can be calculated through multiple iterations.

[0012] Step 5: Using the global scale coefficient and global bias coefficient, the local importance of the channels is converted into global importance. Based on the set target pruning rate, the global importance scores of the channels in the network model are sorted to obtain the threshold. Then, channels with scores below the threshold are removed.

[0013] Step 6: Using the training dataset, perform multi-network joint parallel training on the optimal sub-network obtained by pruning.

[0014] A further improvement of this invention lies in the following: In step 3, the linear dependency independence of the corresponding channels is calculated based on the output feature map. Specifically, the information entropy of the feature map is first calculated, then the linear dependency auxiliary two-dimensional matrix is ​​multiplied with the feature map to calculate its information entropy, and then the two are subtracted. The calculation formula is as follows:

[0015]

[0016] For E(X), the range of element values ​​in the parameter matrix X to be calculated is uniformly divided into N intervals, and then the probability p of each interval is calculated. k Let be the probability distribution of the k-th interval. Let represent the linear dependency auxiliary two-dimensional matrix of the l-th layer, i.e., all columns except the i-th column are 1. A represents the linear dependency independence of the i-th channel in the l-th convolutional layer of the network model. l This is the feature map output by the l-th convolutional layer in the model.

[0017] A further improvement of this invention is that: in step 4, the global scale coefficient is expressed as... The global deviation coefficient is expressed as The specific steps for step 4 are as follows:

[0018] Step 4.1: Treat a {α,β} as a population, then randomly initialize multiple populations and form a population pool;

[0019] Step 4.2: Randomly select a small number of populations from the population pool as mutation samples. Choose the best-performing population from these samples as the mutation target. If the new population obtained through mutation performs better than the worst-performing population in the sample pool, replace the worst-performing population with the new population. Through multiple iterations, gradually obtain superior populations. The mutation process is as follows:

[0020]

[0021] Where Std(·) represents the formula for calculating the standard deviation. This represents the formula for calculating the normal distribution, M. l This represents the weight parameter matrix of the l-th convolutional layer in the network model, where there are L convolutional layers. l Let σ be the standard deviation of the weight parameters of the l-th layer, σ be a random number, and α be a variable. l β l These are the global scale coefficient and global bias coefficient of the l-th layer before mutation. These are the global scale coefficient and global bias coefficient of the l-th layer after mutation;

[0022] Step 4.3: After the iteration is complete, select the best population from the population pool, which is the global scale coefficient and global bias coefficient of each convolutional layer in the network model.

[0023] A further improvement of this invention is that, in step 5, based on the set target pruning rate r, all channels of the network model are pruned according to their global importance. Sort the data from largest to smallest to obtain the threshold:

[0024]

[0025] Where r is the target pruning rate and t is the threshold.

[0026] A further improvement of the present invention is that step 6 is specifically performed as follows:

[0027] Step 6.1: Copy the pruned network to obtain multiple identical networks;

[0028] Step 6.2: Train multiple identical networks using different learning rates and different weight decay rates. After each iteration of multiple networks, perform adaptive weighted fusion based on the information entropy of the weight parameters.

[0029] The beneficial effects of this invention are: 1. This invention starts from the global perspective of the model, using the output feature map as the entry point, to accurately calculate the importance of each channel in the convolutional layer and remove unimportant channels. This approach not only accurately obtains the optimal substructure of the model but also has strong ease of use.

[0030] 2. The compression target of the method proposed in this invention is mainly based on models constructed using deep convolutional neural networks, such as network models built based on ResNet, VGG, and MobileNet architectures. The compression target range can cover most current convolutional neural network models. The network model obtained by the method of this invention does not require additional hardware or software assistance, thus reducing the number of parameters and floating-point operations of the neural network model to a certain extent. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the model compression method based on linear dependency of feature maps proposed in this invention.

[0032] Figure 2 Flowchart of the genetic evolution algorithm. Detailed Implementation

[0033] To more clearly illustrate the technical solution of the present invention, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:

[0034] This invention is a neural network model compression method based on feature map linear dependence, which includes the following steps:

[0035] Step 1: Build a GPU-based version of the PyTorch deep learning framework;

[0036] Step 2: Obtain the number of convolutional layers in the network model to be compressed, the number of channels in each convolutional layer, and for the network model, randomly take a small number of data samples from its training dataset and input them into the model to obtain the feature map output by each channel.

[0037] Step 3: Based on the acquired feature maps, calculate the average value of the corresponding entropy-based linear dependency independence. The higher the average value, the more specific the information that the corresponding channel can extract from the input, and the more important it is for the network model.

[0038] Step 4: Use the average value of the linear dependency independence of the feature maps obtained in Step 3 as the local importance of the corresponding channel. Through iterative genetic evolution algorithm, calculate the global scale coefficient and global bias coefficient of each convolutional layer in the network model, and convert the local importance into global importance;

[0039] Step 5: Based on the target pruning rate, sort the networks according to their global importance from highest to lowest, and calculate the corresponding thresholds. Then remove channels in the network model whose global importance is less than the threshold to obtain the sub-network model, which is the optimal sub-network model after pruning.

[0040] Step 6: Using the training dataset, perform multi-network joint parallel training on the obtained optimal sub-network model to fully restore the network model performance;

[0041] In step 2, the neural network model is traversed to count the number of convolutional layers in the model and the number of channels in each convolutional layer.

[0042] First, a small amount of data is randomly sampled from the training dataset of the model to be pruned, input into the network, and the feature map A output by each convolutional layer is output. l Among them, feature map A l for:

[0043]

[0044] Where A l The output of the l-th convolutional layer in the model has a width of w, a height of h, and a number of channels of c. l Feature map, It is the feature map output of the i-th channel in the vectorized l-th convolutional layer, and T is the matrix transpose.

[0045] Step 3 calculates the local channel independence of each channel in the convolutional layer. This method determines the importance of a channel based on the independence between channels in the convolutional layer. Generally, if a channel depends on other channels in the same convolutional layer, the information extracted by that channel can also be extracted from other channels, meaning that channel is not important to the network model. The more independent a channel is, the lower its substitutability, and the more important it is to the model. express The local channel independence of a channel is also expressed as its local importance to the channel. That is...

[0046]

[0047] Where E(*) represents the calculation of information entropy. Let E(X) represent a linearly dependent auxiliary two-dimensional matrix, where all columns except the i-th column are 0, and all other columns are 1. For E(X), the range of element values ​​in matrix X is uniformly divided into N intervals, and the probability p of each interval is calculated, i.e., the information entropy of X is:

[0048]

[0049] Where p k This represents the percentage of elements in the k-th interval.

[0050] In step 4, a genetic evolutionary algorithm is used to calculate the global scale coefficient α and global bias coefficient β of each convolutional layer in the network model, thereby determining the local channel independence of each channel. Convert to global importance Right now

[0051]

[0052] Where α l β l These represent the global scale coefficient and global bias coefficient of the l-th layer, respectively. Specifically, they represent the global coefficients of the network model. One possible scenario is that the network is treated as a population, with the population score representing the performance of the sub-network model obtained by pruning using the global coefficients. This sub-network model does not undergo performance recovery. Performance metrics may differ for different network models, such as common metrics like accuracy and speed. A randomized population pool is constructed, and the new population obtained after mutation is... The mutation process is represented as follows:

[0053]

[0054] σ is a random number, σ∈[0,1]. For example... Figure 2 As shown, the specific steps of the genetic evolution algorithm are as follows:

[0055] Step 4.1: Initialize 64 groups, calculate the score of each group, and put it into the group pool;

[0056] Step 4.2: Randomly sample 8 populations from the population pool as mutation samples, and select the population with the highest score as the mutation target. Mutate the mutation target and determine whether the score of the new population after mutation is greater than that of the lowest-scoring population in the mutation samples. If so, replace the poor population with the new population and put the mutation sample back into the population pool. If not, put the mutation sample back into the population pool. Through 500 iterations, the superior population is gradually obtained.

[0057] Step 4.3: After the iteration is complete, select the group with the highest score from the population pool, which is the global scale coefficient and global bias coefficient of each convolutional layer in the network model.

[0058] In step 5, based on the set target pruning rate, all channels of the network model are pruned according to their global importance. Sort the data from largest to smallest to obtain the threshold, i.e.:

[0059]

[0060] Where r is the target pruning rate and t is the threshold. Remove Channels smaller than the threshold t. The resulting model is the optimal subnetwork model after accurate pruning.

[0061] The training method in step 6 can restore the network's performance. Since redundant channels in the network model are removed, its performance inevitably declines. Simply fine-tuning the network model does not fully restore its performance. This is because fewer parameters reduce robustness, making it prone to getting trapped in local optima during training. To escape local optima, this method uses a multi-network joint parallel training strategy based on adaptive weighting of weight parameters. First, the pruned network model is copied. The number of copies is set according to the target pruning rate, typically 3 to 6. Then, each model is trained in parallel using different learning rates and different weight decay rates. After multiple networks have undergone one iteration, an adaptive weighted fusion is performed based on the information entropy of the weight parameters of each model. The fusion formula is as follows:

[0062]

[0063] γ = A*(arctan(c*ΔE)) + 0.5;

[0064]

[0065] Where A and c are hyperparameters, A = 0.5 and c = 600. denoted as the weight parameters of the i-th convolutional layer of the 1st and 2nd sub-networks, respectively, where ΔE is the information entropy difference and γ is the adaptive weighting parameter. This represents the weight parameters of the i-th convolutional layer of the two sub-networks after adaptive weighted fusion.

[0066] In the joint parallel training of multiple networks, stochastic gradient descent and gradient concentration are used as optimizers. The initial learning rate of each sub-network is randomly selected from the interval [0.1, 0.01], the weight decay rate is randomly selected from the interval [1e-4, 2e-3], and the learning rate is adjusted by cosine annealing algorithm.

[0067] The following experiments are conducted. Representative state-of-the-art methods from recent years are selected and compared with the present invention on the standard classification dataset CIFAR-100, where the common convolutional neural network ResNet56 undergoes structural pruning. The results are as follows:

[0068]

[0069] Top-1 accuracy represents the accuracy of the classification task, i.e., the performance of the pruned network model; higher accuracy indicates a more effective method. Floating-point computation represents the number of floating-point operations required to run the pruned model once forward. Lower floating-point computation indicates a more compact network model.

[0070] As can be seen from the table, this invention still outperforms current mainstream advanced methods while ensuring that the floating-point computation of the pruned model is less than that of current mainstream advanced methods.

[0071] Those skilled in the art will understand that, unless otherwise defined, the terminology used herein is generally interchangeable with commonly used terminology.

[0072] It should be noted that this invention is not limited to experimental examples, but applies to all current convolutional-based deep neural network models. In practical applications, some parameters in this invention need to be adjusted according to the specific circumstances of different deep neural network models. Any other implementation methods proposed by those skilled in the art based on this invention also fall within the scope of protection of this invention.

Claims

1. A method for compressing neural network models based on linear dependence of feature maps, characterized in that: Includes the following steps: Step 1: Set up a GPU-based PyTorch deep learning framework runtime environment; Step 2: For the trained neural network model to be compressed, obtain the number of convolutional layers in the model and the number of channels in each convolutional layer. Randomly sample a small number of samples from its training dataset and input them into the network model. Extract the feature map output by each channel in the network model. Step 3: Based on linear dependency independence, process the extracted feature maps and use the average value as the independence of the corresponding channels; Step 4: Treat the independence of channels as the local importance of channels, and use the genetic evolution algorithm to calculate the global scale coefficient and global bias coefficient of each convolutional layer in the network model through multiple iterations. Step 5: Using the global scale coefficient and global bias coefficient, the local importance of the channels is converted into global importance. Based on the set target pruning rate, the global importance scores of the channels in the network model are sorted, and channels with scores below the threshold are removed. Step 6: Using the training dataset, perform multi-network joint parallel training on the optimal sub-network obtained by pruning.

2. The neural network model compression method based on feature map linear dependence according to claim 1, characterized in that: The feature map output for each channel in step 2 is as follows: A l This is the feature map output by the l-th convolutional layer in the model. It is the feature map output of the i-th channel in the vectorized l-th convolutional layer, and T is the matrix transpose.

3. The neural network model compression method based on feature map linear dependence according to claim 1, characterized in that: In step 3, the linear dependence independence of the corresponding channels is calculated based on the output feature map. The specific formula is as follows: Where, for E(X), X represents the parameter matrix to be calculated, the range of element values ​​in X is uniformly divided into N intervals, and then the probability p of each interval is calculated. k Let represent the probability distribution of the k-th interval. Let represent the linear dependency auxiliary two-dimensional matrix of the l-th convolutional layer, i.e., all columns except the i-th column are 1. A represents the linear dependency independence of the i-th channel in the l-th convolutional layer of the network model. l This is the feature map output by the l-th convolutional layer in the model.

4. The neural network model compression method based on feature map linear dependence according to claim 3, characterized in that: In step 4, the global scale coefficient is represented as α={α1,..,α...} L The global deviation coefficient is expressed as β={β1,..,β}. L The specific steps for step 4 are as follows: Step 4.1: Treat a {α,β} as a population, then randomly initialize multiple populations and form a population pool; Step 4.2: Randomly select a small number of populations from the population pool as mutation samples. Choose the population with the highest score in the sample pool as the mutation target. If the new population obtained through mutation is better than the worst population in the sample pool, replace the low-scoring population with the new population. Through multiple iterations, gradually obtain superior populations. The mutation process is as follows: Where Std(·) represents the formula for calculating the standard deviation. This represents the formula for calculating the normal distribution, M. l This represents the weight parameter matrix of the l-th convolutional layer in the network model, where L is the total number of layers in the model, and s l Let σ be the standard deviation of the weight parameters of the l-th layer, σ be a random number, and α be a variable. l β l Let $\begin{p}$ be the global scale coefficient and global bias coefficient for the $l$ layer, and $e$ be the natural constant. Step 4.3: After the iteration is complete, select the group with the largest score in the pool, which is the global scale coefficient and global bias coefficient of each convolutional layer in the network model.

5. The neural network model compression method based on feature map linear dependence according to claim 4, characterized in that: In step 5, based on the set target pruning rate r, all channels of the network model are pruned according to their global importance. Sort the data from largest to smallest to obtain the threshold: Where r is the target pruning rate and t is the threshold.

6. The neural network model compression method based on feature map linear dependence according to claim 5, characterized in that: Step 6 involves the following steps: Step 6.1: Copy the pruned network to obtain multiple identical networks; Step 6.2: Train multiple identical networks using different learning rates and different weight decay rates. After each iteration of multiple networks, perform adaptive weighted fusion based on the information entropy of the weight parameters.

Images