A convolutional neural network pruning method for industrial production applications
By combining unstructured and structured pruning, using connectivity information to calculate the perceived amount of the convolution kernel, efficient compression and accuracy improvement of the convolutional neural network are achieved, and the problem of low accuracy in the existing technology is solved. Suitable for industrial production equipment with limited resources.
Patent Information
- Application Number
- CN202210097374.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-27
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-01-27
AI Technical Summary
The existing convolutional neural network pruning method fails to effectively utilize the information of parameter matrix connectivity, resulting in low accuracy when applied on resource-constrained devices, and the existing method fails to fully combine unstructured and structured pruning. The advantages of branches.
Using connectivity-based neural network pruning method, a high-sparse weight matrix is generated through unstructured pruning, and the connection area is calculated using depth-first search, and the perceived amount of the convolution kernel is calculated based on the weight norm in the connection block. , realize the accurate measurement of the importance of the convolution kernel, and perform structured pruning based on the perceived amount.
It improves the compression rate and accuracy of the convolutional neural network, ensures the freedom of cropping, and greatly reduces the amount of parameters and calculations, and is suitable for industrial production applications.
Smart Images

Figure CN114638355B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer technology, specifically the field of deep learning technology, and designs a neural network pruning method based on connectivity. Background Art
[0002] In recent years, with the widespread application of various convolutional neural networks in the field of industrial production, the demand for transplanting convolutional neural networks to automated production equipment has become increasingly strong. However, when applied on these platforms with strict resource constraints, the resources available to implement convolutional neural networks are very limited. The explosive growth of parameters and computational complexity of convolutional neural networks limits their application in resource-constrained devices. Convolutional neural network pruning is an effective technology to solve the problem of over-parameterization of neural networks. Generally speaking, pruning is to judge the importance of weights or convolution kernels in convolutional neural networks by designing importance judgment criteria, and then delete unimportant parts of convolutional neural networks, thereby achieving the goal of reducing the number of network parameters and computational complexity. The current existing parameter pruning methods do not utilize the information of the connectivity of the parameter matrix, which may result in low accuracy of the pruned convolutional neural network.
[0003] Neural network pruning can be divided into unstructured pruning and structured pruning. Unstructured pruning often makes the weight matrix in the layer become a sparse matrix, but it does not reduce the amount of calculation, and the accelerated calculation of sparse matrices requires specific software libraries or hardware support. The main idea of structured pruning is to remove unimportant channels or filters, thereby reducing the number of parameters and the amount of calculation at the same time.
[0004] The present invention is made in order to give full play to the advantages of combining the two pruning strategies in the pruning process and accurately and efficiently measure the importance of the convolution kernel. Summary of the invention
[0005] In view of the defects of the existing pruning technology, the present invention provides a convolutional neural network pruning method for industrial production applications. In the pruning process, the method first uses an unstructured pruning method to prune the trained matrix with high degrees of freedom to obtain a set of weight matrices with high sparsity. Then a mask matrix corresponding to the weight matrix is generated, and the connected area of the mask matrix is calculated by depth-first search. The connected area reflects the perception area of the convolution kernel for the feature. When the connected area is large, it means that the convolution kernel can recognize target features of a larger area, that is, the convolution kernel has a greater contribution rate to the entire convolutional neural network recognition for industrial production applications. Then, the convolution kernel perception is calculated based on the connected area combined with the norm of the weights in the connected block, and the convolution kernels of each layer are ranked in importance. Finally, the network is structured pruned according to the ranking. While ensuring the degree of freedom of pruning and the compression rate of the neural network, this method more accurately measures the importance of the convolution kernel, thereby achieving a higher accuracy rate.
[0006] A convolutional neural network pruning method for industrial production applications includes the following steps:
[0007] (1) Preparation
[0008] Perform preparation work and load the trained convolutional neural network for industrial production applications and the dataset containing industrial production images;
[0009] (2) Unstructured pruning
[0010] (2.1) Set the unstructured target pruning rate R for the convolutional neural network for industrial production applications n ∈[0,1);
[0011] (2.2) Using the weight value as the criterion for measuring the importance of neurons, all neurons are ranked;
[0012] (2.3) For all connections of the entire convolutional neural network for industrial production applications, a set proportion of unimportant neurons are pruned, and the weight value of the pruned neurons is less than or equal to the weight of the remaining neurons;
[0013] (2.4) Restore the convolutional neural network loss by retraining, and repeat steps (2.2-2.4) until the entire network reaches the target pruning rate R n , unstructured pruning terminates;
[0014] (2.5) Retrain the pruned model and monitor the accuracy of the model. When the accuracy of the retrained model no longer increases, the pruned convolutional neural network for industrial production applications is obtained;
[0015] (3) Perform structured pruning on the unstructured pruned model based on the perception quantity;
[0016] (3.1) Set the structured target pruning rate R for the convolutional neural network for industrial production applications s ∈R L , where the structured target pruning rate of the lth layer is {R sl ∈[0,1),1≤l≤L}, L is the number of layers of the neural network;
[0017] (3.2) Calculate the mask matrix corresponding to each weight matrix. When the weight matrix element is less than the set threshold, the corresponding position matrix element of the mask matrix is 0; when the weight matrix element is greater than or equal to the threshold, the corresponding position matrix element of the mask matrix is 1;
[0018] (3.3) Using the depth-first search algorithm, the connected area of the weight matrix is calculated. The connected blocks are formed by connecting adjacent non-zero elements in multiple directions in the matrix. The mask matrix is traversed to calculate the connected areas of all connected blocks in the matrix.
[0019] (3.4) The convolution kernel perception is calculated based on the connected area and the norm of the weights in the connected block. The formula is:
[0020]
[0021] Where F is the weight matrix corresponding to a convolution kernel in the convolutional neural network for industrial production applications, i represents the horizontal coordinate, j represents the vertical coordinate, m is the total number of connected blocks in the matrix, and A k is the kth connected block of the matrix, C k is the area of the kth connected block;
[0022] (3.5) Sort the importance of the convolution kernels of each layer according to the perception value;
[0023] (3.6) Setting a structured target pruning rate to prune the convolution kernels of each layer in the convolutional neural network for industrial production applications, the perception of the pruned convolution kernels is less than or equal to the perception of the remaining convolution kernels;
[0024] (3.7) Restore the convolutional neural network loss by retraining, repeat steps (3.2-3.7) until each layer of the network reaches the target pruning rate, and the structured pruning is terminated.
[0025] Furthermore, the unstructured target pruning rate R n and structured target pruning rate R s Same as the overall target pruning rate R.
[0026] Furthermore, the connected area of the weight matrix is calculated as follows:
[0027] a. For each weight matrix, there is a corresponding mask matrix of the same size. The mask matrix M can be calculated as follows:
[0028]
[0029] Where F is the weight matrix corresponding to a convolution kernel in the convolutional neural network for industrial production applications, N is the number of input channels of this layer of network, W is the width of the weight matrix, H is the height of the weight matrix, i represents the horizontal coordinate, j represents the vertical coordinate, and F ij Represents the element in the i-th row and j-th column of the weight matrix, M ij represents the element in the i-th row and j-th column in the mask matrix, and δ is the set threshold;
[0030] b. Use the depth-first search algorithm to calculate the connected area of the mask matrix:
[0031] The connected block is formed by connecting the adjacent elements of 1 in multiple directions in the matrix. With the help of the depth-first search algorithm, the elements of 1 in the mask matrix are traversed, and starting from their position, each element adjacent to it (and the elements connected to these elements) is explored in multiple directions. If the adjacent element is 1, the connected block is added and marked as visited, that is, the element is set to 0, and then the elements connected to the connected block are continued to be explored. The total number of explored elements of 1 is the area of the connected graph of the block. The connected area reflects the perception area of the convolution kernel for features. When the connected area is large, it means that the convolution kernel can recognize target features of a larger area, that is, the convolution kernel has a greater contribution to the recognition of the entire convolutional neural network for industrial production applications. The exploration direction during the search process is as follows, where directions is the search direction array:
[0032]
[0033] Where x is the horizontal coordinate of the current position, y is the vertical coordinate of the current position, x′ is the horizontal coordinate of the position to be explored later, y′ is the vertical coordinate of the position to be explored later, k is the horizontal coordinate of the directions two-dimensional array, and directions.length is the length of the directions array.
[0034] Furthermore, the specific calculation method of calculating the perception amount of the weight matrix according to the connected area is: multiplying the connected area by the Frobenius norm of the corresponding connected block in the matrix and summing them to obtain the perception amount G of the weight matrix, and its formula is:
[0035]
[0036] Where F is the weight matrix corresponding to a convolution kernel in the convolutional neural network for industrial production applications, i represents the horizontal coordinate, j represents the vertical coordinate, m is the total number of connected blocks in the matrix, and A k is the kth connected block of the matrix, C k is the area of the kth connected block.
[0037] The beneficial effect of the present invention is that the present invention is based on a combination of unstructured pruning and structured pruning, which not only ensures the flexibility of unstructured pruning, but also utilizes structured pruning to greatly reduce the amount of weight matrix parameters and calculation, thereby improving the compression rate of convolutional neural networks for industrial production applications.
[0038] The present invention extracts the connected area of the weight matrix from the mask matrix corresponding to the weight matrix. The connected area reflects the perception area of the convolution kernel to the feature. When the connected area is large, it means that the convolution kernel can identify target features of a larger area, that is, the convolution kernel contributes more to the recognition of the entire network. The connected area is combined with the neuron weight to obtain the perception value of the weight matrix. Compared with the importance judgment algorithm in traditional pruning, the present invention can more accurately identify important weight matrices.
[0039] The importance criterion of the unstructured pruning in the present invention is the size of the weight. However, the unstructured pruning method is not limited to this, and the present invention can be easily integrated into other excellent unstructured pruning methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a schematic diagram of two neural network pruning methods;
[0041] Figure 2 It is a schematic diagram of the calculation process of the weight matrix perception quantity G;
[0042] Figure 3 This is a schematic diagram of a convolutional neural network pruning method for industrial production applications. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific examples described herein are only used to explain the present invention and do not limit the present invention.
[0044] (1) Preparation
[0045] Prepare and load the trained convolutional neural network for industrial production applications and a data set containing industrial production images. The neural network in this embodiment uses the VGG-16 model, which is a general convolutional neural network. The data set uses the CIFAR-10 data set.
[0046] (2) Unstructured Pruning (2.1) Set the unstructured target pruning rate R for the convolutional neural network for industrial production applications. n ∈[0,1);
[0047] (2.2) The weights of the neural network connections are obtained by training the convolutional neural network for industrial production applications. The weight values of the connections are used as a criterion to measure the importance of the connections in the trained neural network model, and the importance of the connections in all layers is ranked;
[0048] (2.3) For all connections of the entire convolutional neural network for industrial production applications, unimportant neurons with a set ratio p are pruned, that is, the least important neurons are selected in turn for pruning until the set ratio p is met. In this embodiment, the maximum value of the lower fifth of all connection weights is used as the threshold. When the connection weight is greater than or equal to the threshold, it is retained. When the connection weight is less than the threshold, it is deleted from the convolutional neural network for industrial production applications. Unstructured pruning is as follows: Figure 1 As shown in a). p is set to R n one fifth of
[0049] (2.4) Restore the loss of the convolutional neural network by retraining. Repeat steps (2.2-2.4) until the entire convolutional neural network for industrial production applications reaches the target pruning rate R n , unstructured pruning terminates;
[0050] (2.5) Retrain the pruned model and monitor the accuracy of the model. When the accuracy of the retrained model no longer increases, the pruned neural network model is obtained;
[0051] (3) Perform structured pruning on the unstructured pruned model based on the perception quantity;
[0052] (3.1) Set the structured target pruning rate R for the convolutional neural network for industrial production applications s ∈R L , where the structured target pruning rate of the lth layer is {R sl ∈[0,1),1≤l≤L}, L is the number of layers of the neural network;
[0053] (3.2) Calculate the mask matrix corresponding to each weight matrix, where the threshold is set to 0. When the absolute value of the weight matrix element is equal to 0, the corresponding position matrix element of the mask matrix is 0; when the absolute value of the weight matrix element is greater than 0, the corresponding position matrix element of the mask matrix is 1. The mask matrix M can be calculated by the following formula:
[0054]
[0055] Where F is the weight matrix corresponding to a convolution kernel in the convolutional neural network for industrial production applications, N is the number of input channels of this layer of network, W is the width of the weight matrix, H is the height of the weight matrix, i represents the horizontal coordinate, j represents the vertical coordinate, and F ij Represents the element in the i-th row and j-th column of the weight matrix, M ij Represents the element at row i and column j in the mask matrix.
[0056] (3.3) The connected area of the weight matrix is calculated with the help of the depth-first search algorithm. The connected block is formed by connecting adjacent non-zero elements horizontally, vertically, and diagonally. Traverse the mask matrix and calculate the connected area of all connected blocks in the matrix. Starting from the position of each element, explore each adjacent element (and the elements connected to these elements) in 8 directions. If the adjacent element is 1, add the connected block and mark it as visited, that is, set the element to 0, and then continue to explore the elements connected to the connected block. Then the total number of explored elements that are 1 is the area of the connected graph of the block. The exploration directions during the search process are as follows:
[0057] directions={{-1,-1},{-1,0},{-1,1},{0,-1},{0,1},{1,-1},{1,0},{1,1}};
[0058]
[0059] Where x is the horizontal coordinate of the current position, y is the vertical coordinate of the current position, x′ is the horizontal coordinate of the position to be explored later, y′ is the vertical coordinate of the position to be explored later, and k is the horizontal coordinate of the directions two-dimensional array.
[0060] (3.4) Calculate the perception of the weight matrix based on the connected area. The larger the perception, the more feature information the convolution kernel can identify. The specific calculation method for calculating the perception of the weight matrix based on the connected area is: multiply the connected area by the Frobenius norm of the corresponding connected block in the matrix and sum them to obtain the perception G of the weight matrix. The formula is:
[0061]
[0062] Where F is the weight matrix corresponding to a convolution kernel in the convolutional neural network for industrial production applications, i represents the horizontal coordinate, j represents the vertical coordinate, m is the total number of connected blocks in the matrix, and A k is the kth connected block of the matrix, C k is the area of the kth connected block. The calculation process of the weight matrix perception G is as follows Figure 2 shown.
[0063] (3.5) Sort the importance of the convolution kernels of each layer according to the perception value;
[0064] (3.6) Prune the convolution kernels of each layer in the convolutional neural network. According to the size of the convolution kernel perception, unimportant convolution kernels are pruned in turn until the set structured target pruning rate is met. The structured pruning is as follows: Figure 1 As shown in b;
[0065] (3.7) Restore the convolutional neural network loss by retraining. Repeat steps (3.2-3.7) until each layer of the network reaches the target pruning rate, and the structured pruning is terminated;
[0066] The above description is only a specific example of the present invention and does not constitute any limitation to the present invention. It is obvious that for professionals in this field, after understanding the content and principles of the present invention, they may make various modifications and changes in form and details without departing from the principles and structures of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.
Claims
1. A convolutional neural network pruning method for industrial production applications, characterized in that: The specific steps include: (1) Preparation Perform preparation work and load the trained convolutional neural network for industrial production applications and the dataset containing industrial production images; (2) Unstructured pruning (2.1) Set the unstructured target pruning rate R for the convolutional neural network for industrial production applications n ∈[0,1); (2.2) Using the weight value as the criterion for measuring the importance of neurons, all neurons are ranked; (2.3) For all connections of the entire convolutional neural network for industrial production applications, a set proportion of unimportant neurons are pruned, and the weight value of the pruned neurons is less than or equal to the weight of the remaining neurons; (2.4) Restore the loss of the convolutional neural network by retraining, and repeat steps (2.2-2.4) until the entire convolutional neural network for industrial production applications reaches the target pruning rate R n , unstructured pruning terminates; (2.5) Retrain the pruned model and monitor the accuracy of the model. When the accuracy of the retrained model no longer increases, the pruned convolutional neural network for industrial production applications is obtained; (3) Perform structured pruning on the unstructured pruned model based on the perception quantity; (3.1) Set the structured target pruning rate R for the convolutional neural network for industrial production applications s ∈R L , the structured target pruning rate of the lth layer is {R sl ∈[0,1),1≤l≤L}, L is the total number of layers of the neural network; (3.2) Calculate the mask matrix corresponding to each weight matrix. When the weight matrix element is less than the set threshold, the corresponding position matrix element of the mask matrix is 0; when the weight matrix element is greater than or equal to the threshold, the corresponding position matrix element of the mask matrix is 1; (3.3) Using the depth-first search algorithm, the connected area of the weight matrix is calculated. The connected blocks are formed by connecting adjacent non-zero elements in multiple directions in the matrix. The mask matrix is traversed to calculate the connected areas of all connected blocks in the matrix. (3.4) The convolution kernel perception is calculated based on the connected area and the norm of the weights in the connected block. The formula is: Where F is the weight matrix corresponding to a convolution kernel in the convolutional neural network for industrial production applications, i represents the horizontal coordinate, j represents the vertical coordinate, m is the total number of connected blocks in the matrix, and A k is the kth connected block of the matrix, C k is the area of the kth connected block; (3.5) Sort the importance of the convolution kernels of each layer according to the perception value; (3.6) Setting a structured target pruning rate to prune the convolution kernels of each layer in the convolutional neural network for industrial production applications, the perception of the pruned convolution kernel is less than or equal to the perception of the remaining convolution kernel; (3.7) Restore the convolutional neural network loss by retraining, and repeat steps (3.2-3.7) until each layer of the network reaches the structured target pruning rate R s , structured pruning terminates.
2. The convolutional neural network pruning method for industrial production applications according to claim 1, characterized in that: The unstructured target pruning rate R n and structured target pruning rate R s The same as the overall target pruning rate R, that is, R n =R s =R.
3. The convolutional neural network pruning method for industrial production applications according to claim 1, characterized in that: The calculation method of the connected area of the weight matrix is specifically as follows: a. For each weight matrix, there is a corresponding mask matrix of the same size. The mask matrix M can be calculated as follows: Where F is the weight matrix corresponding to a convolution kernel in the convolutional neural network for industrial production applications, N is the number of input channels of this layer of network, W is the width of the weight matrix, H is the height of the weight matrix, i represents the horizontal coordinate, j represents the vertical coordinate, and F ij Represents the element in the i-th row and j-th column of the weight matrix, M ij represents the element in the i-th row and j-th column in the mask matrix, and δ is the set threshold; b. Use the depth-first search algorithm to calculate the connected area of the mask matrix: The connected block is formed by connecting adjacent non-zero elements in multiple directions in the matrix. With the help of the depth-first search algorithm, the elements that are 1 in the mask matrix are traversed, and starting from their position, each adjacent element and the elements connected to these elements are explored in multiple directions. If the adjacent element is 1, the connected block is added and marked as visited, that is, the element is set to 0, and then the elements connected to the connected block are continued to be explored. The total number of explored elements that are 1 is the area of the connected graph. The connected area reflects the perception area of the convolution kernel for features. When the connected area is large, it means that the convolution kernel can recognize target features of a larger area, that is, the convolution kernel has a greater contribution to the recognition of the entire convolutional neural network for industrial production applications. The exploration direction during the search process is as follows, where directions is the search direction array: Where x is the horizontal coordinate of the current position, y is the vertical coordinate of the current position, x′ is the horizontal coordinate of the position to be explored later, y′ is the vertical coordinate of the position to be explored later, k is the horizontal coordinate of the directions two-dimensional array, and directions.length is the length of the directions array.
4. The convolutional neural network pruning method for industrial production applications according to claim 1, characterized in that: The specific calculation method of calculating the perception amount of the weight matrix according to the connected area is: multiplying the connected area with the Frobenius norm of the corresponding connected block in the matrix and summing them to obtain the perception amount G of the weight matrix, and its formula is: Where F is the weight matrix corresponding to a convolution kernel in the convolutional neural network for industrial production applications, i represents the horizontal coordinate, j represents the vertical coordinate, m is the total number of connected blocks in the matrix, and A k is the kth connected block of the matrix, C k is the area of the kth connected block.
Citation Information
Patent Citations
Neural network lightweight method and system based on continuous pruning
CN113610192A
Hardware-based neural network real-time pruning method and system and neural network accelerator
CN113657595A
Neural network regularization bit serial calculation compression method and device
CN113902109A
Method and apparatus for neural network model compression with micro-structured weight pruning and weight unification
US20210397963A1
