A Two-Stage Triangle Pruning Method for Deep Neural Networks
Through the two-stage triangular pruning method of deep neural network, combining one-stage triangular pruning and two-stage pruning, the problems of slow pruning speed, low accuracy and insufficient flexibility in the existing technology are solved, and more efficient and more Flexible neural network pruning significantly reduces the index overhead of FPGA and improves recognition accuracy.
Patent Information
- Application Number
- CN202310803484.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-03
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2043-07-03
AI Technical Summary
The existing neural network pruning methods have insufficient speed, accuracy and flexibility, especially the index overhead for FPGAs, and the sparsity caused is not flexible enough, which consumes unnecessary time.
A two-stage triangular pruning method for deep neural networks is proposed. By combining one-stage triangular pruning and two-stage pruning, using square matrix structure and masking processing, the sparsity and reference value are flexibly selected, and the index overhead is reduced. Improve pruning efficiency.
It achieves higher sparsity selection flexibility, improves accuracy compared with structured pruning, saves hardware indexing overhead, and significantly reduces pruning time, improving recognition accuracy and efficiency.
Smart Images
Figure CN116776957B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a two-stage triangle pruning method for a deep neural network, and belongs to the field of artificial intelligence. Background Art
[0002] There are endless methods for data segmentation and compression, which generally include network pruning, low-rank decomposition, compact network design, etc. Compared with method 1, the latter two methods are more complex and difficult to implement, and are not suitable for the later training of neural networks. Therefore, in order to reduce the design complexity, most neural networks today use the method of network pruning.
[0003] Network pruning is a traditional network compression method. It mainly reduces the network's storage and computing resources by removing some connections in the neural network. In recent years, network pruning methods have attracted more and more attention. Network pruning can be divided into structured pruning and unstructured pruning. The classification is based on whether it constrains the position of the elements that can be pruned.
[0004] Unstructured pruning means that the structure of the pruned network is not deliberately pursued, and the parameters at any position are simply pruned. This results in unstructured pruning without any other constraints on its form, so that a high sparsity can be achieved. In 2015, Han et al. proposed a three-stage deep compression framework (Deep Compression) in their paper: network pruning, quantization, and Huffman coding. By using this method, AlexNet can be compressed 35 times without loss of accuracy. After network pruning, the network needs to be retrained to compensate for the loss of accuracy during the pruning process. In their paper, once a weight is pruned, it remains zero during the subsequent retraining process, which may lead to a decrease in accuracy. Therefore, this type of method causes the pruned weight matrix to be irregularly sparse, and the actual acceleration effect is low.
[0005] Unlike unstructured pruning, structured pruning places certain restrictions on the locations where weights are removed. According to the different structured granularities, it can be divided into four types: group-level pruning, vector-level pruning, and filter-level or channel-level pruning. In 2019, Zhang et al. proposed a method of "pruning by structured diagonal matrix" for pruning, such as Figure 6As shown in Figure 2. Using this method, the method generates a regular sparse structure using the permuted block diagonal mask matrix. Unlike previous methods, this processing method eliminates the indexing overhead when tracking non-zero values, which is beneficial for hardware implementation. A new feasible pruning method is proposed for data segmentation compression. However, this method limits the sparsity and is not flexible enough, resulting in a complex pruning process that consumes unnecessary time.
[0006] Therefore, based on the current situation, a pruning method with fast speed, high accuracy and more flexibility is needed. Summary of the invention
[0007] The problem to be solved by the present invention is how to provide an efficient, fast and flexible neural network pruning method and at the same time reduce the index overhead of FPGA when processing data.
[0008] In order to solve the above technical problems, the technical solution proposed in the present invention is: a two-stage triangle pruning method for a deep neural network, performing the following steps:
[0009] S1 performs one-stage triangle pruning on the initial matrix;
[0010] S11 fills the initial matrix with zeros with the square matrix structure as the goal;
[0011] S12 divides the zero-filled initial matrix equally according to zkwd to obtain sub-blocks; the sub-blocks are of a square matrix structure;
[0012] S13 traverses each sub-block, calculates the absolute value of the elements of the four triangular regions corresponding to the sub-block, and then calculates the 11 norm of these four regions;
[0013] S14 compares the l1 norms of the four triangular regions, retains the triangular region with the largest l1 norm and masks it
[0014] After each sub-block is processed, a first-stage pruned matrix is obtained;
[0015] S2 performs a second-stage pruning on the matrix after the first-stage pruning;
[0016] S21 calculates optional values of sparsity; and selects a reference value from the optional values; the number of the optional values is positively correlated with the value of zkwd;
[0017]
[0018] S22 starts pruning from the hypotenuse of the triangular region in the matrix after the first stage of pruning, and retains the pruning result, and then prunes
[0019] The cutting is carried out on the basis of the previous one;
[0020] S23 calculates the current sparsity after pruning and compares it with the benchmark value;
[0021] S231 If the current sparsity is the same as the reference value, stop pruning;
[0022] S232: if the current sparsity is different from the reference value, retain the pruned portion in step S22 and remove the retained portion in step S22;
[0023] S233 calculates the sparsity of the matrix obtained in S232 and compares it with a reference value;
[0024] If they are the same, S2331 stops pruning;
[0025] If they are different, then return to step S22;
[0026] S24 outputs the pruned matrix in step S231 or step S2331, completing the pruning of the initial matrix;
[0027] The zkwd represents the dimension of each sub-block when the initial matrix is divided into blocks.
[0028] An improvement of the above technical solution is that the one-stage triangle pruning can be applied independently.
[0029] The improvement of the above technical solution is that: the optional range of the reference value is [025, 0.75], and the value is calculated by the formula in step S21.
[0030] The improvement of the above technical solution is that: the optional range of the reference value is [035,0.75], and the value is calculated by the formula in step S21.
[0031] The beneficial effects of the present invention are as follows: the pruning method of the present invention can provide more sparsity options and is more flexible than structured pruning; compared with the unstructured threshold pruning method, in the same handwritten digit recognition set experiment and while keeping the pruning sparsity the same, the pruning method of the present invention can provide higher accuracy; the present invention enables the hardware to quickly index non-zero elements through the use of a regular structure and a mask matrix, which greatly saves the hardware indexing overhead. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 A schematic flow chart of a two-stage triangle pruning method for a deep neural network according to an embodiment of the present invention.
[0033] Figure 2 This is a schematic diagram of the flow of a first-stage triangle pruning algorithm in a two-stage triangle pruning method for a deep neural network according to an embodiment of the present invention.
[0034] Figure 3This is a schematic diagram of the two-stage pruning algorithm flow in a two-stage triangle pruning method for a deep neural network according to an embodiment of the present invention.
[0035] Figure 4 This is a schematic diagram of four triangular regions of a first-stage triangular pruning sub-matrix in a two-stage triangular pruning method for a deep neural network according to an embodiment of the present invention.
[0036] Figure 5 A schematic diagram of the effect of two-stage pruning sparsity on pruning results in a two-stage triangle pruning method for a deep neural network according to an embodiment of the present invention.
[0037] Figure 6 Schematic diagram of the diagonal pruning principle.
[0038] Figure 7 This is a schematic diagram of the overall two-stage pruning process in a two-stage triangle pruning method for a deep neural network according to an embodiment of the present invention.
[0039] Figure 8 Comparison of the number of element-wise multiplications required for vectors and matrices that have undergone two-stage triangle pruning.
[0040] Fig. 9 This is a graph showing the impact of the selection of parameters zkwd and sparsity on the accuracy in the two-stage triangle pruning.
[0041] Fig.10 This is a flowchart of the algorithm of the custom magic_matrix() function in the second stage. DETAILED DESCRIPTION
[0042] Example
[0043] The dataset used in this example is the mnist dataset, which contains only the numbers from 0 to 9, a total of ten types of handwritten digits. The mnist dataset contains 60,000 training sets and 10,000 test sets. Each of these numbers is written differently, with different sizes, thicknesses, neatness, and clarity. The size of each image is planned, and the final image size is 28×28 pixels. This example uses an LSTM network for training. The hidden layer dimension of the LSTM is 28, and the input dimension is 28. A single layer of LSTM is used to prune the mnist dataset and quantize it. Finally, FPGA is used to identify the mnist dataset based on the LSTM pruned and quantized data.
[0044] This embodiment provides a two-stage triangle pruning method for a deep neural network, the process is as follows Figure 1 The entire process of the pruning method in this embodiment is demonstrated as follows Figure 7As shown. It has several fixed parameters, among which zkwd represents the dimension of each sub-block when the initial matrix is divided into blocks; the second-stage enable signal flag is used to determine whether to perform the second-stage pruning; sparsity represents the sparsity to be achieved in the second-stage pruning, that is, the benchmark value. The steps of this method are as follows:
[0045] S1 inputs the initial 28×28 matrix, which needs to be subjected to a phase of triangular pruning. The process is as follows: Figure 2 shown.
[0046] S11 does not need to pad with zeros because the input matrix is a square matrix.
[0047] In this example, zkwd is 4, so the initial matrix is divided into new 4×4 sub-blocks.
[0048] S13 traverses each sub-block, uses the abs() function to calculate the absolute value of the elements of the four triangular regions corresponding to the sub-block, and then calculates the l1 norm of these four regions. The abs() function is implemented in Python.
[0049] S14 compares the l1 norms of the four triangular regions, retains the triangular region with the largest l1 norm and performs masking processing on it (disposing non-zero elements as one). After each sub-matrix is processed, the matrix C11 after one-stage pruning is obtained, as shown in Figure 4 shown.
[0050] If S2 does not continue, the example ends here; if secondary segment pruning is required, the flag is set to 1 to continue. The process is as follows Figure 3 In this example, the sparsity selected is 0.5625, which is used as the reference value, and the sparsity is required to meet the required matrix.
[0051] S21 calculates the optional range of the second-stage pruning sparsity. When zkwd is set to 4, it can be calculated by the formula:
[0052]
[0053] The total number of values for calculating sparsity is: 0.0625, 0.1875, 0.375, 0.625, 0.5625, 0.4375, 0.25, such as Figure 5 As shown in the figure, the 1st, 2nd and 3rd columns on the left are the sparsity that can be achieved by pruning from the hypotenuse to the right-angle endpoint, the 4th, 5th and 6th columns on the left are the sparsity that can be achieved from the triangle endpoint to the hypotenuse, and the 7th column 0.625 is the sparsity that has been achieved after the first stage of pruning. It should be noted that the choice of the benchmark value must be within the available range.
[0054] S22 extracts the triangular area in the matrix after the first stage pruning Figure 5Start trimming from the 7th column bevel and keep the trimming result. All subsequent trimming will be based on the previous one.
[0055] That is, the diagonal of the sub-block in step S22 is translated using the triu() or tril() function, and the direction is to translate from the hypotenuse of the triangle to the right angle endpoint. Among them, the triu() or tril() function is a function in the Numpy library, which is an open source Python scientific computing library.
[0056] S23 calculates the current sparsity after pruning and compares it with the benchmark value;
[0057] S231 If the current sparsity is the same as the reference value, stop pruning;
[0058] S232: if the current sparsity is different from the reference value, retain the pruned portion in step S22 and remove the retained portion in step S22;
[0059] That is, using the magic_matrix function, the process is as follows Fig.10 As shown, this embodiment is implemented in Python, and the architecture is pytorch. Specifically, if the sparsity to be achieved is Figure 5 As shown in the first column of . Then we need to use two matrices, one is the matrix of the sixth column and the other is the matrix of the seventh column. These two matrices are the two inputs of the magic_matrix function. In addition, we need to initialize a matrix 3 that is the same as the matrix of the seventh column. At the same time, we traverse these two matrices. If the elements in the same position of the two matrices are equal and not 0, the same position of matrix 3 is set to zero. So the last matrix 3 is the matrix of the first column we want. The second and third column matrices are obtained in the same way.
[0060] S233 calculates the sparsity of the matrix obtained in S232 and compares it with a reference value;
[0061] If they are the same, S2331 stops pruning;
[0062] If they are different, then return to step S22;
[0063] S24 outputs the pruned matrix in step S231 or step S2331, completing the pruning of the initial matrix.
[0064] To prevent the process from getting stuck, S22 of this embodiment may be pruned for a maximum of three times, or the reference value of the sparsity may be required to be within an optional range.
[0065] As can be seen from this embodiment, the advantage of this method is that it can not only reduce the index overhead, but also reduce the amount of calculation. We assume that a weight sub-matrix after two-stage triangle pruning (zkwd = 4) is multiplied element-wise with a 4×1 vector (such as Figure 8 (b)(c)). When the vector is element-wise multiplied by the sparse matrix in the figure, the calculation of multiplication with 0 can be skipped because it has been masked. That is to say, depending on the sparsity of the two stages, it will perform at most 10 calculations and at least 1 calculation.
[0066] But if they do not perform the second-stage triangle pruning, the vector has 4 non-zero elements, and the 4×4 matrix is dense, with 16 elements (such as Figure 8 (a)). In this case, the multiplication of a dense vector and a dense matrix requires 16 element-wise multiplications, which is 1.6 to 16 times the amount of multiplications after the two-stage triangle pruning. This shows the high efficiency of the two-stage pruning. Users can flexibly select the two parameters zkwd and sparsity.
[0067] It can be seen from this embodiment that compared with the structured diagonal pruning, the method has the advantage of greatly saving time cost. In the experiment of the same handwritten digit recognition set, the diagonal pruning took nearly 10 hours, while the two-stage triangle pruning only took 3 hours. It can be seen that the two-stage pruning saves nearly 3 times the time resources compared with the diagonal pruning; compared with the structured diagonal pruning, the invention also has the advantage of greatly improving the recognition accuracy. In the experiment of the same handwritten digit recognition set, the recognition accuracy of the diagonal pruning is 83.8% and the recognition accuracy of the two-stage triangle pruning is as high as 96.93%. It can be seen that the recognition accuracy of the two-stage triangle pruning is higher. Compared with the unstructured threshold pruning method, in the same handwritten digit recognition set experiment, and keeping the same pruning sparsity (sparseness 32%), the threshold pruning method took 5.8 hours and the accuracy was only 69.16%. Although threshold pruning is more flexible than two-stage triangle pruning, the lack of time and accuracy puts the threshold pruning method at a disadvantage.
[0068] It can be seen from this embodiment that the advantage of this method is that Fig. 9It can be seen that in this embodiment, adjusting the two sets of parameters zkwd and sparsity will affect the accuracy. The smaller the zkwd, the larger the maximum value of the optional sparsity; and the larger the zkwd, the wider the range of the optional sparsity method, which reflects the flexibility of the two-stage triangle pruning. They have a common feature that when the sparsity is between 25% and 75%, the range of accuracy is within 5% (the range of accuracy is: 91.3%-96.51%), and when the sparsity is between 35% and 75%, the range of accuracy is only 2% (the range of accuracy is: 94.84%-96.53%). This also reflects the flexibility and inclusiveness of the two-stage pruning method, which can provide users with multiple precision options while maintaining high precision. Therefore, for model compression technology, the present invention is a method that can flexibly adjust parameters and maintain high precision.
Claims
1. A two-stage triangle pruning method for deep neural networks, characterized in that Follow these steps: S1 performs one-stage triangle pruning on the initial matrix; S11 fills the initial matrix with zeros with the square matrix structure as the goal; S12 divides the zero-filled initial matrix equally according to zkwd to obtain sub-blocks; the sub-blocks are square matrix structures; S13 traverses each sub-block, calculates the absolute value of the elements of the four triangular areas corresponding to the sub-block, and then calculates the 11 norm of these four areas; S14 compares the l1 norms of the four triangular regions, retains the triangular region with the largest l1 norm and performs masking on it; after each sub-block is processed, a one-stage pruned matrix is obtained; S2 performs a second-stage pruning on the matrix after the first-stage pruning; S21 calculates the optional value of sparsity; and selecting a reference value from the optional values; the number of the optional values is positively correlated with the value of zkwd; Sparsity = number of non-zero elements / number of sub-block elements; S22 starts pruning from the hypotenuse of the triangular area in the matrix after the first-stage pruning, and retains the pruning result, and the subsequent pruning is performed on the basis of the previous pruning; S23 calculates the current sparsity after pruning and compares it with the benchmark value; S231 If the current sparsity is the same as the reference value, stop pruning; S232: if the current sparsity is different from the reference value, retain the pruned portion in step S22 and remove the retained portion in step S22; S233 calculates the sparsity of the matrix obtained in S232 and compares it with a reference value; If they are the same, S2331 stops pruning; If they are different, then return to step S22; S24 outputs the pruned matrix in step S231 or step S2331, completing the pruning of the initial matrix; The zkwd represents the dimension of each sub-block when the initial matrix is divided into blocks; Digital images are recognized based on deep neural networks after editing.
2. The two-stage triangle pruning method for a deep neural network according to claim 1, characterized in that: The optional range of the reference value is [0.25, 0.75], and is the value obtained by calculating the formula in step S21.
3. The two-stage triangle pruning method for a deep neural network according to claim 1, characterized in that: The optional range of the reference value is [0.35, 0.75], and is the value obtained by calculating the formula in step S21.