A method for optimizing computation of neuron convolution kernel in deep learning network

Abstract

The application discloses a kind of calculation optimization methods of neuron convolution kernel in deep learning network, comprising the following steps: S1, confirm the basic weight in convolution operation, non-basic weight, irrelevant weight, item containing basic weight is recorded as basic weight item, item containing non-basic weight is recorded as non-basic weight item, item containing irrelevant weight is recorded as irrelevant item;S2, non-basic weight item is split with basic weight as unit;S3, the item after splitting is calculated with basic weight item;S4, output convolution operation result.The application utilizes arithmetic operation law, binary number decomposition and shift adder to reduce multiplication in convolution operation, realizes the calculation optimization of neuron convolution kernel in deep learning network.

Description

Technical Field

[0001] This invention relates to the field of deep learning, specifically to a method for optimizing the computation of convolutional kernels in neurons of deep learning networks. Background Technology

[0002] Convolutional Neural Networks (CNNs) are a type of feedforward neural network that excels in large-scale image processing and is now widely used in image classification, localization, and other fields. Compared to other neural network architectures, CNNs require relatively fewer parameters, enabling their widespread application.

[0003] In image processing, a convolution kernel is a weighted average of pixels in a small region of an input image, which is then used to produce each corresponding pixel in the output image. The weights are defined by a function called the convolution kernel.

[0004] In convolutional neural networks (CNNs), the computational cost of convolutional layers accounts for over 90% of the total computation. Therefore, accelerating convolutional layer operations is a crucial component of CNN acceleration. Researchers in this field have conducted in-depth studies to optimize the computation of CNNs.

[0005] CN1374692 discloses a design method for an intrinsically parallel two-dimensional discrete wavelet transform (DWT) VLSI structure. It transforms multiplication operations in filters into shift register and adder operations using shift-add techniques; minimizes the number of shift registers and adders in the filter implementation using common subexpression techniques; merges common terms in low-pass and high-pass filter operations using filter merging techniques, achieving parallel low-pass and high-pass operations in a compact hardware structure; and combines two sets of 2-D DWT column filters into one using filter combination and merging techniques to increase hardware utilization. However, since filter coefficients are often irregular, common subexpressions are not suitable for neural network convolutional kernels and filters. Merging common expressions in two filters is very limited, but this relies on the architecture specific to convolutional neural networks and is not suitable for filters.

[0006] CN110059817A discloses a method for implementing a low-resource-consumption convolutional processor. This method decomposes the multiplication kernel of the convolutional processor, adding data with the same offset within each kernel to form a convolutional addition kernel. Based on the offset corresponding to each addition kernel, the result of the addition kernel is shifted accordingly, and the final convolution result is calculated using an addition tree. This invention saves approximately 5-10% of LUT resources (FPGA implementation) compared to traditional methods and has advantages such as ease of deployment and reuse. This scheme is essentially the same as CN1374692. Although CN1374692 uses wavelet transform and CN110059817A uses convolution, both convert the multiplication operation in convolution into a binary addition sum, then merge terms with the same exponent; then, the merged terms are arithmetically shifted according to the exponent value, and finally all merged terms are summed. The method is not fundamentally different. As described in CN1374692, this method is only suitable for ASIC application-specific integrated circuit design, but not for common programmable processors on the market, especially VLIW digital signal processors. Summary of the Invention

[0007] This invention addresses the problems existing in the background technology by designing an optimized method for the computation of convolutional kernels in deep learning networks. Its main contribution lies in proposing several methods to reduce multiplication operations. Compared to addition, multiplication is much more complex, requiring significantly more on-chip area and power consumption. Therefore, minimizing the number of multiplications in convolutional kernel operations is essential.

[0008] Technical solution: A computational optimization method for neuron convolution kernels in deep learning networks is applied to 1) merging and optimizing weight parameters within the same convolution kernel, and 2) merging and optimizing weight parameters at corresponding input positions across multiple convolution kernels in the same layer. The optimization method includes the following steps: S1. Identify the basic weights, non-basic weights, and irrelevant weights in the convolution operation. Items containing basic weights are denoted as basic weight items, items containing non-basic weights are denoted as non-basic weight items, and items containing irrelevant weights are denoted as irrelevant items. S2. Split the non-basic weight items into units of basic weight; S3. Optimize the calculations of the split items and the basic weight items; S4. Output the convolution operation result.

[0009] Preferably, the method for obtaining the basic weight is as follows: sort all weights and take the weight with the smallest value as the basic weight.

[0010] Preferably, weights other than basic weights and non-basic weights are considered irrelevant weights.

[0011] Preferably, the basic weight is denoted as w. p j Weights that meet the following conditions are considered as non-basic weights: w p i =Aw p j (1) or w p k =w p j +2 m (2) Where A is any positive integer, m is any natural number, and w p i w p k All of these are non-basic weights.

[0012] Preferably, in S2, the non-basic weight items are split into units of basic weights, specifically by rewriting the non-basic weights in the form of equation (1) or equation (2).

[0013] Preferably, the non-basic weights and basic weights are in the same convolution kernel. For convolution operations: x i w p i +x j w p j +x k w p k +… S3 is optimized by: x i w p i +x j w p j +x k w p k +…=w p j (Ax i +x j +x k )+x k 2 m +… Among them, Ax i By x i Obtained by shifting, x k 2 m By x k Obtained by shifting.

[0014] Preferably, A=2 n ±1, Ax i By x i After shifting left by n bits, ±x i Obtain; x k 2 m By x k Obtained by shifting left by m bits.

[0015] Preferably, if the non-basic weights and the basic weights are not in the same convolution kernel, then the optimization calculation described in S3 is: to calculate the non-basic weights in the form rewritten in S2; Specifically, regarding: Convolution operation in kernel P1: x i w p j +… Convolution operation in kernel P2: x i w p i +… Convolution operation in kernel P3: x i w p k +… S3 is optimized by: The output of convolution kernel P1 is: x i w p j +… The output of convolution kernel P2 is: Ax i w p j +… The output of convolution kernel P3 is: x i w p j +x i 2 m +… Among them, Ax i By x i Obtained by shifting, x i 2 m By x i Obtained by shifting.

[0016] Preferred: A=2 n ±1, Ax i By x i After shifting left by n bits, ±x i Obtain; x i 2 m By x i Obtained by shifting left by m bits.

[0017] Beneficial effects of the present invention This invention utilizes arithmetic operation rules, binary number decomposition, and shift adders to reduce multiplication operations in convolution operations, thereby optimizing the computation of neuron convolution kernels in deep learning networks.

[0018] This invention targets programmable processors, such as VLIW, CPUs with digital signal processing extensions, and dataflow processors. These processors have multiple adders and multipliers, and some also include dividers and transcendental function accelerators. In these processors, addition and multiplication are typically completed within one clock cycle, although their power consumption differs significantly. In some processors, multiplication may take two clock cycles, but this is far less than the number of additions that a multiplication operation can be broken down into. For 8-bit binary multiplication, a multiplication operation can be broken down into an average of four additions, requiring four clock cycles to complete, while a single multiplication operation only requires one to two clock cycles. Therefore, reducing the number of multiplication operations not only lowers power consumption but also increases computational speed.

[0019] This invention aims to optimize the computation of convolutional networks on programmable processors, specifically by reducing the number of multiplication operations, thereby reducing the total number of operations. Its purpose and application scope differ from the two comparative references (CN110059817A and CN110059817A). We also conducted optimization experiments using the common ResNet convolutional network, achieving a reduction of the number of multiplication operations to approximately 50-60%. The use of a weighted index reading method also significantly improves performance on programmable processors. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the present invention. Figure 2 This is a schematic diagram showing the 4x4 convolution kernel moving along a specified direction and span in the embodiment. Detailed Implementation

[0021] The present invention will be further described below with reference to embodiments, but the scope of protection of the present invention is not limited thereto: Combination Figure 1 An optimization method for calculating the convolutional kernel of neurons in deep learning networks, the optimization method includes the following steps: S1. Identify the basic weights, non-basic weights, and irrelevant weights in the convolution operation. Items containing basic weights are denoted as basic weight items, items containing non-basic weights are denoted as non-basic weight items, and items containing irrelevant weights are denoted as irrelevant items. S2. Split the non-basic weight items into units of basic weight; S3. Optimize the calculations of the split items and the basic weight items; S4. Output the convolution operation result.

[0022] This invention primarily considers 1) optimization methods within a single convolutional kernel, and 2) optimization methods across multiple convolutional kernels within the same layer. For example... Figure 2 As shown, the four convolutional kernels move along a specified direction and a specified stride. The red dashed line in the image passes through the same point of all four convolutional kernels (on the image).

[0023] For image processing, an (n x n) convolution kernel typically computes the following function: Let N = n × n, p = i × n + j. Expanding the two-dimensional data into one-dimensional data, we obtain: The above formula transforms a two-dimensional sum into an equivalent one-dimensional sum.

[0024] When the stride of the convolutional kernel is less than n, there will be overlaps in the computation of adjacent convolutions. In addition to considering the two optimization methods mentioned above, the computational optimization between adjacent convolutional kernels also needs to be considered.

[0025] This invention utilizes arithmetic operation rules, binary number decomposition, and shift adders to reduce multiplication operations in convolution. For example, when (w p i If the value is 0, then we do not need to calculate w. p i x i Because the result is 0. Or, when w in a convolution kernel p i＝ w p j We only need to calculate w p i (x i +x j Therefore, one multiplication operation is saved.

[0026] The purpose of this invention is to save as many multiplication and addition operations as possible.

[0027] i. Optimization methods within the same convolutional kernel In calculating x i w p i +x j w p j In total, two multiplications and one addition were used. If a weight w p i It is another weight w p iFor integer multiples of , we can optimize using the following formula.

[0028] If (w p i ==w p j ), calculate and output w p i (x i +x j ); If (w p i ==2w p j ), calculate and output w p i (x i +(x j <<1)); If (w p i ==3w p j ), calculate and output w p i (x i +x j +(x j <<1)); If (w p i ==4w p j ), calculate and output w p i (x i +(x j <<2)); If (w p i ==5w p j ), calculate and output w p i (x i +x j +(x j <<2)); If (w p i ==7w p j ), calculate and output w p i (x i -x j +(x j <<3)); If (w p i ==8w pj ), calculate and output w p i (x i +(x j <<3)); If (w p i ==9w p j ), calculate and output w p i (x i +x j +(x j <<3)); In the calculations above, we limited the total number of output operations for each group to three, counting multiplication as one operation and shift-add as one operation. Of course, shift-adders are simpler than multipliers. The calculations are listed in Table 1, with the first column showing the number of multiplications and shift-adds.

[0029] Table 1 shows one example of optimized computation within the same convolutional kernel. According to the table above, a series of products sharing a common divisor in the convolution kernel can be chained together using a multiplication and a set of shift additions. The result has the following form: Multiplication is performed only once; all other operations are shift-and-add operations.

[0030] In calculating x i w p i +x j w p j When a weight differs from another by a power of 2, the following method can be used to calculate the difference in order to reduce the number of multiplication operations.

[0031] If (w p j =(w p i +1), calculate and output w p i (x i +x j )+x j ; If (w p j =(w p i +2), calculate and output w pi (x i +x j )+(x j <<1); If (w p j =(w p i +4), calculate and output w p i (x i +x j )+(x j <<2); If (w p j =(w p i +8), calculate and output w p i (x i +x j )+(x j <<3); The above calculation process and the number of operations are listed in Table 2: Table 2 shows an example of optimized computation within the same convolutional kernel (2). The table above is suitable for adding two product terms, reducing one multiplication operation. We can also calculate the result of three product terms as w. p i (x i +x j )+(x j <<1)+w p k x k =w p i (x i +x j )+(x j <<1)+(w p i +4)x k =w p i (x i +x j +x k )+(x j <<1)+4x k =w p i (x i +x j +x k )+(x j <<1)+(x k<<2). A total of 1 multiplication and four shift additions were used. We summarized 14 cases, listed in Table 3. In Table 3, most calculations used fewer than 4 operations, and a few used 5 operations. Overall, the number of operations was reduced, and only one multiplication was used.

[0032] Table 3 shows example 3 of optimized operations within the same convolutional kernel. In a more preferred embodiment, the specific details are as follows: Step 1. / / Calculation of the terms with the greatest common multiple in the combined weights FOR(each kernel p) Sort all kernel weights in ascending order and send the results to the S linked list; Reset all (marked and used) labels; FOR(i=1 to N) / / Process according to weight order FOR(j=i+1to N) IF(w[p][j] is an integer multiple of w[p][i] && w[j] .marked == false) w[p][j].marked = true; Add w[p][j] to the linked list list[i]; FOR(each list[i]) FOR(each weight w[p][j]in list[i]) IF(w[p][j].used==false) w[p][j].used = true; Calculate the required additive shift value for w[p][j] according to Table 1; Step 2. / / Calculation of items with slightly different weights FOR(eachkernelp) Reset all (checked) flags; FOR(i=1 to N) / / Process according to weight order FOR(j=i+1toN) IF(w[j] satisfies the condition in Table2 && w[j].checked == false) w[j].checked = true; FOR(k=j+1toN) IF(w[i],w[j],w[k] satisfies the conditions in Table 3) &&w[k].checked==false) w[k].checked = true; According to Table 3, combine w[i], w[j], and w[k] to calculate the required shift and addition values; ELSE According to Table 2, combine w[i] and w[j] to calculate the required addition shift value; Exit the innermost (k) loop; Step 3. / / Merge all items in a single kernel FOR(eachkernelp) combines the results of step 1 and step 2, along with the remaining items, to obtain the total result. ii. Optimized computation across a set of convolutional kernels Consider two different convolutional kernels p1 and p2. If a weight w p1 i It is another weight w p2 i For integer multiples of p1 and p2, we can optimize using the following formula. In the formula below, we use 1 and 2 to represent p1 and p2.

[0033] If (w 2 i ==w 1 i ), calculate and output <w 1 i x i ,w 1 i x i >; If (w 2 i ==2w 1 i ), calculate and output <w 1 i x i ,w 1 i x i < <1> ; If (w 2 i ==3w 1 i ), calculate and output <w 1 i x i ,(w 1 i x i <<1)+w1 i x i >; If (w 2 i ==4w 1 i ), calculate and output <w 1 i x i ,(w 1 i x i <<2)>; If (w 2 i ==5w 1 i ), calculate and output <w 1 i x i ,(w 1 i x i <<2)+w 1 i x i >; If (w 2 i ==7w 1 i ), calculate and output <w 1 i x i ,(w 1 i x i <<3)-w 1 i x i >; If (w 2 i ==8w 1 i ), calculate and output <w 1 i x i ,(w 1 i x i <<3)>; If (w 2 i ==9w 1 i ), calculate and output <w 1 i x i ,(w 1 i x i <<3)+w 1i x i >; This set of calculations outputs two results, which are listed in Table 4: Table 4. Example of optimization operation across a set of convolutional kernels (1) For a kernel that meets the above conditions, output1 is always w. 1 i x i ,output2 is (w 1 i x i < <m’)±w 1 i x i The more kernels that can be combined, the more multiplications are saved.

[0034] In calculating x i w p1 i With x i w p2 i When one weight differs from another by a small number, the following method can be used to reduce the number of multiplication operations to zero. The total number of operations is limited to three.

[0035] If (w 2 i ==(w 1 i +1), calculate and output. <w 2 i x i ,w 2 i x i +x i >; If (w 2 i ==(w 1 i +2), calculate and output. <w 2 i x i ,w 2 i +(x i <<1)>; If (w 2 i ==(w 1 i +3), calculate and output. <w 2i x i ,w 2 i x i +(x i <<1)+x i >; If (w 2 i ==(w 1 i +4), calculate and output. <w 2 i x i ,w 2 i x i +(x i <<2)>; If (w 2 i ==(w 1 i +5), calculate and output. <w 2 i x i ,w 2 i x i +(x i <<2)+x i >; If (w 2 i ==(w 1 i +7), calculate and output. <w 2 i x i ,w 2 i x i +(x i <<3)-x i >; If (w 2 i ==(w 1 i +8), calculate and output. <w 2 i x i ,w 2 i x i +(x i <<3)>; If (w 2 i ==(w 1 i +9)), calculate and output. <w 2 ix i ,w 2 i x i +(x i <<3)+x i >; The results of the above calculations are summarized in Table 5: Table 5 Example 2 of optimization operations across a set of convolutional kernels A kernel that meets the above conditions will always output output1 as w. 1 i x i ,output2 is (w 1 i x i +w 1 i x i < <m’± i ·pred(w 2 i Here, pred(w) = 0 if w ∈ {2, 4, 8}; pred(w) = 1 if w ∈ {3, 5, 9}; pred(w) = -1 if w = 7.

[0036] In a more preferred embodiment, the specific details are as follows: Step 1. / / Calculation of the terms with the greatest common multiple in the combined weights FOR(eachweightpositionk) Sort all kernelp values ​​at position k in ascending order of weight, and store the results in the linked list T[k]. Reset all marked and used labels; FOR(eachweightpositionk) FOR(p=1toP) / / kernel sorted according to the T[k] linked list Clear mlist[p]; mlist[p].count = 0; FOR(j=i+1toP) IF(w[j][k] is an integer multiple of w[p][k] && w[j][k].marked == false) w[j][k].marked = true; Add w[j][k] to mlist[p]; mlist[p].count++; FOR(p=1toP) IF(mlist[p] NOT NULL) value[p][k] = w[p][k] x[k]; FOR(eachweightw[j]inmlist[p]) IF(w[j].used==false) w[j].used = true; Calculate the required additive shift value for w[j][k] according to Table 4; Step 2. / / Calculation of items with slightly different weights FOR(eachweightpositionk) Reset all (checked) flags; FOR(eachweightpositionk) FOR(p=1toP) / / kernel sorted according to the T[k] linked list FOR(j=p+1toN) IF(w[j][k] satisfies the conditions in Table 5 && w[j][k].checked == false) w[j].checked = true; Calculate the required additive shift value for w[j][k] according to Table 5; The operations in the five tables above can save us a lot of multiplication operations, and the total number of operations can also be reduced.

[0037] iii. Combining different types of operations First consider optimizations for combinations within the same kernel.

[0038] If r·w p i =w p j With w p k =w p i +q, we can calculate w p i x i +w p j x j +w p k x k =w p i x i +r·w pi x j +(w p i +q)x k =w p i (x i +r·x j +x k )+q·x k .

[0039] ·If both r and q are a certain power of 2, then w p i (x i +r·x j +x k )+q·x k =w p i (x i +x j <<r'+x k )+x k <<q', which uses one multiplication and three shift-add operations in total.

[0040] ·If r is a certain power of 2 and q∈{3,5,7,9}, w p i (x i +r·x j +x k )+q·x k =w p i (x i +x j <<r'+x k )+(x k <<q'±x k ), which uses one multiplication and four shift-add operations in total.

[0041] ·If r and q∈{3,5,7,9}, w p i (x i +r·x j +x k )+qx k =w p i (x i +x j <<r'±x j +x k )+x k <<q'±x k , which uses one multiplication and five shift-add operations in total. 2 multiplications are saved, and one more shift-add operation is used, which is still cost-effective.

[0042] If r·w p i =w p j and w p k =w p j +q, we can calculate w p i x i +w p j x j +w p k x k =w p i x i +r·w p i x j +(w p j +q)x k =w p i x i +r·w p i x j +(r·w p i +q)x k =w p i (x i +r·x j +r·x k )+q·x k . · If both r and q are a certain power of 2, then w p i (x i +r·x j +r·x k )+q·x k =w p i (x i +x j <<r’+x k <<r)+x k <<q’, one multiplication and three shift-add operations are used in total.

[0043] · If r is a certain power of 2 and q∈{3,5,7,9}, w p i (x i +r·x j +x k )+q·x k=w p i (x i +x j < <r’+x k < <r’)+(x k < <q’±x k It uses one multiplication and four shift additions.

[0044] If r and q∈{3,5,7,9}, w p i (x i +r·x j +x k )+qx k =w p i (x i +x j < <r’±x j +x k < <r’±x k )+x k < <q’±x k It uses one multiplication operation and six shift addition operations, which may not be very cost-effective.

[0045] Secondly, consider optimizations that combine multiple kernels.

[0046] If r·w 1 i =w 2 i And there is w 1 i =w 3 i +q, we need to calculate <w 1 i x i ,w 2 i x i ,w 3 i x i >= <w 1 i x i ,r·w i x i ,(w 1 i -q)x i > If r and q are both powers of 2, then we have <w 1 i x i ,r·w 1 i x i,(w 1 i -q)x i >= <w 1 i x i ,w 1 i x i < <r’,w i x i -x i <<q’> It uses one multiplication and two shift additions.

[0047] If r is some power of 2 and q∈{3,5,7,9}, <w 1 i x i ,r·w 1 i x i ,(w 1 i -q)x i >= <w 1 i x i ,w 1 i x i < <r’,w 1 i x i -x i < <q’±x i It uses one multiplication and three shift additions.

[0048] If r and q ∈ {3, 5, 7, 9}, <w 1 i x i ,r·w 1 i x i ,(w 1 i -q)x i >= <w 1 i x i ,w 1 i x i < <r’±w 1 i x i ,w i x i -x i < <q’±x i It uses one multiplication and three shift additions.

[0049] If r·w 1i =w 2 i And there is w 2 i =w 3 i +q, we need to calculate <w 1 i x i ,w 2 i x i ,w 3 i x i >= <w 1 i x i ,r·w i x i ,(w 2 i -q)x i >= <w 1 i x i ,r·w 1 i x i ,(r·w 1 i -q)x i >= <w 1 i x i ,r·w 1 i x i ,r·w 1 i x i -qx i > If r and q are both powers of 2, then we have <w 1 i x i ,r·w 1 i x i ,r·w 1 i x i -qx i >= <w 1 i x i ,w 1 i x i < <r’,w i x i < <r’-x i <<q’> It uses one multiplication and two shift additions.

[0050] If r is some power of 2 and q∈{3,5,7,9}, <w 1 i x i ,r·w 1 i x i ,r·w 1 i x i -qx i >= <w 1 i x i ,w i x i < <r’,w 1 i x i < <r’-x i < <q’±x i It uses one multiplication and three shift additions.

[0051] If r and q ∈ {3, 5, 7, 9}, <w 1 i x i ,r·w 1 i x i ,r·w 1 i x i -qx i >= <w 1 i x i ,w 1 i x i < <r’±w 1 i x i ,w i x i < <r’±w 1 i x i -qx i < <q’±x i It uses one multiplication and three shift additions.

[0052] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for optimizing the computation of convolutional kernels in neurons of deep learning networks, characterized in that, Applied to image processing, given an input image, the weighted average of pixels in a small region of the input image becomes each corresponding pixel in the output image, where the weights are defined by a function called the convolution kernel. The optimization method includes the following steps: S1. Identify the basic weights, non-basic weights, and irrelevant weights in the convolution operation. Items containing basic weights are denoted as basic weight items, items containing non-basic weights are denoted as non-basic weight items, and items containing irrelevant weights are denoted as irrelevant items. The method for obtaining the basic weights is as follows: sort all the weights and take the weight with the smallest value as the basic weight. Let the basic weights be denoted as Weights that meet the following conditions are considered as non-basic weights: （1） or (2) in, For any positive integer, Any natural number, , All are non-basic weights; Weights other than basic weights and non-basic weights are considered irrelevant weights. S2. The non-basic weight items are split into units of basic weights. Specifically, the non-basic weights are rewritten in the form of formula (1) or formula (2). S3. Optimize the calculations of the split items and the basic weight items; S4. Output the convolution operation result.

2. The method according to claim 1, characterized in that, When non-basic weights and basic weights are in the same convolution kernel, for the convolution operation: ； S3 is optimized by: ； in, By Obtained by shifting. By Obtained by shifting.

3. The method according to claim 2, characterized in that, , By Shift left After that, get; By Shift left The position was obtained.

4. The method according to claim 1, characterized in that, If the non-basic weights are not in the same convolution kernel as the basic weights, then the optimization calculation described in S3 is: to calculate the non-basic weights in the form rewritten in S2; Specifically, regarding: Convolution operation in kernel P1: , Convolution operation in kernel P2: , Convolution operation in kernel P3: ; S3 is optimized by: The output of convolution kernel P1 is: , The output of convolution kernel P2 is: , The output of convolution kernel P3 is: ; in, By Obtained by shifting. By Obtained by shifting.

5. The method according to claim 4, characterized in that: , By Shift left After that, get; By Shift left The position was obtained.

Images