Method for improving hardware utilization rate of input layer output 16 channels

Abstract

The invention provides a method for improving the hardware utilization rate of 16 output channels of an input layer, which comprises the following steps of: S1, combining two adjacent points on the width ow of an output feature in pairs by combining the MAC (Media Access Control) operation of an NNA (Network Network Alliance) and the characteristic that only four channels exist in the input feature of the input layer, and rearranging input points corresponding to the two combined output points; s2, performing MAC (Media Access Control) calculation on the rearranged input points, and for each row of output feature, the front 16 output channels in 32 output channels of the 0th output point obtained by the 0th MAC calculation of the current output row are the 0th point of the current output feature row, and the rear 16 output channels correspond to the first point of the current output feature row; similarly, the front 16 output channels in 32 output channels of the 0th output point calculated by the Xth MAC of the current output line are 2 * Xth points of the current output feature line, and the rear 16 output channels correspond to the 2 * X + 1 points of the current output feature line; wherein X is equal to ow / N, and N is the number of columns of output points obtained through fixed calculation of each MAC.

Description

Technical Field

[0001] The present invention belongs to the technical field of neural network processing, and in particular relates to a method for improving the hardware utilization rate of 16 output channels of an input layer. Background Art

[0002] In the prior art, NNA stands for Neural Networks Accelerate, a hardware accelerator on the CPU's SIMD pipeline. Its operation is controlled by special CPU / SIMD instructions and runs on one thread to solve most convolution multiplication and addition. Since NNA itself only needs to configure relevant parameters (convolution kernel size, input resolution, number of input channels, etc.), NNA can perform related convolution calculations on its own. And NNA uses MAC as the basic computing unit, and each MAC can easily calculate the convolution result of 32 channels of one output point.

[0003] When the output channel of the input layer is 16, the output channel is simply filled to 32 channels. In this way, NNA can be used to quickly implement convolution.

[0004] However, since the computing unit of NNA is 1 MAC, and each MAC can easily calculate the convolution result of 32 channels of 1 output point, the above solution simply fills the 16 output channels into 32 channels. Under this solution, the valid calculation results of each MAC output point are only the first 16 channels, and the last 16 channels are invalid values, thus generating redundancy.

[0005] In addition, common terms in the prior art include:

[0006] 1. SIMD instruction: The full name is Single Instruction Multiple Data, a set of instructions that can copy multiple operands and pack them into large registers.

[0007] 2. MAC: NNA-based convolutional multiplication and computation unit.

[0008] 3. BGRA: pixel format.

[0009] 4. Bit: A computer professional term, it is a unit of information volume and is a transliteration of the English word BIT. Summary of the invention

[0010] In order to solve the above problems, the purpose of this application is to fully utilize the redundancy of the output channel of NNA when performing MAC operations, improve the utilization rate of each MAC calculation, and reduce the overall calculation amount.

[0011] Specifically, the present invention provides a method for improving the hardware utilization of 16 channels of input layer output, the method comprising the following steps:

[0012] S1, combined with the characteristics of NNA's MAC operation, and when the input feature of the input layer of the neural network is in BGRA format, the input feature has only 4 channels; the 0th and 1st points, the 2nd and 3rd points...the ow-2nd and ow-1th points of each row of the output feature, that is, the two adjacent points on the output feature width ow are combined, and the input points corresponding to the two combined output points are rearranged;

[0013] The characteristic of the MAC operation of the NNA is that the NNA calculation is based on MAC operation, and each MAC performs at least KH*KW*J data multiplication and accumulation operations, and finally calculates M*N K channel output points, where KH and KW refer to the height and width of the convolution kernel kernel; J refers to the number of primitive channels for the NNA MAC to perform multiplication and accumulation operations, which is also the minimum number of channels; M and N refer to the output points of M rows and N columns calculated by each MAC; K refers to the number of channels of the output points that can be calculated by each MAC, that is, each MAC operation can obtain a fixed result of M*N K channel output points;

[0014] The rearrangement of the input points is to rearrange the KH*KW*IC input points corresponding to each output point of the convolution in the order of 1*1*P, where:

[0015] P = KH*KW*IC;

[0016] Before sorting, the shape of the weight is KH*KW*IC*OC. After rearrangement, the shape of the weight becomes 1*1*Q*R, where R=2*OC, and OC is the number of channels of the output feature.

[0017] Assume that ow0 and ow1 are two output points, and explain the weight reordering. Assume that i and j are the subscripts of Q and R respectively, where:

[0018] 0≤i<Q,0≤j<R

[0019] When 0≤j<OC, 0≤i<P, the P weight data at this time correspond to the KH*KW*IC weights of the output point ow0 before sorting, and the corresponding rule of the rearrangement subscript is i=x*KW*IC+y*IC+z, which means that the weight of the xth row, yth column, zth input channel of the jth output channel before sorting corresponds to the weight of the 0th row, 0th column, ith input channel of the jth output channel after reordering;

[0020] When \(0\leq j\lt oc, p\leq i\lt Q\), the \(P\) weights at this time are due to the influence of the superposition of the weights corresponding to two adjacent output points, which doubles the number of channels. To eliminate interference and keep the MAC calculation unchanged before and after sorting, the \(P\) weights in this range are all filled with \(0\).

[0021] When \(oc\leq j\lt R, 0\leq i\lt P\), the same processing procedure as when \(0\leq j\lt oc, p\leq i\lt Q\) is followed, and all \(P\) weights are filled with \(0\).

[0022] When \(oc\leq j\lt R, P\leq i\lt Q\), the \(P\) weight data at this time is obtained by rearranging the \(KH\times KW\times IC\) weights of the output point \(ow1\) before sorting. The corresponding rule for the rearrangement subscript is \(i = x\times KW\times IC + y\times IC+z + P\), indicating that the weight of the \(z\)-th input channel in the \(y\)-th column and \(x\)-th row of the \((j - oc)\)-th output channel before sorting corresponds to the weight of the \(i\)-th input channel in the 0-th column and 0-th row of the \(j\)-th output channel after rearrangement.

[0023] S2. Perform MAC calculation on the rearranged input points.

[0024] Among the 32 output channels of the 0-th output point obtained by the 0-th MAC calculation of the current output row, the first 16 output channels are the 0-th point of the current output feature row, and the last 16 output channels correspond to the 1-st point of the current output feature row.

[0025] Among the 32 output channels of the 0-th output point obtained by the 1-st MAC calculation of the current output row, the first 16 output channels are the 2-nd point of the current output feature row, and the last 16 output channels correspond to the 3-rd point of the current output feature row.

[0026] And so on. Similarly, among the 32 output channels of the 0-th output point obtained by the \(X\)-th MAC calculation of the current output row, the first 16 output channels are the \(2\times X\)-th point of the current output feature row, and the last 16 output channels correspond to the \((2\times X + 1)\)-th point of the current output feature row.

[0027] Where:

[0028] \(X = ow / N\), where \(N\) is the number of columns of output points fixed by each MAC calculation.

[0029] In the characteristics of the MAC operation of the NNA,

[0030] If the number of channels \(IC\) of the input feature is less than \(J\), each MAC still calculates according to the minimum input channel calculation unit \(J\). Then, in this case, the extra \(J - IC\) channels are redundant calculations.

[0031] Similarly, if the channel OC of the output feature is less than K, each MAC will still perform calculations according to the minimum output channel calculation unit K. In this case, the subsequent K-OC channels of the K channels of M*N output points calculated by each MAC are also redundant calculations.

[0032] The method combines the characteristics of the MAC of the NNA and utilizes the redundant calculation of the MAC to improve efficiency when the output channels are less than K:

[0033] Take KH=3, KW=3, J=32, M=1, N=1, K=32 as example parameters, and combine with the neural network input layer description: 3*3 convolution kernel, 8-bit input, 8-bit weight, 8-bit output input layer convolution, then the input layer convolution corresponding to one convolution of the input layer has an effective data volume of 3*3*3=27;

[0034] Based on the MAC operation characteristics of NNA, the 3*3 convolution of the input layer, corresponding to the valid data of 3 channels, is equivalently converted into a 1*1 convolution, corresponding to the valid data of 27 channels.

[0035] The rearrangement is to rearrange the KH*KW*IC input points corresponding to the current output point into data of shape 1*1*P according to the priority order of channel, width and height;

[0036] The point located at the xth row, yth column, and zth channel before reordering is located at the x*KW*IC+y*IC+zth channel at the 0th row, 0th column after reordering; where: 0≤x<KH,0≤y<KW,0≤z<IC

[0037] Then, the 1*1*P data corresponding to the output points of the adjacent combinations are superimposed to obtain 1*1*Q data, where Q=2*P;

[0038] Correspondingly, the KH*KW*IC weights corresponding to the KH*KW*IC input points corresponding to the current output point must also be reordered to ensure that the corresponding convolutions before and after the sorting are equivalent.

[0039] Therefore, the advantages of the present application are: the method is simple, and the method utilizes the redundancy of the output channel of the NNA when performing MAC operations, thereby improving the utilization rate of each MAC calculation and reducing the overall amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0041] Figure 1 is a schematic diagram of the process of this method.

[0042] Figure 2 It is a schematic diagram of data rearrangement in this method. DETAILED DESCRIPTION

[0043] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0044] The present application relates to a method for improving the hardware utilization of 16 channels of input layer output, such as Figure 1 As shown, the following steps are included:

[0045] S1, combined with the characteristics of NNA's MAC operation, NNA calculation is based on MAC operation, each MAC performs at least KH*KW*J data multiplication and accumulation operations, and finally calculates M*N K channel output points. Among them, KH and KW refer to the height and width of the convolution kernel; J refers to the number of primitive channels for NNA MAC to perform multiplication and accumulation operations, which is also the minimum number of channels. M and N refer to the output points of M rows and N columns calculated by each MAC; K refers to the number of channels of the output points that can be calculated by each MAC. That is, each MAC operation can get a fixed result of M*N K channel output points.

[0046] If the channel IC of the input feature is less than J, each MAC still performs calculations according to the minimum input channel calculation unit J. In this case, the extra J-IC channels are redundant calculations.

[0047] Similarly, if the channel OC of the output feature is less than K, each MAC will still perform calculations according to the minimum output channel calculation unit K. In this case, the subsequent K-OC channels of the K channels of M*N output points calculated by each MAC are also redundant calculations.

[0048] The technology of the present invention is to combine the characteristics of the MAC of the above NNA and to fully utilize the redundant calculation of the MAC when the output channels are less than K to improve the efficiency.

[0049] The following will use KH=3, KW=3, J=32, M=1, N=1, K=32 as example parameters, and combine them with the neural network input layer to explain the main steps of this technology.

[0050] When the feature input in the input layer of the neural network is in BGRA format, the input feature has only 4 channels.

[0051] Combine the 0th and 1st points, the 2nd and 3rd points, ... the ow-2th and ow-1th points of each row of the output feature, that is, combine two adjacent points on the output feature width (ow), and rearrange the input points corresponding to the two combined output points;

[0052] The rearrangement of input points refers to rearranging the KH*KW*IC input points corresponding to each output point of the convolution in the order of 1*1*P, where:

[0053] P = KH*KW*IC;

[0054] Combination Figure 2 As shown, the rearrangement is to rearrange the KH*KW*IC input points corresponding to the current output point into data of shape 1*1*P in the order of channel first, width, height. The point located at the xth row, yth column, zth channel before the rearrangement is located at the xth*KW*IC+y*IC+zth channel at the 0th row, 0th column after the rearrangement. Among them: 0≤x<KH, 0≤y<KW, 0≤z<IC;

[0055] Then, the 1*1*P data corresponding to the output points of the adjacent combinations are superimposed to obtain 1*1*Q data, where Q=2*P;

[0056] Correspondingly, the KH*KW*IC weights corresponding to the current output point and the KH*KW*IC weights corresponding to the input point must also be reordered to ensure that the corresponding convolutions before and after the sorting are equivalent.

[0057] Before sorting, the shape of the weight is KH*KW*IC*OC. After rearrangement, the shape of the weight becomes 1*1*Q*R, where R=2*OC, oc is the number of channels of the output feature;

[0058] Take the combination of two output points ow0 and ow1 as an example to explain the weight reordering. Assume that i and j are the subscripts of Q and R respectively, where:

[0059] 0≤i<Q,0≤j<R

[0060] When 0≤j<OC, 0≤i<P, the P weight data at this time correspond to the KH*KW*IC weights of the output point ow0 before sorting, and the corresponding rule of the rearrangement subscript is i=x*KW*IC+y*IC+z, which means that the weight of the xth row, yth column, zth input channel of the jth output channel before sorting corresponds to the weight of the 0th row, 0th column, ith input channel of the jth output channel after reordering;

[0061] When \(0\leq j\lt oc, p\leq i\lt Q\), the \(P\) weights at this time are due to the influence of the superposition of the weights corresponding to two adjacent output points, which doubles the number of channels. To eliminate interference and keep the MAC calculation unchanged before and after sorting, the \(P\) weights in this range are all filled with \(0\);

[0062] When \(oc\leq j\lt R, 0\leq i\lt P\), the same processing procedure as when \(0\leq j\lt oc, p\leq i\lt Q\) is adopted, and all \(P\) weights are filled with \(0\);

[0063] When \(oc\leq j\lt R, P\leq i\lt Q\), the corresponding \(P\) weight data at this time are rearranged from the \(KH\times KW\times IC\) weights of the \(ow1\) output point before sorting. The corresponding rule of the rearrangement subscript is \(i = x\times KW\times IC + y\times IC + z + P\), indicating that the weight of the \(z\)-th input channel in the \(x\)-th row and \(y\)-th column of the \((j - oc)\)-th output channel before sorting corresponds to the weight of the \(i\)-th input channel in the \(0\)-th row and \(0\)-th column of the \(j\)-th output channel after rearrangement;

[0064] S2. Perform MAC calculation on the rearranged input points,

[0065] Among the 32 output channels of the 0-th output point obtained by the 0-th MAC calculation of the current output row, the first 16 output channels are the 0-th point of the current output feature row, and the latter 16 output channels correspond to the 1-st point of the current output feature row;

[0066] Among the 32 output channels of the 0-th output point obtained by the 1-st MAC calculation of the current output row, the first 16 output channels are the 2-nd point of the current output feature row, and the latter 16 output channels correspond to the 3-rd point of the current output feature row; and so on, ……

[0067] Similarly, among the 32 output channels of the 0-th output point obtained by the \(X\)-th MAC calculation of the current output row, the first 16 output channels are the \(2\times X\)-th point of the current output feature row, and the latter 16 output channels correspond to the \((2\times X + 1)\)-th point of the current output feature row;

[0068] Where:

[0069] \(X = ow / N\), where \(N\) is the number of columns of output points fixed by each MAC calculation.

[0070] The method assumes that:

[0071] For the input layer convolution with a \(3\times3\) convolution kernel, 8-bit input, 8-bit weights, and 8-bit output, then the effective data volume of the input FeatureMap corresponding to one convolution of the input layer is \(3\times3\times3 = 27\);

[0072] Based on the MAC operation characteristics of NNA, the 3*3 convolution of the input layer, corresponding to the valid data of 3 channels, is equivalently converted into a 1*1 convolution, corresponding to the valid data of 27 channels.

[0073] In summary, the present application provides a method for improving the hardware utilization of the 16 channels of the input layer output. The core of this solution is to combine the characteristics of the MAC operation of the NNA and the particularity of the input feature feature of the input layer (only 4 channels), and combine the output points 0 and 1, 2 and 3, and other adjacent points in pairs, and rearrange the input points corresponding to the two combined output points. In this way, when the MAC calculation is performed on the rearranged input points, the first 16 output channels of the 32 output points of the 0th output point obtained by the 0th MAC calculation are the 0th point of the output feature, and the last 16 output channels correspond to the first point of the output feature; the first 16 output channels of the 32 output points of the output point obtained by the 1st MAC calculation are the second point of the output feature, and the last 16 output channels correspond to the third point of the output feature, and so on. Figure 2 shown.

[0074] Taking the input layer convolution with 3*3 convolution kernel, 8-bit input, 8-bit weight and 8-bit output as an example, the effective data volume of input FeatureMap corresponding to one convolution of the input layer is 3*3*3=27. Based on the MAC operation characteristics of NNA, the 3*3 convolution of the input layer (corresponding to the effective data of 3 channels) can be equivalently converted to 1*1 convolution (corresponding to the effective data of 27 channels).

[0075] Then the utilization rate of each MAC in the existing solution is (27 / 32)*(16 / 32)=42.1875%, where 27 in 27 / 32 refers to the valid input data in each MAC calculation, 32 refers to the multiplication and accumulation of 32 input channels each time the MAC calculation is performed; and 16 / 32 refers to the calculation results of 32 output channels of each output point obtained by each MAC operation, in which only 16 channels are valid.

[0076] MAC time = convolution kernel length * convolution kernel width * (number of input channels calculated by MAC) * (32), where 32 means that each MAC can calculate the convolution result of 32 channels of one output point.

[0077] Therefore, the running time of each MAC in the existing solution is 1*1*(32)*(32)=1024.

[0078] In summary, the overall computational complexity of the existing solution = (ow)*oh*(each MAC time + quantization time), where ow and oh are the width and height of the output feature.

[0079] Accordingly, the utilization rate of each MAC of the solution described by the present method is (27*2 / 64)*(32 / 32)=84.375%, wherein 27*2 is because the solution superimposes the input points corresponding to two adjacent output points in one MAC, and (32 / 32) is because all 32 channels of the output points obtained by each MAC operation of the solution are valid channels.

[0080] The running time of each MAC of the solution is 1*1*(64)*(32)=2048.

[0081] The total computational effort of the solution = (ow / 2)*oh*(each MAC time + quantization time). (ow / 2) is because the input channel of each MAC operation of the solution is to superimpose the inputs corresponding to two adjacent output points.

[0082] In summary, compared with the original solution, the solution described in this method increases the utilization rate of each MAC from 42.1875% to 84.375%.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for improving the hardware utilization of 16 channels of input layer output, characterized in that: The method includes the following steps: S1. Combining the characteristics of the MAC operation of the NNA, and when the input feature in the input layer of the neural network is in the BGRA format, there are only 4 channels in the input feature at this time; for each row of the output feature, the 0th and 1st points, the 2nd and 3rd points... the (ow - 2)th and (ow - 1)th points are combined, that is, the two adjacent points on the width ow of the output feature are combined pairwise, and the input points corresponding to the two combined output points are rearranged. The characteristics of the MAC operation of the NNA are that the NNA calculation is based on the MAC operation. Each MAC performs at least KH * KW * J times of multiplication and accumulation operations on data, and finally calculates the results of M * N K-channel output points. Among them, KH and KW refer to the height and width of the convolution kernel kernel; J refers to the primitive channel number of the multiplication and accumulation operation of the NNA MAC, which is also the minimum channel number; M and N refer to the M-row and N-column output points calculated by each MAC; K refers to the number of channels of the output points that can be calculated by each MAC, that is, each MAC operation can fixedly obtain the results of M * N K-channel output points. The rearrangement of the input points is to rearrange the KH * KW * IC input points corresponding to each output point of the convolution in the order of 1 * 1 * P, where: P = KH * KW * IC; Before sorting, the shape of the weight is KH * KW * IC * OC. After rearrangement, the shape of the weight becomes 1 * 1 * Q * R, where R = 2 * OC and OC is the number of channels of the output feature. Assume that ow0 and ow1 are two output points, and an explanation of the weight rearrangement is given. Assume that i and j are the subscripts of Q and R respectively, where: 0 ≤ i < Q, 0 ≤ j < R When 0 ≤ j < OC and 0 ≤ i < P, the P weight data at this time are correspondingly rearranged from the KH * KW * IC weights before sorting of the output point ow0. The corresponding rule of the rearrangement subscript is i = x * KW * IC + y * IC + z, indicating that the weight of the z-th input channel in the x-th row and y-th column of the j-th output channel before sorting corresponds to the weight of the i-th input channel in the 0th row and 0th column of the j-th output channel after rearrangement. When 0 <= j < oc and p <= i < Q, the P weights at this time are due to the influence of the superposition of the weights corresponding to 2 adjacent output points, resulting in the number of channels becoming twice the original. To exclude interference and keep the mac calculation unchanged before and after sorting, all P weights in this range are filled with 0. When oc <= j < R and 0 <= i < P, the same processing procedure as when 0 <= j < oc and p <= i < Q is followed, and all P weights are filled with 0. When oc <= j < R and P <= i < Q, the P weight data at this time are correspondingly obtained by rearranging the KH * KW * IC weights before sorting of the output point ow1. The corresponding rule of the rearrangement subscript is i = x * KW * IC + y * IC + z + P, indicating that the weight of the z-th input channel in the x-th row and y-th column of the (j - oc)-th output channel before sorting corresponds to the weight of the i-th input channel in the 0-th row and 0-th column of the j-th output channel after rearrangement; S2. Perform MAC calculation on the rearranged input points. Among the 32 output channels of the 0-th output point obtained by the 0-th MAC calculation of the current output row, the first 16 output channels are the 0-th point of the current output feature row, and the last 16 output channels correspond to the 1-st point of the current output feature row; Among the 32 output channels of the 0-th output point obtained by the 1-st MAC calculation of the current output row, the first 16 output channels are the 2-nd point of the current output feature row, and the last 16 output channels correspond to the 3-rd point of the current output feature row; And so on. Similarly, among the 32 output channels of the 0-th output point obtained by the X-th MAC calculation of the current output row, the first 16 output channels are the 2 * X-th point of the current output feature row, and the last 16 output channels correspond to the (2 * X + 1)-th point of the current output feature row; Wherein: X = ow / N, and N is the number of columns of output points fixedly calculated by each MAC.

2. According to claim 1, a method for improving the hardware utilization of the input layer output 16 channels is characterized in that: In the characteristics of the MAC operation of the NNA, If the number of channels IC of the input feature is less than J, each MAC still calculates according to the minimum input channel calculation unit J. Then, in this case, the extra J - IC channels belong to redundant calculations; Similarly, if the number of channels OC of the output feature is less than K, each MAC still calculates according to the minimum output channel calculation unit K. Then, in this case, among the K channels of the M * N output points calculated by each MAC, the last K - OC channels also belong to redundant calculations.

3. A method for improving the hardware utilization rate of the 16-channel output of the input layer according to claim 2, characterized in that The method combines the characteristics of the MAC of the NNA, and when the number of output channels is less than K, uses the redundant calculation of the MAC to improve the efficiency: Taking KH = 3, KW = 3, J = 32, M = 1, N = 1, K = 32 as example parameters, and combining with the input layer of the neural network for illustration: For the convolution of the input layer with a 3 * 3 convolution kernel, 8-bit input, 8-bit weights, and 8-bit output, then the effective data volume of the input FeatureMap corresponding to one convolution of the input layer is 3 * 3 * 3 = 27; Based on the MAC operation characteristics of the NNA, the 3 * 3 convolution of the input layer corresponding to the effective data of 3 channels is equivalently converted into a 1 * 1 convolution corresponding to the effective data of 27 channels.

4. The method for improving the hardware utilization of 16 channels of input layer output according to claim 1, characterized in that: The rearrangement: is to rearrange the KH * KW * IC input points corresponding to the current output point in the priority order of channels, width, and height into data with a shape of 1 * 1 * P; The point located at the xth row, yth column, and zth channel before reordering is located at the x*KW*IC+y*IC+zth channel at the 0th row, 0th column after reordering; where: 0≤x<KH,0≤y<KW,0≤z<IC Then, the 1*1*P data corresponding to the output points of the adjacent combinations are superimposed to obtain 1*1*Q data, where Q=2*P; Correspondingly, the KH*KW*IC weights corresponding to the KH*KW*IC input points corresponding to the current output point must also be reordered to ensure that the corresponding convolutions before and after the sorting are equivalent.

Images