Hardware-based PReLU activation convolution operator calculation method

Abstract

The invention provides a hardware-based PReLU activation convolution operator calculation method, which comprises the following steps: S1, offline calculation: calculating parameters S and B which can be calculated in advance at a PC (Personal Computer) end in advance; s2, a chip carried on the development board selects a correct parameter according to a symbol of an actual value in actual operation, a tensor is subjected to convolution, then batchnormal calculation and finally activation calculation, and the tensor is expressed as a local operator conv-gt; the method comprises the following steps: carrying out batchnormal-gt; original input and output, including internal calculation, of the three operators are calculation of floating-point numbers, and floating-point data are converted into low-bit integer data through quantization. According to the method, the characteristics of hardware calculation are combined, and expansion is carried out on the basis of an original operator, so that the PReLU activation function is supported.

Description

Technical Field

[0001] The present invention belongs to the technical field of neural network operator calculation, and in particular relates to a method for calculating a PReLU activated convolution operator based on hardware. Background Art

[0002] In the prior art, with the development of neural network technology, more and more activation functions are used in applications. Among them, PReLU and LeakyReLU activation functions are increasingly used due to their excellent characteristics. However, due to the problem of calculation process, they are different from general ReLU and other functions, especially when it comes to quantization, PReLU and LeakyReLU are not well supported.

[0003] Commonly used terms in the prior art include:

[0004] BatchNormal: Normalize the input batch data and map it to a normal distribution with a mean of 0 and a variance of 1. At the same time, because the input data is mapped around the origin, the activation function's expression ability will be poor, so scaling and translation are introduced. The calculation formula is as follows:

[0005]

[0006] Local operator: A local operator refers to an operator whose value at a certain point is only affected by the values ​​within a certain range around the point, but not by the values ​​at other positions. In layman's terms, it means that the value of a point depends only on the values ​​near the point. Common local operators include gradient operators, Laplace operators, Sobel operators, etc. In the field of image processing, mean filter and median filter are also commonly used local operators. Since local operators only consider local information, they have a fast calculation speed and can be applied to scenarios such as real-time processing. A neural network consists of many operators connected by input and output to form a directed acyclic graph. Generally, when quantizing, it is necessary to consider the fusion between operators. For example, convolution-batch normalization-activation can be fused as one operator. The local operator in this article refers to the local structure in the neural network, which is a subgraph in the neural network. Because this article discusses the PReLu activation of convolution, it can be understood that the local operator is equivalent to conv-bn-act. Summary of the invention

[0007] In order to solve the above problems, the purpose of this application is to combine the characteristics of hardware computing and expand the original operator to support the PReLU activation function.

[0008] Specifically, the present invention provides a method for calculating a PReLU activated convolution operator based on hardware, the method comprising:

[0009] S1, offline calculation, is to calculate the pre-calculated parameters S and B on the PC side, where:

[0010]

[0011]

[0012] S x is the quantization scale of the input,

[0013] S w is the quantization scale of the convolution weight,

[0014] S bn is the simplified parameter scale of batchnormal,

[0015] B bn is the simplified parameter bias of batchnormal,

[0016] max is the maximum value of the quantization threshold of the output tensor,

[0017] h, j, k, g are parameters that vary according to the minimum value of the input quantization threshold and the output quantization threshold;

[0018] W int is the quantized integer weight,

[0019] sif is the integer domain range corresponding to the quantization bit width of the input tensor,

[0020] S2, the chip on the development board selects the correct parameters according to the sign of the actual value in actual operation. The tensor is first convolved, then batchnormal is calculated, and finally activated, which is expressed as: local operator conv-->batchnormal-->act. The original input and output of these three operators, including the internal calculations, are floating-point calculations. The floating-point data is converted into low-bit integer data through quantization; further including:

[0021] S2.1, existing local operator conv-->batchnormal-->act, expression:

[0022]

[0023]

[0024] act=clip(bn,min,max)

[0025] y=act

[0026] Among them, x is the input of the convolution, w is the weight of the convolution, bias is the bias of the convolution; var, eps, γ, β are the parameters of bn (abbreviation of batchnormal);

[0027] The act is set to the PReLU operator (act is the abbreviation of the activation function), and the formula is as follows:

[0028]

[0029] S2.1.1 Fusion of conv and bn operations:

[0030]

[0031]

[0032] set up

[0033]

[0034] Here, according to the formula of conv and bn, the conv-bn calculation is simplified, where x is the input of conv and w is the weight of conv;

[0035] S2.2, substitute the integer Xint and Wint into the calculation formula of bn;

[0036] The quantization operator of the conv on the board receives the quantized integer values ​​of x and w and calculates the quantized integer output value so that the network can use the integer value for reasoning;

[0037] Among them, the relationship between the floating point x and the integer Xint, and the relationship between the floating point w and Wint are as follows;

[0038] S2.2.1, after quantizing x and w, the floating point value and the integer value have the following relationship:

[0039] When input_clip_min=0

[0040] When input_clip_min<0

[0041] x=s x *(j*x int -k*sif)

[0042]

[0043]

[0044] There is int =(2*w int -sw)

[0045]

[0046] sif=2 n -1, sw=2 n -1;

[0047] j and k take different values ​​depending on the sign of input_clip_min;

[0048] For weight, w int =(2*w int -sw) is because the board weight range is [-(2 n -1), (2 n -1)];

[0049] S2.2.2, substitute x and w into the calculation formula of bn:

[0050]

[0051] set up

[0052]

[0053] At this point, the result of bn output is expressed using fixed-point data, and the activation function needs to determine the positive or negative value of bn;

[0054] S2.3, the board will calculate The value of , where the calculation symbol represents the convolution operation, so B and calibration need to be added to get the correct sign;

[0055] Since PReLU activation only changes the negative half-axis data of the input, if the input value is greater than or equal to 0, it is directly output, and if it is less than 0, it is multiplied by a parameter;

[0056] The above calculation calculates the output bn of conv-bn, that is, the activated input, and needs to judge its sign to determine the calculation method;

[0057] S2.3.1, now only consider the case of negative numbers:

[0058] The act refers to PReLU, the input is the bn above, and the output is named prelu;

[0059] prelu=act(bn)

[0060]

[0061] Substitute the previous expression of bn into:

[0062]

[0063] Set up new

[0064]

[0065] in,

[0066]

[0067]

[0068] When input_clip_min<0, j=2k=1

[0069] When input_clip_min=0, j=1k=0

[0070] When output_clip_min<0, h=0.5g=0.5

[0071] When output_clip_min<0, h=1g=0

[0072] The S and B calculated above are both negative before activation, denoted as S neg and B neg ;

[0073] S2.3.2, if the activation input is positive then S neg and B neg The alpha value in is 1, denoted as S pos and B pos ;

[0074] S2.4, the final calculation logic on the chip is:

[0075]

[0076]

[0077] In the step S1, the offline operation refers to an operation environment other than a development board.

[0078] In step S2, the chip controls program logic and calculations, and the program runs in the environment of the development board. The operators running on the development board need to minimize the calculation steps.

[0079] The step S2 is to select the correct parameters, that is, to derive the parameters required on the development board to perform the calculation process of the operator. The chip runs the final derived formula, that is, the chip runs the quantized operator.

[0080] In the step S2.3, since the output needs to be an integer value, prelu also needs to be quantized according to the quantization formula of the feature map. When only the negative half axis is considered, the activated calculation logic is to multiply the input by a parameter Alpha.

[0081] Therefore, the advantage of the present application is that it provides a method for PReLU activation convolution calculation to speed up the operation of the neural network on the chip. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0083] Figure 1 It is a code diagram of quantization features.

[0084] Figure 2 This is a code diagram for quantizing weights.

[0085] Figure 3 It is a flow chart of the present application method. DETAILED DESCRIPTION

[0086] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0087] The embodiment of the present application relates to a method for calculating a PReLU activated convolution operator based on hardware. First, quantization is introduced because quantization is the most basic operation. Quantization features QuantizeFeature and quantization weights QuantizeWeight are the quantization principles of tensors, and the quantized operators are run on the chip. Quantization refers to the use of low-bit width integers to represent floating-point data, with the purpose of reducing bandwidth and accelerating operations.

[0088] 1. QuantizeFeature, such as Figure 1 As shown:

[0089] If min<0,

[0090] If x is between [min,max], x=clip(x,min,max);

[0091] If x is between [-1,1], x=x / max;

[0092] If x is between [0,1], x=0.5x+0.5;

[0093] x is between [0,1], x=[x*(2 n -1)] / (2 n -1), all exponential operations below comply with the general arithmetic operator priority;

[0094] If x is between [-1,1], x=2x–1;

[0095] When x == 0;

[0096] If x is between [min,max], x=clip(x,min,max);

[0097] If x is between [0,1], x=x / max;

[0098] x is between [0,1], x=[x*(2 n -1)] / (2 n -1);

[0099] Among them, min and max are the value ranges of the feature map, which are obtained by network training and can also be directly specified according to the activation function or data distribution characteristics (relu6 can be specified as min=0, max=6). n is the quantized bit width; x is the feature map; the clip function represents limiting the value of the first input to between the next two values, and [] in the formula represents rounding operation.

[0100] 2. Quantize weight QuantizeWeight, such as Figure 2 As shown:

[0101] w is between [-max1, max1], max1 = perchannel_absmax(w);

[0102] w is a normal distribution, w=clip(w,mean-3std,mean+3std);

[0103] Let max=perchannel_absmax(w);

[0104] w is between [-1,1], w=w / max;

[0105] w is between [0,1], w=0.5w+0.5;

[0106] w is between [0,1], w=[w*(2 n -1)] / (2 n -1);

[0107] w is between [-1,1], w=2w–1;

[0108] w is between [-max1, max1], which is expressed as: w = w*max1; here, after weight quantization, it is necessary to return to the original threshold range to ensure that the difference between input and output in this operation is minimized to reduce quantization loss.

[0109] Among them, perchannel_absmax is the maximum value of the absolute value on each channel of the weight calculated by channel, and the channel means that each output channel is a channel; mean and std are also calculated by channel, including the rest of the calculations, which are understood as broadcast calculations. mean, std are the mean and standard deviation of the weight. The above formula uses the data properties of the normal distribution and intercepts 99.73% of the data. Let the weight dimension be [32,16,3,3], and the arrangement order is oc, ic, kh, kw. Channel division means dividing the weight into 32 tensors according to the oc dimension, and calculating each tensor separately. Then you will get 32 ​​means, 32 standard deviations, 32 maximum values, etc. 'The rest of the calculations are divided into channels' means that the remaining addition, subtraction, multiplication and division calculations are also aligned according to the oc dimension, and each tensor is calculated separately.

[0110] Specifically, if Figure 3 As shown, the method relates to a method for calculating a PReLU activated convolution operator based on hardware, and the method includes:

[0111] S1, offline calculation, is to calculate the pre-calculated parameters S and B on the PC side in advance, and the offline refers to the operating environment of the non-development board;

[0112]

[0113]

[0114] S x is the input quantization scale, see step S2.2.1;

[0115] S w is the quantization scale of the convolution weight, see step S2.2.1;

[0116] S bn is the simplified parameter scale of batchnormal, see step S2.1.1;

[0117] B bn is the parameter bias after batchnormal simplification, see step S2.1.1;

[0118] max is the maximum value of the quantization threshold of the output tensor,

[0119] h, j, k, g are parameters that vary according to the minimum value of the input quantization threshold and the output quantization threshold, see step S2.3.1; W int is the quantized integer weight,

[0120] sif is the integer domain range corresponding to the quantization bit width of the input tensor, such as 8 bits, and its value is 2 to the 8th power minus 1;

[0121] S2, the chip mounted on the development board selects the correct parameters according to the sign of the actual value in actual operation. The tensor is first convolved, then batchnormal is calculated, and finally activated, which is expressed as: local operator conv-->batchnormal-->act. The original input and output of these three operators, including the internal calculations, are floating-point calculations. The floating-point data is converted into low-bit integer data through quantization. The chip controls the program logic and calculations. The program runs in the development board environment. The operators running on the development board need to minimize the calculation steps.

[0122] The selection of correct parameters is to derive the parameters required on the development board for the calculation process of the operator, and the chip runs the final derived formula, that is, the chip runs the quantized operator;

[0123] Further including:

[0124] S2.1, existing local operator conv-->batchnormal-->act, expression:

[0125]

[0126]

[0127] act = clip(bn, min, max)

[0128] y=act

[0129] Among them, x is the input of the convolution, w is the weight of the convolution, bias is the bias of the convolution; var, eps, γ, β are the parameters of bn;

[0130] The act is now set to the PReLU operator, and the formula is as follows:

[0131]

[0132] Fusion of conv and bn operations:

[0133]

[0134]

[0135] set up

[0136]

[0137] Here, according to the formula of conv and bn, the conv-bn calculation is simplified, where x is the input of conv and w is the weight of conv;

[0138] S2.2, substitute the integer Xint and Wint into the calculation formula of bn;

[0139] The quantization operator of the conv on the board receives the quantized integer values ​​of x and w and calculates the quantized integer output value so that the network can use the integer value for reasoning;

[0140] Among them, the relationship between the floating point x and the integer Xint, and the relationship between the floating point w and Wint are as follows;

[0141] S2.2.1, after quantizing x and w, the floating point value and the integer value have the following relationship:

[0142]

[0143]

[0144] x=s x *(j*x int -k*sif)

[0145]

[0146]

[0147] There is int =(2*w int -sw)

[0148]

[0149] sif=2 n -1, sw=2 n -1;

[0150] j and k take different values ​​depending on the sign of input_clip_min;

[0151] For weight, w int =(2*w int -sw) is because the board weight range is [-(2 n -1), (2n -1)];

[0152] S2.2.2, substitute x and w into the calculation formula of bn:

[0153]

[0154] set up

[0155]

[0156] At this point, the result of bn output is expressed using fixed-point data, and the activation function needs to determine the positive or negative value of bn;

[0157] S2.3, the board will calculate The value of , where the calculation symbol represents the convolution operation, so B and calibration need to be added to get the correct sign;

[0158] Since PReLU activation only changes the negative half-axis data of the input, if the input value is greater than or equal to 0, it is directly output, and if it is less than 0, it is multiplied by a parameter;

[0159] The above calculation calculates the output bn of conv-bn, that is, the activated input, and needs to judge its sign to determine the calculation method;

[0160] Since the output needs to be an integer value, prelu must also be quantized according to the quantization formula of the feature map. When only the negative half axis is considered, the activated calculation logic is to multiply the input by a parameter Alpha;

[0161] Further including:

[0162] S2.3.1, now only consider the case of negative numbers:

[0163] The act refers to PReLU, the input is the bn above, and the output is named prelu;

[0164] prelu=act(bn)

[0165]

[0166] Substitute the previous expression of bn into:

[0167]

[0168] Set up new

[0169]

[0170] in:

[0171]

[0172]

[0173] When input_clip_min<0, j=2 k=1

[0174] When input_clip_min=0, j=1 k=0

[0175] When output_clip_min<0, h=0.5g=0.5

[0176] When output_clip_min=0, h=1g=0

[0177] The S and B calculated above are both negative before activation, denoted as S neg and B neg ;

[0178] S2.3.2, if the activation input is positive then S neg and B neg The alpha value in is 1, denoted as S pos and B pos ;

[0179] S2.4, the final calculation logic on the chip is:

[0180]

[0181]

[0182] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for calculating a PReLU activated convolution operator based on hardware, characterized in that: The method comprises: S1, offline calculation, is to calculate the pre-calculated parameters S and B on the PC side, where: S x is the quantization scale of the input, S w is the quantization scale of the convolution weight, S bn is the simplified parameter scale of batchnormal, B bn is the simplified parameter bias of batchnormal, max is the maximum value of the quantization threshold of the output tensor, h, j, k, g are parameters that vary according to the minimum value of the input quantization threshold and the output quantization threshold; W int is the quantized integer weight, sif is the integer domain range corresponding to the quantization bit width of the input tensor, S2, the chip on the development board selects the correct parameters according to the sign of the actual value in actual operation. The tensor is first convolved, then batchnormal is calculated, and finally activated, which is expressed as: local operator conv-->batchnormal-->act. The original input and output of these three operators, including the internal calculations, are floating-point calculations. The floating-point data is converted into low-bit integer data through quantization; further including: S2.1, existing local operator conv-->batchnormal-->act, expression: act=Clip(bn,mini,max) y=act Among them, x is the input of the convolution, w is the weight of the convolution, bias is the bias of the convolution; var, eps, γ, β are the parameters of bn; The act is now set to the PReLU operator, and the formula is as follows: S2.1.1 Fusion of conv and bn operations: Here, according to the formula of conv and bn, the conv-bn calculation is simplified, where x is the input of conv and w is the weight of conv; S2.2, substitute the integer Xint and Wint into the calculation formula of bn; The quantization operator of the conv on the board receives the quantized integer values ​​of x and w and calculates the quantized integer output value so that the network can use the integer value for reasoning; Among them, the relationship between the floating point x and the integer Xint, and the relationship between the floating point w and Wint are as follows; S2.2.1, after quantizing x and w, the floating point value and the integer value have the following relationship: When input_clip_min=0 When input_clip_min<0 x=s x *(j*x int -k*sif) There is w int = (2 * w int - sw) sif=2 n -1,sw=2 n -1; j and k take different values ​​depending on the sign of input_clip_min; For weight, w int =(2*w int -sw) is because the board weight range is [-(2 n -1), (2 n -1)]; S2.2.2, substitute x and w into the calculation formula of bn: set up At this point, the result of bn output is expressed using fixed-point data, and the activation function needs to determine the positive or negative value of bn; S2.3, the board will calculate The value of , where the calculation symbol represents the convolution operation, so B and calibration need to be added to get the correct sign; Since PReLU activation only changes the negative half-axis data of the input, if the input value is greater than or equal to 0, it is directly output, and if it is less than 0, it is multiplied by a parameter; The above calculation calculates the output bn of conv-bn, that is, the activated input, and needs to judge its sign to determine the calculation method; S2.3.1, now only consider the case of negative numbers: The act refers to PReLU, the input is the bn above, and the output is named prelu; prelu=act(bn) Substitute the previous expression of bn into: Set up new in, When input_clip_min<0, j=2 k=1 When input_clip_min=0, j=1 k=0 When output_clip_min<0, h=0.5 g=0.5 When output_clip_min<0, h=1 g=0 The S and B calculated above are both negative before activation, denoted as S neg and B neg ; S2.3.2, if the activation input is positive then S neg and B neg The alpha value in is 1, denoted as S pos and B pos ; S2.4, the final calculation logic on the chip is:

2. A method for calculating a PReLU activated convolution operator based on hardware according to claim 1, characterized in that: In the step S1, the offline operation refers to an operation environment other than a development board.

3. The method for calculating a PReLU activated convolution operator based on hardware according to claim 1, characterized in that: In step S2, the chip controls program logic and calculations, and the program runs in the environment of the development board. The operators running on the development board need to minimize the calculation steps.

4. The method for calculating a PReLU activated convolution operator based on hardware according to claim 1, characterized in that: The step S2 is to select the correct parameters, that is, to derive the parameters required on the development board to perform the calculation process of the operator. The chip runs the final derived formula, that is, the chip runs the quantized operator.

5. The method for calculating a PReLU activated convolution operator based on hardware according to claim 1, characterized in that: In the step S2.3, since the output needs to be an integer value, prelu also needs to be quantized according to the quantization formula of the feature map. When only the negative half axis is considered, the activated calculation logic is to multiply the input by a parameter Alpha.

Images