Implementation method of high-performance inference circuit of general artificial intelligence nonlinear activation function

Abstract

The invention discloses a high-performance inference circuit implementation method for a general artificial intelligence nonlinear activation function, and the method comprises the steps: carrying out the iteration of a segmentation threshold thds of the nonlinear activation function in a given range and a quadratic polynomial coefficient of a corresponding segment through employing a self-adaptive segmentation quadratic polynomial fitting method; the coefficient screening circuit is constructed to gate the quadratic polynomial coefficient of the segment where the input is located according to a threshold value and an input numerical value, and the quadratic polynomial reasoning part circuit receives the input and the quadratic polynomial coefficient of the segment where the input is located and outputs a high-precision quantitative calculation result. Test on a standard test set ImageNet shows that compared with an existing design, the method has higher reasoning precision and performance, and high-precision fitting and full-quantization calculation of most nonlinear activation functions in existing deep learning models such as CNN and Transformer can be supported in a full interval.

Description

Technical Field

[0001] The present invention relates to the field of electronic information technology, and in particular to a high-performance reasoning circuit implementation method for a general artificial intelligence nonlinear activation function. Background Art

[0002] Deep learning algorithm models based on CNN and Transformer, such as the YOLO series and Llama series, have achieved the most advanced results in tasks such as machine vision and natural language processing. However, due to the limitations of memory usage, inference latency, and power consumption, the inference effect and real-time performance of this series of algorithm models are insufficient when deployed and applied on the end. The quantization method can represent weights and activation values ​​with lower precision, significantly reducing the model size of the neural network and effectively reducing memory usage. At the same time, integer operations have advantages over floating-point operations in reducing latency and hardware power consumption. However, quantization will significantly affect the calculation accuracy of nonlinear activation functions, causing the attenuation of the inference accuracy of deep learning models and the decrease in convergence speed during training. Therefore, in the end-side inference process, the hardware implementation of nonlinear activation functions must not only meet the restrictions of area and power consumption, but also achieve high computing efficiency and maintain extremely high accuracy.

[0003] Therefore, exploring the design of high-performance fully quantized inference circuits for nonlinear activation functions in algorithm models such as CNN and Transformer is the key to achieving efficient deployment of deep learning algorithm models on resource-constrained edge devices. At present, most of the quantization methods used in nonlinear activation function inference circuits belong to analog quantization and partial quantization. Floating-point operations are not completely eliminated during model inference. Nonlinear activation functions such as gelu (Gaussian error linear unit), swish, mish, softplus, etc. must call floating-point operation units, which seriously limits the application of deep learning algorithm models on edge devices. The I-ViT algorithm adopts the GELU function approximate calculation formula based on the Sigmoid function and proposes the ShiftGELU method. ShiftGELU uses shifting instead of multiplication, and calls the natural exponential operation and division operation modules to realize GELU high-precision pure integer operations. Although this method can ensure high calculation accuracy, the natural exponential operation and division operation will consume a large number of clock cycles, and the processing delay and calculation time are large. The MobilenetV3 algorithm proposes to use the h-Sigmoid method to approximate Sigmoid with pure integers, which avoids natural exponential operations and division operations, significantly reducing the computational complexity, but with poor accuracy. Test results show that this method will cause a significant decrease in the accuracy of the algorithm's reasoning results. In addition, there are also approximate fitting methods such as high-order Pade and Maclaurin fitting formulas that can be used to fit the exponential function with high accuracy, reducing the absolute value error of the nonlinear activation function to 2×10 -7, but multiple floating-point operations are still required during the inference process. The above methods and designs can only support inference of a single or several nonlinear activation functions, and none of them can guarantee high inference accuracy and low computational complexity at the same time. Summary of the invention

[0004] The purpose of the present invention is to provide a high-performance reasoning circuit implementation method for a general artificial intelligence nonlinear activation function to solve the problems in the background technology.

[0005] In order to solve the above technical problems, the present invention provides a high-performance reasoning circuit implementation method of a general artificial intelligence nonlinear activation function, comprising:

[0006] Step A, using an adaptive piecewise quadratic polynomial fitting method to give a piecewise threshold thds of a nonlinear activation function within a given range and a corresponding piecewise floating-point quadratic polynomial coefficient, and quantifying the accuracy of the threshold and the coefficient;

[0007] Step B: Input the quantized independent variable Q in , Q in The quantization coefficient k in , all segment thresholds thds and all quadratic polynomial coefficients to the coefficient screening circuit, output Q in The quadratic polynomial coefficients A2, A1 and A0 of the segment;

[0008] Step C: quantify the independent variable Q in , Q in The quantization coefficient k in , output value quantization coefficient k ou and Q in The quadratic polynomial coefficients A2, A1, A0 of the segment are input to the quadratic polynomial reasoning circuit, and the high-precision quantized calculation result Q is output. ou .

[0009] In one embodiment, the adaptive piecewise quadratic polynomial fitting method in step A gives the piecewise threshold thds of the nonlinear activation function within a given range and the corresponding piecewise floating-point quadratic polynomial coefficients, and the process is as follows:

[0010] (1) Initialize N at equal intervals within the given range [C1, C2] 0 The segmentation threshold thds is {thd 0 ,thd 1 ,…,thd N0-1}, get N 0 - 1 segment;

[0011] (2) Sampling is performed in each segment to obtain a set of sampling points X and the corresponding activation function value Y; the sampling of the i-th segment curve is (Xi ,Y i ), the corresponding quadratic fitting formula is:

[0012]

[0013] (3) Considering the continuity condition of fitting at the threshold point,

[0014] L i (thd i+1 )=L i+1 (thd i+1 )

[0015] Where thd i+1 is the i-th segment of curve L i and the i+1th curve L i+1 Connect the threshold nodes; substitute the continuity condition of the fitting into the quadratic fitting formula,

[0016]

[0017] Substitute the interpolation condition into the quadratic fitting formula,

[0018] L i (thd i )=Act(thd i )

[0019] L i+1 (thd i+1 )=Act(thd i+1 )

[0020] Where thd i is the i-1th segment of curve L i-1 and the i-th curve L i The threshold node of the connection; Act is the activation function; the above formula is expanded into:

[0021]

[0022] X i and Y i are column vectors, and the fitting formula is constructed using the least squares method.

[0023]

[0024] E is the fitting loss; ||||2 is the 2-norm; considering the continuity condition, interpolation condition and least square fitting formula, the calculation formula for the optimal fitting coefficients a2, a1, and a0 of the quadratic polynomial is obtained as follows:

[0025]

[0026] thd i The activation function value at x n is the sampling point, y n is the activation function value at the sampling point; after quantizing a2, a1, and a0, the quantized quadratic polynomial coefficients A2, A1, and A0 are obtained; at the same time, the goodness of fit R is calculated 2 ,as follows:

[0027]

[0028] In the formula, is the residual sum of squares, Y i is a 1-dimensional vector consisting of the activation function values ​​at the sampling points, is the fitted value; is the sum of squares of deviations, where Y i The mean of the goodness of fit R 2 The larger the value, the better the fitting effect;

[0029] (4) Check the goodness of fit (R 2 ) i and (R 2 ) i+1 ; If (R 2 ) i Greater than (R 2 ) i+1 , indicating that the fitting effect of curve segment i is better than that of curve segment i+1. Move the middle threshold node to the side of curve segment i+1, and the moving step is as follows:

[0030] step=min{0.001,α(thd i+1 -thd i )}

[0031] Where α is a constant between 0 and 1; the step size does not exceed 0.001;

[0032] If (R 2 ) i Less than (R 2 ) i+1 , move the middle threshold node to the side of curve segment i; (R 2 ) i and (R 2 ) i+1 If equal, do not move

[0033] (5) Repeat steps (2) to (4) until R 2 The change is less than the threshold thd_mv;

[0034] (6) Check the goodness of fit of two adjacent quadratic curves (R2 ) i and (R 2 ) i+1 ; If (R 2 ) i and (R 2 ) i+1 are greater than the "goodness of fit deletion threshold thd_del_R 2 ", indicating that the two-segment curve fitting effect is excellent, delete the threshold node thd between the two segments i +1 ; If (R 2 ) i Less than the "goodness of fit increase threshold thd_add_R 2 ", indicating that the curve fitting effect of this section is not good, and a threshold node is added at the midpoint; (R 2 ) i+1 The same operation is also used; in actual operation, thd_add_R 2 Greater than thd_del_R 2 ;

[0035] (7) Repeat steps (2) to (6) until the number and position of the threshold nodes become stable.

[0036] In one implementation, the coefficient screening circuit implementation process in step B is as follows:

[0037] (1) Input the quantized independent variable Q in , Q in The quantization coefficient k in , all segment thresholds thds and all quadratic polynomial coefficients;

[0038] (2) According to Q in The value and segmentation threshold thds are used to determine Q in The curve segment number; the selection process is as follows:

[0039]

[0040] In the formula, the threshold quantization coefficient k a is a constant integer; if Q in If the above formula is satisfied, then Q in Located in the i-th fitting interval;

[0041] (3) Output the quadratic polynomial coefficients A2, A1 and A0 of the corresponding curve segment.

[0042] In one implementation, the circuit implementation process of the quadratic polynomial reasoning part in step C is as follows:

[0043] (1) Input the quantized independent variable Q in , Qin The quantization coefficient k in , output value quantization coefficient k ou and Q in The coefficients of the quadratic polynomial of the segment are A2, A1, A0;

[0044] (2) Calculate the high-precision quantization result Q of the nonlinear activation function according to the quadratic polynomial inference formula ou :

[0045]

[0046] In the formula, Q ou and Q in are the dynamic precision quantization results of the floating point output o and input x respectively; A2, A1 and A0 are the quantization results of the quadratic polynomial fitting coefficients a2, a1 and a0 respectively, and their quantization coefficients k a is a constant integer; the multiplication of powers of 2 in the quadratic polynomial reasoning circuits are all replaced by shift operations to improve the calculation efficiency.

[0047] The high-performance reasoning circuit implementation method of a general artificial intelligence nonlinear activation function provided by the present invention has the following advantages:

[0048] (1) It can support high-precision fitting and full quantization calculation of most nonlinear activation functions in existing deep learning models such as CNN and Transformer in the entire range;

[0049] (2) The calculation accuracy of the activation function by the inference circuit is better than the current mainstream method, and it takes up less resources and has low computational complexity;

[0050] (3) The inference circuit is decoupled from the algorithm model, and the delay is constant and is independent of the type of nonlinear activation function fitted;

[0051] (4) The inference circuit supports inter-layer multiplexing. According to the inference accuracy requirements of various algorithm models for nonlinear activation functions, the inference circuit only needs to reconfigure the pre-calculated quadratic polynomial coefficients and the thresholds of each interval segment. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is an overall architecture diagram of a high-performance reasoning circuit of a general artificial intelligence nonlinear activation function proposed by the present invention;

[0053] Figure 2 It is the schematic diagram of the quadratic polynomial reasoning part of the reasoning circuit;

[0054] Figure 3 It is the timing diagram of the quadratic polynomial reasoning part of the circuit;

[0055] Figure 4It is a diagram of the inference effect of the present invention on gelu, mish, softplus, swish, tanh, erf, and sigmoid functions.

[0056] Figure 5 It is a waveform diagram of the present invention's reasoning on the erf function. DETAILED DESCRIPTION

[0057] The following is a further detailed description of a high-performance reasoning circuit implementation method for a general artificial intelligence nonlinear activation function proposed by the present invention in combination with the accompanying drawings and specific embodiments. According to the following description, the advantages and features of the present invention will become clearer. It should be noted that the accompanying drawings are all in a very simplified form and are not in precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0058] The present invention adopts an adaptive piecewise quadratic polynomial fitting method to iteratively fit the nonlinear activation function within a given range, and finally gives all the segmented thresholds thds and the corresponding segmented floating-point quadratic polynomial coefficients, and quantizes the thresholds and coefficients in advance to configure the coefficient screening circuit. in , Q in The quantization coefficient k in , all segment thresholds thds and all quadratic polynomial coefficients, input coefficient screening circuit, get Q in The quadratic polynomial coefficients A2, A1 and A0 of the segment. Input Q in , k in , k ou , A2, A1, A0 to the quadratic polynomial reasoning circuit, outputting the high-precision quantization result Q of the nonlinear activation function ou , where k ou is the output quantization coefficient.

[0059] The specific implementation steps of the adaptive piecewise quadratic polynomial fitting method to give the piecewise threshold thds of the nonlinear activation function within a given range and the corresponding piecewise floating-point quadratic polynomial coefficients are as follows:

[0060] Step 1: Initialize N at equal intervals within the given range [C1, C2]. 0 The segmentation threshold thds is {thd 0 ,thd 1 ,…,thd N0-1}, get N 0 -1 segment.

[0061] Step 2: Sample the nonlinear activation function in each segment to obtain the sampling point X and the corresponding activation function value Y, and calculate the optimal fitting coefficients a2, a1, and a0 of the quadratic polynomial of each segment according to the following formula:

[0062]

[0063] thd i The activation function value at x n is the sampling point, y n is the activation function value at the sampling point; at the same time, the goodness of fit R is calculated 2 ,as follows:

[0064]

[0065] is the residual sum of squares, Y i is a 1-dimensional vector consisting of the activation function values ​​at the sampling points, is the fitted value; is the sum of squares of deviations, where Y i The mean of

[0066] Step 3: Check the goodness of fit (R 2 ) i and (R 2 ) i+1 , if (R 2 ) i Greater than (R 2 ) i+1 , it means that the fitting effect of curve segment i is better than that of curve segment i+1. Move the middle threshold node to the side of curve segment i+1. The moving step is as follows:

[0067] step=min{0.001,α(thd i+1 -thd i )}

[0068] Where α is a constant between 0 and 1; the step size does not exceed 0.001.

[0069] On the contrary, if (R 2 ) i Less than (R 2 ) i+1 , then move the middle threshold node to the side of curve segment i; (R 2 ) i and (R 2 ) i+1 If they are equal, do not move.

[0070] Step 4: Repeat steps 2 to 3 until the goodness of fit R 2 The change is less than the threshold thd_mv.

[0071] Step 5: Check the goodness of fit (R 2 ) i and (R 2 ) i+1 If (R 2 ) i and (R 2 ) i+1 are greater than the "goodness of fit deletion threshold thd_del_R 2 ", it means that the two curves have excellent fitting effect, and the threshold node thd between the two segments is deleted. i+1 ; If (R 2 ) i Less than the "goodness of fit increase threshold thd_add_R 2 ", it means that the curve fitting effect of this section is not good, and a threshold node is added at the midpoint; (R 2 ) i+1 The same operation is also used. In actual operation, thd_add_R 2 Greater than thd_del_R 2 .

[0072] Step 6: Repeat steps 2 to 5 until the number and position of the threshold nodes become stable.

[0073] The overall architecture of the inference circuit is as follows Figure 1 As shown, the threshold thds and the corresponding segmented floating-point quadratic polynomial coefficients pre-generated in the above steps are used to perform precision quantization to configure the coefficient screening circuit, and the steps of implementing the inference are as follows:

[0074] Step 1: Input the quantized independent variable Q in , Q in The quantization coefficient k in , all segment thresholds thds and all quadratic polynomial coefficients to the coefficient screening circuit, output Q in The quadratic polynomial coefficients A2, A1 and A0 of the segment;

[0075] Step 2: Quantify the independent variable Q in , Q in The quantization coefficient k in , output value quantization coefficient k ou and Q in The quadratic polynomial coefficients A2, A1, A0 of the segment are input to Figure 2 The quadratic polynomial reasoning circuit shown in the figure outputs a high-precision quantization result Q according to the following formula ou ,

[0076]

[0077] In the formula, Q ouand Q in are the dynamic precision quantization results of floating-point output o and input x, respectively, a is the polynomial coefficient quantization coefficient. The specific reasoning sequence is as follows Figure 3 shown.

[0078] Example 1: Demonstrate the erf function circuit reasoning calculation effect of the ViT (Vision Transformer) algorithm on the ImageNet dataset. Figure 4 It is a diagram of the inference effect of the present invention on gelu, mish, softplus, swish, tanh, erf, and sigmoid functions, showing the fitting effect of different activation functions; wherein the solid line represents the original function and the dotted line represents the fitting function.

[0079] The relevant parameters for implementing the high-performance reasoning circuit design of the erf function are set as follows: the quantization bit width is 16 bits, the goodness of fit deletion threshold thd_del_R 2 " is set to 0.990, and the goodness of fit threshold thd_add_R 2 " is set to 0.995, and the polynomial coefficient quantization coefficient k a is 15. The step size α is set to 0.05 and R 2 The change threshold thd_mv is set to 5%.

[0080] 1. This embodiment uses the ImageNet dataset for testing. The test image is input into the original ViT algorithm, and the recognition accuracy rate P = 84.53%.

[0081] 2. Since the erf function is an odd function, only the interval greater than 0 is fitted. In the interval [-3.0, 3.0], 3 segmentation thresholds thds are initialized to {0, 1.5, 3.0} at equal intervals, and 2 segments are obtained.

[0082] 3. Sample the nonlinear activation function erf to obtain the sampling point X and the corresponding activation function value Y. Call the adaptive piecewise quadratic polynomial fitting method to calculate the segmentation threshold thds and the optimal fitting coefficient of each segment quadratic polynomial, and use k a Quantization is performed as a quantization coefficient.

[0083] 4. Inject the test image into the ViT algorithm and input Q after quantization of the erf function in and its quantization coefficient k in , all segment thresholds thds and all quadratic polynomial coefficients to the coefficient screening circuit, output Q in The quadratic polynomial coefficients A2, A1 and A0 of the segment;

[0084] 5. Input the quantized independent variable Q in, Q in The quantization coefficient k in , output value quantization coefficient k ou and Q in The quadratic polynomial coefficients A2, A1, A0 of the segment are Figure 2 The quadratic polynomial reasoning circuit shown in the figure outputs high-precision quantized calculation results Q ou ;

[0085] 6. Dequantize the quantization result to get a floating point value, which is used for the calculation of the subsequent layers of the ViT algorithm. The recognition accuracy of the ViT algorithm is recorded as P our =84.47%.

[0086] 7. Close the present invention, use the quadratic polynomial coefficients of the int16 full quantization method of the erf function in I-BERT, repeat steps 5 and 6, and record the recognition accuracy of the ViT algorithm as P ref =84.20%.

[0087] 8. Compare P and P our , P ref The comparison results show that P and P our The recognition results are consistent, and the recognition accuracy rate drops by only 0.06%. ref The recognition accuracy dropped by 0.33%. Figure 5 The waveform diagram of the output result obtained by injecting test data into the circuit shows that the delay of the erf function inference implemented by the present invention is only 2clk. The logical resource consumption of VIVADO 2019.2 evaluation is as follows: (1) DSP 4; (2) LUT 1721; (3) FF 80.

[0088] In summary, the present invention achieves low-latency and high-performance reasoning of nonlinear activation functions with less logic resources, and the reasoning accuracy is greatly improved compared with the existing methods.

[0089] Parts of the present invention that are not disclosed in detail belong to the common knowledge in the art.

[0090] The above description is only a description of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention. Any changes or modifications made by a person skilled in the art in the field of the present invention based on the above disclosure shall fall within the scope of protection of the claims.

Claims

1. A high-performance reasoning circuit implementation method for a general artificial intelligence nonlinear activation function, characterized in that: include: Step A, using an adaptive piecewise quadratic polynomial fitting method to give a piecewise threshold thds of a nonlinear activation function within a given range and a corresponding piecewise floating-point quadratic polynomial coefficient, and quantifying the accuracy of the threshold and the coefficient; Step B: Input the quantized independent variable Q in , Q in The quantization coefficient k in , all segment thresholds thds and all quadratic polynomial coefficients to the coefficient screening circuit, output Q in The quadratic polynomial coefficients A2, A1 and A0 of the segment; Step C: quantify the independent variable Q in , Q in The quantization coefficient k in , output value quantization coefficient k ou and Q in The quadratic polynomial coefficients A2, A1, A0 of the segment are input to the quadratic polynomial reasoning circuit, and the high-precision quantized calculation result Q is output. ou .

2. The high-performance reasoning circuit implementation method of the general artificial intelligence nonlinear activation function according to claim 1, characterized in that: The adaptive piecewise quadratic polynomial fitting method in step A gives the piecewise threshold thds of the nonlinear activation function within a given range and the corresponding piecewise floating-point quadratic polynomial coefficients, and the process is as follows: (1) Initialize N at equal intervals within the given range [C1, C2] 0 The segmentation threshold thds is {thd 0 ,thd 1 ,…,thd N0-1 }, get N 0 - 1 segment; (2) Sampling is performed in each segment to obtain a set of sampling points X and the corresponding activation function value Y; the sampling of the i-th segment curve is (X i ,Y i ), the corresponding quadratic fitting formula is: (3) Considering the continuity condition of fitting at the threshold point, L i (thd i+1 )=L i+1 (thd i+1 ) Where thd i+1 is the i-th segment of curve L i and the i+1th curve L i+1 Connect the threshold nodes; substitute the continuity condition of the fitting into the quadratic fitting formula, Substitute the interpolation condition into the quadratic fitting formula, L i (thd i )=Act(thd i ) L i+1 (thd i+1 )=Act(thd i+1 ) Where thd i is the i-1th segment of curve L i-1 and the i-th curve L i The threshold node of the connection; Act is the activation function; the above formula is expanded into: X i and Y i are column vectors, and the fitting formula is constructed using the least squares method. E is the fitting loss; || ||2 is the 2-norm; considering the continuity condition, interpolation condition and least square fitting formula, the calculation formula for the optimal fitting coefficients a2, a1, and a0 of the quadratic polynomial is obtained as follows: y thdi thd i The activation function value at x n is the sampling point, y n is the activation function value at the sampling point; after quantizing a2, a1, and a0, the quantized quadratic polynomial coefficients A2, A1, and A0 are obtained; at the same time, the goodness of fit R is calculated 2 ,as follows: In the formula, is the residual sum of squares, Y i is a 1-dimensional vector consisting of the activation function values ​​at the sampling points, is the fitted value; is the sum of squares of deviations, where Y i The mean of the goodness of fit R 2 The larger the value, the better the fitting effect; (4) Check the goodness of fit (R 2 ) i and (R 2 ) i+1 ; If (R 2 ) i Greater than (R 2 ) i+1 , indicating that the fitting effect of curve segment i is better than that of curve segment i+1. Move the middle threshold node to the side of curve segment i+1, and the moving step is as follows: step=min{0.001,α(thd i+1 -thd i )} Where α is a constant between 0 and 1; the step size does not exceed 0.001; If (R 2 ) i Less than (R 2 ) i+1 , move the middle threshold node to the side of curve segment i; (R 2 ) i and (R 2 ) i+1 If equal, do not move (5) Repeat steps (2) to (4) until R 2 The change is less than the threshold thd_mv; (6) Check the goodness of fit of two adjacent quadratic curves (R 2 ) i and (R 2 ) i+1 ; If (R 2 ) i and (R 2 ) i+1 All greater than the "goodness of fit deletion threshold thd_del_R 2 ", indicating that the two-segment curve fitting effect is excellent, delete the threshold node thd between the two segments i+1 ; If (R 2 ) i Less than the "fitting goodness increase threshold thd_add_R 2 ", indicating that the curve fitting effect of this section is not good, and a threshold node is added at the midpoint; (R 2 ) i+1 The same operation is also used; in actual operation, thd_add_R 2 Greater than thd_del_R 2 ; (7) Repeat steps (2) to (6) until the number and position of the threshold nodes become stable.

3. The high-performance reasoning circuit implementation method of the general artificial intelligence nonlinear activation function as claimed in claim 2, characterized in that: The implementation process of the coefficient screening circuit in step B is as follows: (1) Input the quantized independent variable Q in , Q in The quantization coefficient k in , all segment thresholds thds and all quadratic polynomial coefficients; (2) According to Q in The value and segmentation threshold thds are used to determine Q in The curve segment number; the selection process is as follows: thd i ≤Q in 2 kα-kin <thd i In the formula, the threshold quantization coefficient k a is a constant integer; if Q in If the above formula is satisfied, then Q in Located in the i-th fitting interval; (3) Output the quadratic polynomial coefficients A2, A1 and A0 of the corresponding curve segment.

4. The high-performance reasoning circuit implementation method of the general artificial intelligence nonlinear activation function according to claim 1, characterized in that: The circuit implementation process of the quadratic polynomial reasoning part in step C is as follows: (1) Input the quantized independent variable Q in , Q in The quantization coefficient k in , output value quantization coefficient k ou and Q in The coefficients of the quadratic polynomial of the segment are A2, A1, A0; (2) Calculate the high-precision quantization result Q of the nonlinear activation function according to the quadratic polynomial inference formula ou : In the formula, Q ou and Q in are the dynamic precision quantization results of the floating point output o and input x respectively; A2, A1 and A0 are the quantization results of the quadratic polynomial fitting coefficients a2, a1 and a0 respectively, and their quantization coefficients k a is a constant integer; the multiplication of powers of 2 in the quadratic polynomial reasoning circuits are all replaced by shift operations to improve the calculation efficiency.

Images