An adaptive quantization method for neural network accelerators running on FPGA

Abstract

The present invention discloses an adaptive quantization method adapted for a neural network accelerator running on an FPGA, belonging to the field of neural networks. The present invention automatically predicts the degree of overflow during the neural network accelerator's calculations based on the actual bit width of the neural network accelerator's calculation process, and adaptively adjusts quantization parameters based on the degree of overflow, thereby avoiding data overflow problems during the calculation of the neural network algorithm on the FPGA, thereby ensuring the correctness of the neural network model results. The adaptive quantization method of the present invention combines quantization operations with resource planning of the neural network accelerator hardware, ensuring the correctness of the results when the neural network accelerator deploys the algorithm. Without sacrificing model accuracy and execution efficiency, it can effectively compress the model size, making it easy to deploy and implement under resource-constrained conditions, saving storage space and computing resources, and having important research significance and application value.

Description

Technical Field

[0001] The present invention belongs to the field of neural networks, and in particular to an adaptive quantization method adapted for a neural network accelerator running on an FPGA. Background Art

[0002] To achieve high-speed, low-power computing, neural network accelerators generally support low numerical precision, such as 8-bit or 6-bit fixed-point operations. However, the native numerical precision of neural network models is generally 32-bit floating-point numbers. Therefore, when deploying neural network algorithms on neural network accelerators, it is necessary to automatically compress the neural network model into an 8-bit or 6-bit integer network through quantization.

[0003] NVIDIA, a global giant, dominates the neural network accelerator market with its comprehensive GPU+CUDA ecosystem. This approach maps floating-point data to integer data, but its products are expensive, lack independent controllability, and GPU computing performance and power consumption are inferior to those of FPGAs and ASIC chips. The mainstream domestic neural network accelerator solution is a scalable, domestically produced CPU+FPGA solution. Neural network models run on the FPGA for computation, relatively independent of the CPU, with the CPU primarily responsible for initializing the model and reading model results. To improve computational efficiency, this architecture requires the FPGA to autonomously perform the quantization calculations for each layer of the neural network model. Existing quantization methods require quantization software to pre-analyze the distribution and range of input, weight, and output data at each layer of the neural network model, calculate quantization parameters, and generate guidance files for the neural network accelerator to map floating-point data to integer data. This process fails to account for overflow issues that may arise during integer data operations when deployed on FPGAs, making it impossible to ensure the accuracy of computational results. This can lead to significant deviations in the results, especially for neural network model algorithms with high precision requirements. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of existing quantization methods that do not consider the overflow problem that may occur during integer data operations when deployed on FPGAs and cannot ensure the correctness of the calculation results, and to provide an adaptive quantization method suitable for neural network accelerators running on FPGAs.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] An adaptive quantization method for neural network accelerators running on FPGAs predicts the range and distribution of data in intermediate calculations based on the input and weight values ​​of the convolutional layer and the data bit width during the FPGA calculation process.

[0007] Calculating the overflow degree based on the actual calculation bit width in the FPGA calculation process and the range and distribution of the intermediate calculation process data;

[0008] Based on the overflow degree, the KL-divergence method is used to adaptively adjust the quantization parameters until the difference between the distribution of the data after the adjustment of the quantization parameters and the distribution of the original data is within a preset range.

[0009] Furthermore, the quantization parameters are adjusted according to the KL-divergence method, and the relative entropy method is used to measure the difference between the data distribution after the adjustment of the quantization parameters and the original data distribution, and the optimal threshold is found as the quantization parameter.

[0010] Furthermore, a loop count i for constructing new samples is set, and reference samples P and new samples Q are continuously constructed. The relative entropy between the two is calculated, and the minimum relative entropy is obtained. At this time, the corresponding threshold is the optimal threshold.

[0011] Further, the specific steps are:

[0012] (1) Continuously truncate the reference sample P and sum all the values ​​outside the truncation area;

[0013] (2) Add the value outside the truncation region to the last value of the truncation sample;

[0014] (3) Obtain the probability distribution of the reference sample P;

[0015] (4) Create a new sample Q, whose element values ​​are the truncated sample P;

[0016] (5) Extend the length of the new sample Q to i so that the sample Q and the reference sample P have the same length;

[0017] (6) Obtain the probability distribution of the new sample Q;

[0018] (7) Calculate the KL divergence value of the reference sample P and the new sample Q.

[0019] Furthermore, when the input and weight values ​​of the convolutional layer are both 8-bit integers, if the data bit width during FPGA calculation is 20 bits, the data range that can be represented is -524288 to 524287, and data outside this range will be truncated;

[0020] Based on the actual calculation bit width during the FPGA calculation process, the range and distribution of the data in the intermediate calculation process are predicted. According to the degree of overflow, the KL-divergence method is used to adaptively adjust the quantization parameters until the distribution of the data after the adjustment is closest to the original real data distribution.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention's adaptive quantization method, adapted for neural network accelerators running on FPGAs, automatically predicts the degree of overflow during neural network accelerator calculations based on the actual bit width of the neural network accelerator's computations and adaptively adjusts quantization parameters based on the degree of overflow, preventing data overflow during neural network algorithm calculations on the FPGA and thus ensuring the correctness of neural network model results. The present invention's adaptive quantization method combines quantization operations with resource planning for neural network accelerator hardware, ensuring the correctness of results when algorithms are deployed on the neural network accelerator. Without sacrificing model accuracy or execution efficiency, it can effectively compress model size, making it easier to deploy and implement in resource-constrained situations, saving storage space and computing resources. The method has significant research significance and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the process of calculating the convolutional layer of the neural network using FPGA of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] Existing quantization methods do not consider the overflow issues that may arise during integer data operations when deployed on a neural network accelerator. The present invention proposes an adaptive quantization method suitable for neural network accelerators running on FPGAs. This method automatically predicts overflow conditions during neural network accelerator calculations based on the actual bit width of the neural network accelerator's calculation process and adaptively adjusts quantization parameters based on the degree of overflow, avoiding data overflow issues during neural network algorithm calculations on the FPGA and thus ensuring the correctness of neural network model results. The adaptive quantization method of the present invention combines quantization operations with resource planning for neural network accelerator hardware, ensuring the correctness of results when the algorithm is deployed on the neural network accelerator. Without sacrificing model accuracy and execution efficiency, it can effectively compress the model size, making it easier to deploy and implement under resource-constrained conditions, saving storage space and computing resources, and has important research significance and application value.

[0027] The present invention is described in further detail below with reference to the accompanying drawings:

[0028] See also Figure 1 , Figure 1 This diagram illustrates the FPGA computation process for a neural network convolutional layer. Both the input and weight values ​​of the convolutional layer are 8-bit integers. If the data bit width during FPGA computation is 20 bits, the range of data that can be represented is -524288 to 524287; data outside this range is truncated. Considering the actual bit width limitations of the FPGA computation process, the range and distribution of data during intermediate computations are predicted. Based on the degree of overflow, the KL-divergence method is used to adaptively adjust the quantization parameters, ensuring that the distribution of the adjusted data closely matches the original data distribution.

[0029] Example

[0030] The present invention is further described in detail by taking a neural network model with only one Convolution layer as an example.

[0031] (1) Without adopting the present invention, that is, without considering the overflow problem that may occur during integer data calculation when deployed on FPGA, the calculation process and results are derived as follows:

[0032] The Convolution layer parameters in this neural network model are pad=0, stride=1, and the input size is 1×3×3. The filter size is 2×2 and its value is The bias is 1.213093. The process of deploying the neural network model on the neural network accelerator is as follows:

[0033] (a) Quantization coefficients are pre-calculated by quantization software

[0034] Take the quantization bit width as 8 as an example. The quantization parameter th is obtained by the quantization software. in =11.32375,th w =8.134685,th iout =153.37865. The quantization coefficients of input, weight, and output are:

[0035]

[0036]

[0037]

[0038] (b) Pre-quantize inputs and weights to integers

[0039] Input data Quantized to integer according to the quantization coefficient Weight Quantized to integer

[0040] (c) Pre-calculate the multiplication, addition, and shift parameters required for deployment on the FPGA

[0041] The shift parameter n, multiplication parameter, and addition parameter are:

[0042]

[0043]

[0044]

[0045] (d) FPGA calculation process

[0046] The input and output data of the FPGA convolution layer are 8 bits wide. The intermediate calculation process is assumed to have a 20-bit bit width, which can represent data in the range of -524288 to 524287. Data outside this range will be truncated.

[0047] After quantization, the integer input data of the neural network accelerator is The weight is The result of multiplication and accumulation of the two is Multiply by the multiplication parameter 76, and we get The data exceeds the range of -524288 to 524287 that can be represented by a 20-bit width and is truncated to Add the parameter 16039 and truncate it to get Shift right 14 bits in binary to get

[0048] (e) Verification of the correctness of neural network accelerator inference calculation results

[0049] The output of the Convolution layer calculated by the quantized neural network accelerator is: The corresponding true value can be obtained from out_float=out_Integer·scaling The actual result of calculating the Convolution layer in floating-point data format is The calculation results led to huge errors.

[0050] (2) This invention considers the overflow problem that may occur during integer data operations when deployed on FPGA and adaptively adjusts the quantization parameters. The calculation process and results are derived as follows:

[0051] The Convolution layer parameters in this neural network model are pad=0, stride=1, and the input size is 1×3×3. The filter size is 2×2 and its value is The bias is 1.213093. The process of deploying the neural network model on the neural network accelerator is as follows:

[0052] (a) Quantization coefficients are pre-calculated by quantization software

[0053] Take the quantization bit width as 8 as an example. The quantization parameter th is obtained by the quantization software. in =11.32375,th w =8.134685,th iout =153.37865. Based on the actual bit width of the neural network accelerator calculation process, the overflow condition of the neural network accelerator calculation is automatically predicted. According to the degree of overflow, the KL-divergence method is used according to the data distribution to adaptively adjust the quantization parameter to th in =22.6475,th w =16.26937,th iout =153.37865, the quantization coefficients of input, weight, and output are:

[0054]

[0055]

[0056]

[0057] (b) Pre-quantize inputs and weights to integers

[0058] Input data Quantize to integer according to the adaptively adjusted quantization coefficient Weight Quantized to integer

[0059] (c) Pre-calculate the multiplication, addition, and shift parameters required for deployment on the FPGA

[0060] The shift parameter n, multiplication parameter, and addition parameter are:

[0061]

[0062]

[0063]

[0064] (d) FPGA calculation process

[0065] The input and output data of the FPGA convolution layer are 8 bits wide. The intermediate calculation process is assumed to have a 20-bit bit width, which can represent data in the range of -524288 to 524287. Data outside this range will be truncated.

[0066] After quantization, the integer input data of the neural network accelerator is The weight is The result of multiplication and accumulation of the two is Multiply by the multiplication parameter 74, and we get The data range that can be represented by the 20-bit width is -524288 to 524287, plus the parameter 4009 to get Shift right 12 bits in binary to get

[0067] (e) Verification of the correctness of neural network accelerator inference calculation results

[0068] The output of the Convolution layer calculated by the quantized neural network accelerator is: The corresponding true value can be obtained from out_float=out_Integer·scaling The actual result of calculating the Convolution layer in floating-point data format is Comparing the two, the data error caused by quantification is The average error is 0.83%, which proves that the invention can effectively avoid the overflow problem when calculating the neural network model on FPGA and ensure the correctness of the calculation results.

[0069] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. An adaptive quantization method adapted for a neural network accelerator running on an FPGA, characterized in that: Predict the range and distribution of data in the intermediate calculation process based on the input and weight values ​​of the convolutional layer and the data bit width during the FPGA calculation process; Calculating the overflow degree based on the actual calculation bit width in the FPGA calculation process and the range and distribution of the intermediate calculation process data; Based on the overflow degree, the KL-divergence method is used to adaptively adjust the quantization parameters until the difference between the distribution of the data after the adjustment of the quantization parameters and the distribution of the original data is within the preset range; Adjust the quantization parameters according to the KL-divergence method, use relative entropy to measure the difference between the data distribution after the adjustment and the original data distribution, and find the optimal threshold as the quantization parameter; When the input and weight values ​​of the convolutional layer are both 8-bit integers, if the data bit width during FPGA calculation is 20 bits, the data range that can be represented is -524288 to 524287. Data outside this range will be truncated. Based on the actual calculation bit width during the FPGA calculation process, the range and distribution of the data in the intermediate calculation process are predicted. According to the degree of overflow, the KL-divergence method is used to adaptively adjust the quantization parameters until the distribution of the data after the adjustment is closest to the original real data distribution.

2. The adaptive quantization method for a neural network accelerator adapted to run on an FPGA according to claim 1, characterized in that: Set the loop count i for constructing new samples, continuously construct reference samples P and new samples Q, calculate the relative entropy between the two, and obtain the minimum relative entropy. At this time, the corresponding threshold is the optimal threshold.

3. The adaptive quantization method for a neural network accelerator adapted to run on an FPGA according to claim 2, characterized in that: The specific steps are: (1) Continuously truncate the reference sample P and sum all the values ​​outside the truncation area; (2) Add the values ​​outside the truncation area to the last value of the truncation sample; (3) Obtain the probability distribution of the reference sample P; (4) Create a new sample Q, whose element value is the truncated sample P; (5) Extend the length of the new sample Q to i so that the sample Q and the reference sample P have the same length; (6) Obtain the probability distribution of the new sample Q; (7) Calculate the KL divergence value of the reference sample P and the new sample Q.

Images