A mixed-precision quantization method for deep convolutional neural networks applied to FPGA

Abstract

The present invention discloses a mixed-precision quantization method for deep convolutional neural networks applied to FPGAs. First, several appropriate quantization bit widths are selected based on the logic resource capacity of the FPGA hardware platform where the neural network will be deployed and the accuracy requirements for each layer; second, the importance of the current layer weight is judged based on the peak value of the weight distribution histogram of the neural network used, and a quantization bit width is selected as the quantization bit width of the current layer weight; then, during the forward reasoning process, the importance of the current layer activation value is judged based on the peak value of the activation value distribution histogram, and a quantization bit width is selected as the quantization bit width of the current layer activation value; finally, a symmetric uniform quantization method is adopted. This method analyzes the weight distribution of different layers of the deep convolutional neural network, and assigns corresponding quantization bit widths to the weights and activation values based on the sensitivity of different layers to quantization. Unlike the traditional method of obtaining the optimal mixed bit quantization model based on iterative search of the mixed bit quantization space, it realizes a more convenient and fast mixed-precision quantization strategy.

Description

Background Art

[0001] Deep convolutional neural networks (DCNNs) have significantly improved the accuracy of object detection applications. However, the enormous computational overhead and complexity of DCNNs, as well as the large number of memory access operations, pose significant challenges to the core execution of object detection tasks. Currently, mainstream DCNNs have hundreds of millions of connections and are both memory- and computation-intensive. As existing applications demand more precision and functionality from DCNNs, the scale of neural networks continues to increase. When the size of the weight matrix is much larger than the cache size, memory access becomes a major bottleneck in neural network computation.

[0002] The basic operations of DCNNs involve a large number of multiplication-accumulation operations. Multiplications consume a lot of hardware resources, consume a lot of power, and have long latency. This problem is particularly prominent in edge intelligence scenarios. In most cases, edge devices lack sufficient storage and computing resources and are power-constrained, making deep convolutional neural networks difficult to deploy on embedded devices. Therefore, at the algorithmic level, model compression is necessary through methods such as pruning, quantization, low-rank approximation, and knowledge distillation. Pruning and quantization are the most commonly used. Quantization is one of the most effective methods for reducing neural network computation time and power consumption. In neural network quantization, the bit width of weight and activation tensors is reduced compared to the 16-bit or 32-bit precision commonly used in training. This effectively reduces storage memory overhead and the computational cost of matrix multiplication. This process is also known as software optimization. To address these issues at the hardware level, it is necessary to build optimized accelerators using high-parallelism and high-data-reusability optimization methods on the deployment platforms of these models. Currently, the main hardware accelerators include graphics processing units (GPUs), application-specific integrated circuits (ASICs), and field programmable logic arrays (FPGAs). GPUs consume high power, ASICs have long customization cycles, and lack flexibility. FPGAs, on the other hand, offer high parallelism, good data locality, and reconfigurability. FPGA-based hardware accelerators balance the gaps between performance, power consumption, and flexibility. To further improve computational efficiency, DNN hardware accelerators are beginning to support mixed precision (1-8 bits). Compared to single-precision quantization, mixed-precision quantization strategies can better leverage the flexibility of hardware logic resources. However, this also presents a significant challenge: how to select the optimal bit width for each layer using a mixed-precision quantization strategy. This requires domain experts to explore a vast design space to achieve a trade-off between accuracy, latency, energy efficiency, and model size.

[0003] Based on current research, the enormous amount of computation and memory access required for floating-point convolution calculations in neural networks can lead to resource and power constraints. Existing, mature quantization techniques, such as DOREFA-NET, can quantize weights, activations, and gradients with arbitrary precision. In order to better adapt to hardware platforms, quantization strategies tend to adopt mixed-precision quantization. Currently, most mixed-precision quantization algorithms for deep convolutional neural networks focus solely on designing search algorithms to achieve the desired mixed-precision quantization model, but fail to consider the cumbersome and time-consuming implementation caused by the large search space for quantization precision. Summary of the Invention

[0004] In order to solve the problem that most current mixed-precision quantization algorithms for deep convolutional neural networks usually only focus on how to design a search algorithm to obtain a mixed-precision quantization model that can achieve the expected results, but do not consider the problem that the search space for quantization precision is too large, which makes the implementation more cumbersome and time-consuming, the present invention proposes a mixed-precision quantization method for deep convolutional neural networks applied to FPGA.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A mixed-precision quantization method for deep convolutional neural networks applied to FPGAs includes the following steps:

[0007] Step 1: Select the dataset and the required network model, select a set of quantization bit width values to be used in mixed precision quantization to allocate quantization bit widths to each layer of the model, and arrange them in ascending order according to the element values within the set;

[0008] Step 2: Compute the weight distribution histogram of the neural network used. The horizontal axis of the histogram corresponds to the weight, and the vertical axis corresponds to the number of weights. The weight points concentrated in the histogram reflect the large number of current weights. The peak of the histogram is used as the typical value of the current layer weight. The importance of the current layer weight is judged according to the size of the typical value and the quantization bit width of the weight is allocated accordingly. The convolution layer with a large typical value is assigned a higher quantization bit width.

[0009] Step 3: During the forward inference process during training, the peak value of the activation value distribution histogram is used as the typical value of the activation value of the current layer. The importance of the activation value of the current layer is judged according to the size of the typical value and the quantization bit width of the activation value is allocated accordingly. The convolution layer with a large typical value is assigned a higher quantization bit width;

[0010] Step 4: Train the network model with appropriate bit width assigned to each layer, using symmetric uniform quantization.

[0011] Step 5: Perform inference on the FPGA side and use bit operations to implement convolution operations between low-bit-width fixed-point integers.

[0012] In particular, step 1 specifically includes the following steps:

[0013] The dataset to be used is divided into training set, test set and validation set. A subset of the divided training set is randomly selected as the calibration set, which includes images and label values. A set of quantization bit width values is selected for mixed precision quantization to allocate quantization bit widths to each layer of the model. The elements in the set are arranged in ascending order according to their element values. The length of the quantization set is N. A deep convolutional neural network that has been trained to convergence is given.

[0014] In particular, step 2 specifically includes the following steps:

[0015] The weight distribution histogram of each layer of the given deep convolutional neural network is statistically obtained; for each layer, the weight value corresponding to the peak of the weight distribution histogram is taken as the typical value of the current layer weight; the typical values of the weight of each layer obtained are sorted from small to large according to the size after taking the absolute value to obtain a typical value set, and the obtained typical value set is divided into N subsets according to the length N of the quantization set obtained in step 1, and the values in the quantization set are sequentially assigned to these N sub-typical value sets; the quantization bit width corresponding to each typical value in the typical value set is the weight quantization bit width of the corresponding layer of this typical value in the model.

[0016] In particular, step 3 specifically includes the following steps:

[0017] Take the image and label value of the calibration set in step 1 as input and perform a forward inference process on the given floating-point model to obtain the activation value calculated for each layer. Statistically calculate the distribution histogram of the activation value of each layer. For each layer, take the activation value corresponding to the peak of the activation value distribution histogram as the typical value of the current layer activation. Sort the typical values of the activation values of each layer obtained from small to large according to the absolute value to obtain a typical value set. Divide the obtained typical value set into N subsets according to the length N of the quantization set obtained in step 1, and assign the values in the quantization set to these N sub-typical value sets in turn. The quantization bit width corresponding to each typical value in the typical value set is the quantization bit width of the activation value of the corresponding layer in the model.

[0018] In particular, step 4 specifically includes the following steps:

[0019] Combine steps 2 and 3 to obtain the combination of quantization bit widths for each layer of the model. Use the images and label values of the training set in step 1 as input to perform quantization-aware training on the current network model. Selecting symmetric uniform quantization as the quantization rule for each layer can make it easier to implement in hardware, as follows:

[0020] During the training process, the floating-point numbers of each layer are converted to their corresponding k-bit fixed-point numbers using the following formula;

[0021]

[0022] α=max|R| (2)

[0023]

[0024] where r i Represents the input floating point number, r i The value range is [-α,α]; r0 is the k-bit fixed-point number after quantization; R represents the number of r in this layer. i The set of all values; max|R| represents the absolute value of the floating-point value set and then takes the maximum value; Δ represents the scaling factor; in order to avoid the problem of zero gradient in backpropagation, the straight-through estimator STE is used to approximate the derivative of the floating-point number with the derivative of the low-precision weight:

[0025]

[0026] Among them, Loss represents the total loss error, W and W Q Represent floating-point weights and quantized weights respectively.

[0027] In particular, step 5 specifically includes the following steps:

[0028] When performing inference on the FPGA side, refer to DoreFa-Net, which uses bit operations to implement convolution operations between low-bit-width fixed-point integers:

[0029]

[0030] in are sets of M-bit and K-bit fixed-point integer sequences, and is a bit vector, and the computational complexity of formula (4) is O(MK), which is proportional to the bit width of x and y.

[0031] Beneficial effects of the present invention:

[0032] Compared with the traditional method of obtaining the optimal mixed bit quantization model based on iterative search in mixed bit quantization space, the mixed precision quantization method for deep convolutional neural network applied to FPGA described in the present invention targets the characteristics that different layers of deep convolutional neural network have different weight value distributions and different sensitivities to quantization. In combination with the characteristics of the hardware, a higher quantization bit width is allocated to the layers with higher peaks in the weight or activation distribution histogram, i.e., the layers that are more sensitive to quantization, and a lower quantization bit width is allocated to the layers with lower peaks in the weight or activation distribution histogram, i.e., the layers with lower quantization sensitivity. In order to find the optimal bit width for each layer, the traditional mixed precision quantization strategy requires field experts to explore a huge design space to achieve a trade-off between accuracy, latency, energy efficiency and model size. Unlike the traditional method of obtaining the optimal mixed bit quantization model based on iterative search in mixed bit quantization space, the present invention narrows the search space to a certain extent and designs a more convenient and fast mixed precision quantization strategy.

[0033] The present invention uses a mixed-precision quantization strategy to perform low-bit quantization on both the weights and activation values in the neural network. Referring to existing mature quantization technologies such as DOREFA-NET, the hardware side uses a bit operation calculation method, and its computational complexity is proportional to the bit width of each layer's weights and activation values. Different layers have different quantization bit widths and their computational complexity is also different. Compared with traditional calculation methods, the overall computational complexity will be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of a mixed-precision quantization method for a deep convolutional neural network for FPGA according to the present invention;

[0035] Figure 2 It is a detailed flowchart of the steps for quantizing the weight accuracy of each layer of the mixed precision quantization method of the deep convolutional neural network for FPGA according to the present invention;

[0036] Figure 3 This is a detailed flowchart of the steps for quantizing the accuracy of activation values of each layer of the mixed-precision quantization method for deep convolutional neural networks for FPGA according to the present invention. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings in the embodiments of the present disclosure:

[0038] like Figure 1 As shown, the present invention provides a mixed-precision quantization method for a deep convolutional neural network applied to FPGA, comprising the following steps:

[0039] S1. Divide the dataset to be used into training set, test set and validation set, and randomly select a subset of the divided training set as the calibration set; select a set of quantization bit width values (8bit, 4bit, 2bit) and sort them from small to large to obtain a quantization set of length N (using 8bit, 4bit, 2bit, N=3); given a deep convolutional neural network that has been trained to convergence.

[0040] S2, use the TensorBoard tool to view the weight distribution histogram of each layer of the given deep convolutional neural network; for each layer, take the weight value corresponding to the peak of the weight distribution histogram as the typical value of the current layer weight; sort the typical values of each layer from small to large according to their absolute values to obtain a typical value sequence, and divide the obtained typical value sequence into N subsequences according to the length N of the quantization set obtained in S1. The specific method is as follows Figure 2 As shown:

[0041] S21 adds up the values in the obtained typical value sequence and calculates the average, divides out the values greater than the average to form typical value subsequence 1, then adds up the remaining typical values and calculates the average, divides out the values greater than the average to form typical value subsequence 2, and repeats the above steps for the remaining values until N subsequences are divided out; S22 assigns the N values in the quantization set to the above N typical value subsequences in order from large to small, and the quantization bit width corresponding to each typical value in the typical value sequence is the weight quantization bit width of the corresponding layer of this typical value in the model.

[0042] S3, the activation value is quantized as follows Figure 3 As shown, S31 uses the image and label value of the calibration set in S1 as input to perform a forward inference process on the given floating-point model to obtain the activation value calculated for each layer, and statistically calculates the distribution histogram of the activation value of each layer. For each layer, the activation value corresponding to the peak of the activation value distribution histogram is taken as the typical value of the activation value of the current layer; the typical values of each layer are sorted from small to large according to their absolute values to obtain a typical value sequence, and the obtained typical value sequence is divided into N subsequences according to the length N of the quantization set obtained in S1. The specific method is:

[0043] S32 adds up the values in the obtained typical value sequence and calculates the average, divides out the values greater than the average to form typical value subsequence 1, then adds up the remaining typical values and calculates the average, divides out the values greater than the average to form typical value subsequence 2, and repeats the above steps for the remaining values until N subsequences are divided out; S33 assigns the N values in the quantization sequence to the above N typical value subsequences in order from large to small, and the quantization bit width corresponding to each typical value in the typical value sequence is the quantization bit width of the activation value of the corresponding layer of this typical value in the model.

[0044] S4, obtaining a combination of quantization bit widths of each layer to be quantized of the model according to the mixed precision quantization strategy (S2, S3).

[0045] S5, according to the mixed precision quantization strategy, uses the images and label values of the training set in S1 as input to perform quantization-aware training on the current network model; symmetric uniform quantization is used as the quantization rule for each layer, which can be more easily implemented in hardware. The details are as follows:

[0046] During the training process, the floating-point number of each layer is converted into its corresponding k-bit fixed-point number using the following formula.

[0047]

[0048] α=max|R| (2)

[0049]

[0050] where r i Represents the input floating point number, r i The value range is [-α,α]; r0 is the k-bit fixed-point number after quantization; R represents the number of r in this layer. i The set of all values; max|R| represents the absolute value of the floating-point value set and then takes the maximum value; Δ represents the scaling factor; in order to avoid the problem of zero gradient in back propagation, STE (Straight-ThroughEstimator) is used to approximate the derivative of the floating-point number with the derivative of the low-precision weight:

[0051]

[0052] Among them, Loss represents the total loss error; W and W Q Represent floating-point weights and quantized weights respectively.

[0053] S6, performing inference on the FPGA side, refers to DoreFa-Net and uses bit operations to implement convolution operations between low-bitwidth fixed-point integers:

[0054]

[0055] in are sets of M-bit and K-bit fixed-point integer sequences, and is a bit vector. The computational complexity of formula (4) is O(MK), which is proportional to the bit width of x and y.

[0056] To sum up, the present invention takes into account the characteristics that different layers of deep convolutional neural networks have different weight value distributions and different sensitivities to quantization, and combines the logical resource capacity of the hardware platform and the requirements for the accuracy of each layer to allocate higher bit precision to layers that are more sensitive to quantization and lower bit precision to layers with low quantization sensitivity. Compared with most current deep convolutional neural network mixed-precision quantization algorithms, which usually only focus on how to design search algorithms to obtain a mixed-precision quantization model that can achieve the expected results, but do not take into account the shortcomings of cumbersome and time-consuming implementation due to the large search space for quantization accuracy, the present invention designs a more convenient and fast mixed-precision quantization strategy.

Claims

1. A mixed-precision quantization method for deep convolutional neural networks applied to FPGAs, comprising the following steps: Step 1: Select the dataset and the required network model, select a set of quantization bit width values for mixed precision quantization to allocate quantization bit widths to each layer of the model, and arrange the elements in ascending order within the set, including: The dataset to be used is divided into a training set, a test set, and a validation set. A subset of the training set is randomly selected as the calibration set, which includes images and label values. A set of quantization bit width values is selected to allocate quantization bit widths to each layer of the model in mixed precision quantization. The elements in the set are arranged in ascending order according to their element values. The length of the quantization set is N. A deep convolutional neural network that has been trained to convergence is given. Step 2: Compute the weight distribution histogram of the neural network used. The horizontal axis of the histogram corresponds to the weight, and the vertical axis corresponds to the number of weights. The weight points concentrated in the histogram reflect the large number of current weights. The peak of the histogram is used as the typical value of the current layer weight. The importance of the current layer weight is judged according to the size of the typical value and the quantization bit width of the weight is allocated accordingly. The convolution layer with a large typical value is assigned a higher quantization bit width. Step 3: During the forward inference process during training, the peak value of the activation value distribution histogram is used as the typical value of the activation value of the current layer. The importance of the activation value of the current layer is judged according to the size of the typical value and the quantization bit width of the activation value is allocated accordingly. The convolution layer with a large typical value is assigned a higher quantization bit width; Step 4: Train the network model with appropriate bit width assigned to each layer, using symmetric uniform quantization. Step 5: Perform inference on the FPGA side and use bit operations to implement convolution operations between low-bit-width fixed-point integers.

2. The mixed-precision quantization method for deep convolutional neural networks applied to FPGAs according to claim 1, characterized in that: Step 2 specifically includes the following steps: The weight distribution histogram of each layer of the given deep convolutional neural network is statistically obtained; for each layer, the weight value corresponding to the peak of the weight distribution histogram is taken as the typical value of the current layer weight; the typical values of the weight of each layer obtained are sorted from small to large according to the size after taking the absolute value to obtain a typical value set, and the obtained typical value set is divided into N subsets according to the length N of the quantization set obtained in step 1, and the values in the quantization set are sequentially assigned to these N sub-typical value sets; the quantization bit width corresponding to each typical value in the typical value set is the weight quantization bit width of the corresponding layer of this typical value in the model.

3. The mixed-precision quantization method for deep convolutional neural networks applied to FPGA according to claim 1, characterized in that: Step 3 specifically includes the following steps: Take the image and label value of the calibration set in step 1 as input and perform a forward inference process on the given floating-point model to obtain the activation value calculated for each layer. Statistically calculate the distribution histogram of the activation value of each layer. For each layer, take the activation value corresponding to the peak of the activation value distribution histogram as the typical value of the current layer activation. Sort the typical values of the activation values of each layer obtained from small to large according to the absolute value to obtain a typical value set. Divide the obtained typical value set into N subsets according to the length N of the quantization set obtained in step 1, and assign the values in the quantization set to these N sub-typical value sets in turn. The quantization bit width corresponding to each typical value in the typical value set is the quantization bit width of the activation value of the corresponding layer in the model.

4. The mixed-precision quantization method for deep convolutional neural networks applied to FPGA according to claim 1, characterized in that: Step 4 specifically includes the following steps: Combine steps 2 and 3 to obtain the combination of quantization bit widths for each layer of the model. Use the images and label values of the training set in step 1 as input to perform quantization-aware training on the current network model. Selecting symmetric uniform quantization as the quantization rule for each layer can make it easier to implement in hardware, as follows: During the training process, the floating-point numbers of each layer are converted to their corresponding k-bit fixed-point numbers using the following formula; (1) (2) (3) in Represents the input floating point number, The value range is [- , ]; is the k-bit fixed-point number after quantization; R represents the number of The set of all values; max| R | means taking the absolute value of the values in the floating-point set and then taking the maximum value; Represents the scaling factor; in order to avoid the problem of zero gradient in back propagation, the straight-through estimator STE is used to approximate the derivative of the floating-point number with the derivative of the low-precision weight: in, represents the total loss error, W and Represent floating-point weights and quantized weights respectively.

5. The mixed-precision quantization method for deep convolutional neural networks applied to FPGA according to claim 1, characterized in that: Step 5 specifically includes the following steps: When performing inference on the FPGA side, refer to DoreFa-Net, which uses bit operations to implement convolution operations between low-bit-width fixed-point integers: (4) Where x= , are sets of M-bit and K-bit fixed-point integer sequences, and is a bit vector, the computational complexity of formula (4) is O(MK), which is proportional to the bit width of x and y.

Images