An AI accelerator quantization algorithm based on deep learning

Abstract

The application discloses an AI accelerator quantization algorithm based on deep learning, which comprises the following steps: obtaining parameters of a pre-trained model, limiting and mapping input activation values of weights to obtain bit width of input activation; preprocessing floating-point data of the pre-trained model; grouping and quantizing a first tensor based on the bit width of the input activation; accumulating absolute values of the grouped and quantized data, and calculating errors between the absolute value accumulations of the original data and the grouped and quantized data; setting a hyperparameter space and adjusting hyperparameter combinations; screening the grouped and quantized data based on different hyperparameter combinations and the absolute value accumulations; training the pre-trained model and minimizing a loss function to obtain optimal bit weight distribution and corresponding grouped and quantized data. By using the application, data of each layer can be quantized into appropriate quantization bit width flexibly under the premise of maintaining accuracy. The application can be widely applied in the field of artificial intelligence technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to an AI accelerator quantization algorithm based on deep learning. BACKGROUND

[0002] In recent years, convolutional neural networks have achieved great success in many computer vision problems such as image classification, object detection and instance segmentation. However, as the performance of neural networks continues to improve, the computational complexity of the model also increases dramatically. Modern deep learning models usually need to run on platforms with large capacity of memory and high performance computing devices, such as graphics processing units, which leads to high computing cost. At the same time, there is an increasing demand for neural network models on mobile devices and embedded devices, and the computing performance, power consumption and available memory of these devices are limited. Therefore, how to efficiently deploy deep neural network models on edge devices has become a problem to be solved.

[0003] However, the existing method can only quantize the data to different precisions layer by layer while maintaining the accuracy, and cannot quantize different data in the same layer to different precisions. Or similar to using a binary neural network, although it can be quantized to the desired precision, it will also cause significant accuracy degradation. The existing quantization algorithm cannot quantize the data of each layer to the appropriate quantization bit width flexibly according to the characteristics and distribution of the data while maintaining the accuracy. SUMMARY

[0004] In order to solve the above technical problems, the purpose of the present application is to provide an AI accelerator quantization algorithm based on deep learning, which can quantize the data of each layer to the appropriate quantization bit width flexibly while maintaining the accuracy.

[0005] The first technical solution adopted by the present application is: an AI accelerator quantization algorithm based on deep learning, comprising the following steps:

[0006] Obtain a pre-trained model for a specific task to obtain parameters of the pre-trained model;

[0007] Limit and map the input activation value of the weight to obtain the bit width of the input activation;

[0008] Preprocess the floating point data of the pre-trained model to obtain a first tensor;

[0009] Group quantize the first tensor based on the bit width of the input activation to obtain group quantization data;

[0010] Accumulate the absolute value of the group quantization data, and calculate the error between the absolute value accumulation value of the original data and the absolute value accumulation value of the group quantization data;

[0011] A hyperparameter space is set, and a hyperparameter combination is adjusted based on a grid search algorithm to obtain different hyperparameter combinations;

[0012] Group quantization data is screened based on the error of different hyperparameter combinations and the absolute value accumulation value to obtain group quantization data of different bit weight value distributions;

[0013] The pre-training model is trained based on the group quantization data of different bit weight value distributions, and the loss function is minimized to obtain the optimal bit weight value distribution.

[0014] Further, the step of limiting and mapping the input activation value of the weight to obtain the bit width of the input activation, specifically includes:

[0015] The activation value of the weight is limited and mapped, and the integer representation of the activation value is obtained by rounding;

[0016] The integer representation of the activation value is quantized to obtain the bit width of the input activation.

[0017] Further, the step of preprocessing the floating-point data of the pre-training model to obtain the first tensor, specifically includes:

[0018] The floating-point data is mapped based on the hyperbolic tangent function to obtain a mapping tensor;

[0019] The mapping tensor is normalized based on the maximum absolute value of the floating-point data to obtain the first tensor.

[0020] Further, the step of grouping quantization of the first tensor based on the bit width of the input activation to obtain quantization data, specifically includes:

[0021] The number of groups of data in the first tensor is determined based on the bit width of the input activation to obtain a plurality of group data;

[0022] The maximum value of the group data is traversed, and the highest bit value of the group data is obtained based on the logarithm function with base 2 and the floor function to obtain the value of the highest bit;

[0023] The value of the highest bit is precision divided based on the power function of 2 to obtain a quantization parameter;

[0024] The group data is quantized based on the quantization parameter to obtain a second tensor;

[0025] The second tensor is subjected to bit sparsity operation to obtain quantization data.

[0026] Further, the step of screening group quantization data based on the error of different hyperparameter combinations and the absolute value accumulation value to obtain group quantization data of different bit weight value distributions, specifically includes:

[0027] compare the error sum of said absolute value accumulation value with the threshold of quantization error of different hyperparameter combinations, to obtain a comparison result;

[0028] filter the grouped quantization data based on the comparison result, to obtain grouped quantization data with different bit weight distribution.

[0029] Through the preferred steps, the weight distribution effect can be comprehensively evaluated under different error conditions.

[0030] Further, the hardware design method of the AI accelerator quantization algorithm based on deep learning combines the lowest calculation units freely through addition and shifting to obtain a hybrid calculation array, so as to realize hardware calculation of the grouped quantization data with optimal bit weight distribution.

[0031] The method has the following beneficial effects: the method explores the optimal hyperparameter combination by using the hyperparameter space and the grid search algorithm, comprehensively evaluates the weight distribution effect under different error conditions, automatically searches and optimizes the quantization strategy of each layer without manual intervention and adjustment, realizes the global optimal solution of the weight distribution through loss function minimization, improves the universality and scalability of the quantization algorithm, realizes different quantization bit widths and methods for different data in the same layer by grouping and quantizing the data of each layer into mixed precision, thereby better adapting to the diversity and dynamics of the data, and finally maximally compresses the model size without affecting the model accuracy, reduces the memory occupation and access overhead of the model, reduces the calculation complexity and energy consumption of the model, and improves the running speed and energy efficiency of the AI accelerator. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a step flowchart of the AI accelerator quantization algorithm based on deep learning of the present application;

[0033] Figure 2 is a quantization flowchart of the AI accelerator quantization algorithm based on deep learning of the present application;

[0034] Figure 3 is a hyperparameter search flowchart of the AI accelerator quantization algorithm based on deep learning of the present application;

[0035] Figure 4 is a bit sparsity conversion schematic diagram of the AI accelerator quantization algorithm based on deep learning of the present application;

[0036] Figure 5 is a quantization-aware training schematic diagram of the AI accelerator quantization algorithm based on deep learning of the present application;

[0037] Figure 6This is a schematic diagram of a 4-bit*2-bit multiplication of an AI accelerator quantization algorithm based on deep learning in the present invention;

[0038] Figure 7 This is a schematic diagram of two 2-bit multiplication, shift and accumulation operations of an AI accelerator quantization algorithm based on deep learning in the present invention;

[0039] Figure 8 This is a schematic diagram of a 2-bit hybrid computing array for an AI accelerator quantization algorithm based on deep learning in the present invention;

[0040] Figure 9 This is a schematic diagram of a 4-bit hybrid computing array of an AI accelerator quantization algorithm based on deep learning in the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0042] Reference Figure 1 and Figure 2 , the present invention provides an AI accelerator quantization algorithm based on deep learning, the method comprising the following steps:

[0043] S1. Obtain a pre-trained model for a specific task and obtain the parameters of the pre-trained model;

[0044] Specifically, the pre-trained model selected for the specific implementation of the present invention is the ResNet convolutional neural network model, which is widely used in many fields such as image classification, target detection, and semantic segmentation. For example, in image classification tasks, ResNet can improve classification accuracy by continuously increasing the network depth; in target detection tasks, ResNet can be used as a feature extraction network to extract image features and input them into the target detector for target detection; in semantic segmentation tasks, ResNet can be used as an encoder to extract image features and pass the features to the decoder for pixel-level semantic segmentation. This embodiment applies it to the classification scenario of male and female images, collects 500 male and female images with labeled semantics in advance, pre-trains the ResNet model, obtains the pre-trained model, and obtains various parameters of the pre-trained model.

[0045] S2. Limit and map the input activation value of the weight to obtain the bit width of the input activation;

[0046] Specifically, the activation value of the weight is limited and mapped, and is quantized to an integer representation; the integer representation of the activation value is quantized to obtain the bit width k of the input activation; that is, the full-precision between [-1, 1] is quantized to a k-bit quantized number between [-1, 1]. When quantizing, note that the forward quantization function is not unique and can be different quantization forms. In order to avoid the problem of 0 gradient, a straight-through estimator (STRAIGHT-THROUGHESTIMATOR, STE) is used. The reason for using the straight-through estimator (STRAIGHT-THROUGHESTIMATOR, STE) can be illustrated with an example. Suppose the network has a ReLU activation function, and the network is initially provided with a set of weights. The input of these ReLUs can be negative, which will cause the output of the ReLU to be 0. For these weights, the derivative of the ReLU will be 0 during backpropagation, which means that the network cannot learn anything from these derivatives, and the weights cannot be updated. In view of this, the straight-through estimator sets the gradient of the input to a threshold function equal to the gradient of its output, regardless of the actual derivative of the threshold function itself; the mapping expression of the input activation value of the weight is as follows:

[0047] Forward inference:

[0048] Backward inference:

[0049] where k represents the bit width of quantization, r o represents a k-bit fixed-point integer, r o ∈[-1,1], c represents the objective function, and by setting the value of k, the data can be quantized to the desired bit width, r i represents the input 32-bit floating-point data.

[0050] S3, pre-processing the floating-point data of the pre-trained model to obtain a first tensor;

[0051] Specifically, the pre-trained 32-bit floating-point pre-trained model is first input, and the tanh (hyperbolic tangent function) function in pyTorch is used to map the input floating-point data x to the range [-1, 1]. By limiting the data in this range, it can be ensured that the input of the activation function is always within its effective range, preventing the activation function from being over-saturated or over-saturated, which helps to stabilize the training of the model. After mapping each element of the model data through the tanh function, save it in the mapping tensor Tanh. At the same time, traverse the absolute values of all floating-point data in the model to find the maximum value max, and perform data normalization by dividing all mapped values by the maximum value max and saving them in the first tensor y. Normalizing the input data also helps to speed up the convergence of the model, because it is usually easier to optimize the parameters of a neural network in a smaller range. The expressions for data mapping and normalization are as follows:

[0052] Tanh = tanh(x)

[0053]

[0054] wherein Tanh represents a mapping tensor, tanh represents a tanh function, x represents a floating-point number data, y represents a first tensor, Tanh max represents a maximum value of the mapping tensor.

[0055] The first tensor after data normalization can be multiplied by 2 raised to the power of the bit width to be quantized minus 1, such as 255 for quantization to 8-bit maximum, to obtain the quantized fixed-point number.

[0056] S4, grouping quantization is performed on the first tensor based on the bit width of the input activation, to obtain grouped quantization data;

[0057] Specifically, with reference to Figure 5 , the present quantization method adopts QAT (quantization-aware training), which means that during forward propagation, a pseudo-quantization operation is used, x1 = S(Q(x) + Z), where x1 is the pseudo-quantization data corresponding to the input x (including quantization and dequantization processes), since x1 and x are not always equal, in QAT, the formula is used to introduce a simulation of quantization loss, and by combining back propagation with straight-through estimation STE to update network parameters, the network parameters become more suitable for quantization to reduce quantization loss. This process is usually referred to as fine-tuning.

[0058] S4.1, determine the number of groups of data within the first tensor based on the bit width of the input activation, to obtain a plurality of grouped data;

[0059] S4.2, traverse the maximum value of the grouped data, and perform the highest bit value taking of the maximum value of the grouped data based on the logarithm function with base 2 and the floor function to obtain the value of the highest bit;

[0060] Specifically, a for loop is used to traverse the grouped data to find the maximum value, and a small value (1e-5) is added to the maximum value to prevent the input value of the logarithm function from being zero during subsequent operations. The logarithm function with base 2 is used to obtain the highest bit (Most Significant Bit, MSB) of the maximum value using the floor function to take the integer part,

[0061] S4.3, perform precision division on the value of the highest bit based on the power function of 2 to obtain a quantization parameter;

[0062] Specifically, the precision division has the following expression:

[0063]

[0064]

[0065] where msb represents the value of the highest bit, div 4bit represents the quantization parameter of 4 bits, div 2bit represents the quantization parameter of 2 bits.

[0066] In the precision division process, the torch.clamp function is used to ensure that the index is not less than zero to prevent negative indices, which are meaningless in quantization.

[0067] S4.4, quantizing the grouped data based on the quantization parameter to obtain a second tensor;

[0068] Specifically, the way to quantize the grouped data to 4 bits is to divide the data by the quantization parameter of 4 bits div_4bit and then take the integer part, and then multiply the quantization parameter of 4 bits div_4bit, and store the data calculated in the second tensor y_4bit. The way to quantize the grouped data to 2 bits is the same as that of 4 bits, except that the data calculated is stored in the second tensor y_2bt.

[0069] S4.5, performing bit sparsification on the second tensor to obtain quantized data.

[0070] Specifically, while quantizing the data to 4 bits and 2 bits, there is also bit sparsification, referring to Figure 4 , the msb of the input data 18 and 19 is 4, and the number obtained by dividing the data by 4 and taking the integer part is multiplied by 4 to achieve bit-level sparsification of the low-bit data. When the data is divided and the integer part is taken, it is similar to right shift operation, which will sparsify the low-bit data, and the number multiplied back is a power of 2, similar to left shift operation, which will only fill the low-bit data with 0, which will further sparsify the weights. At the same time, in the AI accelerator, the number of low-bit 0 in a group will be counted during the final convolution calculation, and the data after removing the 0 values will be stored, and the shifted data will be shifted back after the convolution calculation.

[0071] The data after this grouping quantization will be stored in three forms in the storage unit, namely sign, weight and base. Each time the data is quantized to a low-bit width, the sign bit of each calculation unit will be saved in the sign array, and the number of 0 removed after a group of data will be marked as base, with a range of 0-7, stored in base. The weight stores the shifted data. During the accelerator calculation, the data is further compressed to a lower bit width through this storage method.

[0072] S5, absolute value accumulation is performed on the grouped quantized data, and an error between the absolute value accumulation of the original data and the absolute value accumulation of the grouped quantized data is calculated;

[0073] Specifically, by accumulating the absolute values of the data quantized into 4 bits and 2 bits respectively, and then calculating the error between the absolute value accumulation of the original data and the quantized data, this measurement method more comprehensively reflects the difference between the quantized data and the original data, providing more accurate guidance for searching for the optimal weight distribution.

[0074] S6, setting the hyperparameter space, and adjusting the hyperparameter combination based on the grid search algorithm to obtain different hyperparameter combinations;

[0075] Specifically, referring to Figure 3 The threshold of quantization error and other related parameters are clearly defined in the hyperparameter space. The selection of these hyperparameters directly affects the quantization effect and model performance. Through the careful design of the hyperparameter space, we can more targetedly explore the best hyperparameter combination and improve the search efficiency. In the process of grid search, we generate an array of 0.1 to 0.9 with a step size of 0.05 to represent the threshold of quantization error. Through nested for loops, different hyperparameter combinations can be generated flexibly.

[0076] S7, based on different hyperparameter combinations and the error of the absolute value accumulation, the grouped quantized data is screened to obtain grouped quantized data with different bit weight distributions;

[0077] Specifically, the error of the absolute value accumulation and the threshold of quantization error of different hyperparameter combinations are compared. If the error of the absolute value accumulation is greater than the threshold of quantization error, the original data is retained; if the error of the absolute value accumulation is less than the threshold of quantization error, the quantized data is retained. Through screening, grouped quantized data with different bit weight distributions can be obtained.

[0078] S8, based on the grouped quantized data with different bit weight distributions, the pre-trained model is trained and the loss function is minimized to obtain the optimal bit weight distribution and its corresponding grouped quantized data.

[0079] Specifically, under each group of hyperparameter combinations, the model is trained and the loss is calculated. Through the minimization of the loss function, the optimal 2 / 4 / 8 bit weight distribution under different error conditions can be found. This loss-based evaluation method more objectively reflects the pros and cons of the quantization effect. The grid search method is systematic and comprehensive, and by establishing a discrete grid in the pre-defined hyperparameter space, various possible hyperparameter combinations are systematically traversed. This ensures that we are not limited to local search, but can find the global optimal solution.

[0080] The quantized weights obtained by the quantization method are configured to the convolution calculation unit of the corresponding AI accelerator through hardware design, and are practically applied on the FPGA development board, and the application provides a flexible calculation unit for providing mixed precision calculation, and based on the lowest 2bit*2bit calculation unit, nine kinds of mixed precision calculation of 2 / 4 / 8 of the activation and weight can be realized through addition and shift. Figure 6 As shown in the following 4*2(1011*10) multiplication can be decomposed into two 2*2 calculations, i.e. Figure 7 Referring to , 1011 is decomposed into 10 and 11, and 10 is multiplied to obtain two results, i.e. , the result is 6 obtained by left shifting 0 bits and the result is 16 obtained by left shifting 2 bits, and then the two results are added to obtain 6+16=22, which is consistent with the original 4*2 multiplication result. By using low-bit-width calculation to replace high-bit-width multiplication and accumulation operation, the use of calculation resources is significantly reduced.

[0081] At the same time, the lowest calculation unit of 2bit multiplication unit can be freely combined into 2 / 4 / 8 mixed calculation array through flexible shift such as (0, 2, 4, 6), as shown in Figure 8 The input data of the input buffer (IBUF) and the weight data of the weight buffer (WBUF) are split and reorganized, and are quantized into lower bit-width data through mixed precision, and then the high-bit-width calculation is replaced by low-bit-width calculation to greatly improve the calculation efficiency.

[0082] At the same time, in order to aim at the less 2bit data of some layers, the lowest 4bit calculation unit is used, and the 8 / 4 mixed mixed precision calculation can also be realized, as shown in Figure 9 That is, the weight and activation data of each input will be split into 4bit, and mixed precision calculation is realized.

[0083] The above is a specific description of the preferred embodiment of the application, but the application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the application.

Claims

1. A deep learning-based AI accelerator quantization method, characterized by, The AI accelerator quantization method is applied to the image classification field, and includes the following steps: obtaining a pre-training model of a specific task to obtain parameters of the pre-training model; limiting and mapping the input activation value of the weight to obtain the bit width of the input activation; preprocessing the floating-point data of the pre-training model to obtain a first tensor; grouping and quantizing the first tensor based on the bit width of the input activation to obtain grouped and quantized data; performing absolute value accumulation on the grouped and quantized data, and calculating the error between the absolute value accumulation value of the original data and the absolute value accumulation value of the grouped and quantized data; setting a hyperparameter space and adjusting the hyperparameter combination based on a grid search method to obtain different hyperparameter combinations; screening the grouped and quantized data based on the different hyperparameter combinations and the error of the absolute value accumulation value to obtain grouped and quantized data with different bit weight value distributions; training the pre-training model based on the grouped and quantized data with different bit weight value distributions and minimizing the loss function to obtain the optimal bit weight value distribution and the corresponding grouped and quantized data. 2.The AI accelerator quantization method based on deep learning of claim 1, wherein, The step of limiting and mapping the input activation value of the weight to obtain the bit width of the input activation specifically includes: limiting and mapping the activation value of the weight and obtaining the integer representation of the activation value by rounding; quantizing the integer representation of the activation value to obtain the bit width of the input activation.

3. The AI ​​accelerator quantization method based on deep learning according to claim 1, characterized in that: The step of preprocessing the floating-point data of the pre-training model to obtain a first tensor specifically includes: mapping the floating-point data based on the hyperbolic tangent function to obtain a mapped tensor; normalizing the mapped tensor based on the maximum absolute value of the floating-point data to obtain the first tensor. 4.The AI accelerator quantization method based on deep learning of claim 1, wherein, The step of grouping and quantizing the first tensor based on the bit width of the input activation to obtain grouped and quantized data specifically includes: determining the number of groups of data within the first tensor based on the bit width of the input activation to obtain a plurality of grouped data; traversing the maximum value of the grouped data and obtaining the value of the highest bit of the maximum value of the grouped data based on the logarithm function with base 2 and the floor function to obtain the value of the highest bit; performing precision division on the value of the highest bit based on the power function of 2 to obtain a quantization parameter; quantizing the grouped data based on the quantization parameter to obtain a second tensor; performing bit sparsification on the second tensor to obtain quantized data. 5.The AI accelerator quantization method based on deep learning of claim 1, wherein, The step of screening the grouped and quantized data based on the different hyperparameter combinations and the error of the absolute value accumulation value to obtain grouped and quantized data with different bit weight value distributions specifically includes: comparing the error of the absolute value accumulation value and the threshold of the quantization error of the different hyperparameter combinations to obtain a comparison result; screening the grouped and quantized data based on the comparison result to obtain grouped and quantized data with different bit weight value distributions.

6. A hardware design method using the AI accelerator quantization method based on deep learning according to any one of claims 1-5, characterized in that, freely combining the lowest calculation units through addition and shifting to obtain a hybrid calculation array to realize hardware calculation of the grouped and quantized data with the optimal bit weight value distribution.

Images