A hardware device for an AI accelerator based on deep learning

Abstract

The application discloses an AI accelerator hybrid quantization method and hardware design based on deep learning, and the method comprises the following steps: determining an initial base two-dimensional tensor according to an experience value, a size parameter of a convolution kernel and a bit number required for quantization; setting a threshold two-dimensional tensor based on the initial base two-dimensional tensor; extracting a sign value of the original weight and saving the sign value in a first three-dimensional tensor; and obtaining a second three-dimensional tensor based on the threshold two-dimensional tensor and a quantized weight value. The hardware design comprises a convolution calculation unit, a processing array unit, a multiplication and accumulation module and an addition tree module. By using the application, hybrid quantization of different layer parameters in a network can be realized, network parameters with different influence degrees are reasonably quantized into different bits, and the performance and accuracy of a light-weight network are improved. The application can be widely applied to the chip technical field.

Description

TECHNICAL FIELD

[0001] The application relates to the chip technical field, in particular to an AI accelerator hybrid quantization method based on deep learning and a hardware design. BACKGROUND

[0002] With the development of deep learning, various high-performance and deep-level convolutional neural networks emerge, and are widely used in semantic segmentation, image recognition and target detection fields. However, high performance brings high computing and memory costs. The edge deployment of convolutional neural networks is currently a popular trend in deep learning. However, the resources of embedded devices on the edge are limited, and there are high requirements for energy consumption and real-time performance. It is not realistic to directly deploy a large convolutional neural network. Under the premise of ensuring the accuracy of the network model, the model needs to be compressed as much as possible to reduce the size of the model and the storage space of the network parameters, so as to make the neural network model lightweight and improve the application effect of the embedded device.

[0003] There are many methods to realize the lightweight of the model, including pruning, quantization, low-rank decomposition and knowledge distillation. Among them, quantization can reduce the dimension of weights and activations in the neural network by quantizing the parameters represented by a large number of bits to a smaller number of bits. In recent years, with the development of model lightweight, various quantization methods have been proposed, which can be combined with hardware optimization to effectively reduce the consumption of hardware multiplier resources and adder resources. In 2011, researchers successfully represented the network parameters in the neural network with 32-bit floating-point numbers with 8-bit fixed-point numbers, successfully reducing the storage demand of the network to one fourth, achieving the goal of model compression. Later, a binary quantization method was developed, which quantized weights and activations to 1 and -1. Although this method can greatly reduce storage overhead and computational complexity, it will cause a great loss of accuracy. Later, a ternary quantization method was derived, which compressed the neural network parameters to -1, 0 and 1. Compared with the binary network, the ternary quantization method has higher accuracy in some scenarios.

[0004] However, these quantization methods will affect the accuracy of the network to varying degrees. Because the importance of weight parameters in different layers of a convolutional neural network is different, quantizing them to the same bit fixed-point number will cause a decline in network accuracy and performance, thereby affecting the real-time performance and accuracy of edge deployment. SUMMARY

[0005] In order to solve the above technical problems, the purpose of the present application is to provide a deep learning-based AI accelerator hybrid quantization method and hardware design, which realizes the hybrid quantization of different layer parameters in a network, reasonably quantizes network parameters with different influence degrees to different bits, and improves the performance and accuracy of the lightweight network.

[0006] The first technical solution adopted by the present application is: a deep learning-based AI accelerator hybrid quantization method, comprising the following steps:

[0007] Obtain the convolution kernel parameters of the neural network model according to the scene task;

[0008] Determine the initial base two-dimensional tensor according to the experience value, the size parameter of the convolution kernel and the number of bits to be quantized;

[0009] Determine all possible combinations of 0 and 1 according to the number of bits to be quantized, and obtain a first two-dimensional tensor;

[0010] Set the quantization threshold and save it in a second two-dimensional tensor;

[0011] Based on the matrix product of the first two-dimensional tensor and the initial base two-dimensional tensor, a third two-dimensional tensor is obtained;

[0012] Sort the third two-dimensional tensor to obtain a fourth two-dimensional tensor and a serial number tensor;

[0013] Based on the matrix multiplication of the fourth two-dimensional tensor and the second two-dimensional tensor, a fifth two-dimensional tensor is obtained;

[0014] Convert the original weight to be quantized to obtain a sixth two-dimensional tensor;

[0015] Extract the sign value of the original weight and save it in a first three-dimensional tensor;

[0016] Take the absolute value of the original weight, and select the quantized weight in the fourth two-dimensional tensor combined with all values of the fifth two-dimensional tensor, and save it to a seventh two-dimensional tensor;

[0017] Based on the selected quantized weight value, the corresponding position value in the first two-dimensional tensor is found to obtain a second three-dimensional tensor;

[0018] Reduce the dimension of the first three-dimensional tensor, and modify the weight sign of the seventh two-dimensional tensor based on the obtained eighth two-dimensional tensor to obtain a modified seventh two-dimensional tensor.

[0019] Further, the base two-dimensional tensor represents a set of bases shared by all weights of the convolution kernel; the first three-dimensional tensor represents the data of the sign bit; and the second three-dimensional tensor is a linear tensor unique to each weight data in a convolution kernel.

[0020] Further, the step of taking the absolute value of the original weight, and selecting the quantized weight in the fourth two-dimensional tensor combined with all values of the fifth two-dimensional tensor, and saving it to the seventh two-dimensional tensor, specifically includes:

[0021] Taking the absolute value of the quantized weight, sorting from small to large, and comparing with all values of the fifth two-dimensional tensor respectively to obtain a comparison result;

[0022] Based on the comparison result, the quantized weight value is selected at the corresponding position of the fourth two-dimensional tensor, and the quantized weight value is saved to the seventh two-dimensional tensor.

[0023] Further, the step of modifying the weight sign of the seventh two-dimensional tensor based on the obtained eighth two-dimensional tensor after reducing the dimension of the first three-dimensional tensor to obtain the modified seventh two-dimensional tensor, specifically includes:

[0024] The first three-dimensional tensor is reduced in dimension to obtain an eighth two-dimensional tensor;

[0025] The weight sign of the seventh two-dimensional tensor is modified based on the sign value in the eighth two-dimensional tensor to obtain a modified seventh two-dimensional tensor.

[0026] The second technical solution adopted by the application is: a hardware device of an AI accelerator based on deep learning, applied to the AI accelerator hybrid quantization method based on deep learning as described above, and the designed convolution calculation unit includes a plurality of processing array units, wherein:

[0027] The processing array unit includes a plurality of multiply-accumulate modules;

[0028] The multiply-accumulate module includes an addition tree module;

[0029] The addition tree module is used for cumulative operation on input activation and modified vector bits;

[0030] The multiply-accumulate module performs multiplication operation on the output of the addition tree module and the base data, and performs accumulation on the multiplication result, and outputs the intermediate value result of the multiplication and accumulation;

[0031] The processing array unit accumulates the intermediate value result output by the multiply-accumulate module until the convolution operation of one convolution kernel and the corresponding region of the input activation is completed;

[0032] The convolution calculation unit is used for simultaneously performing a plurality of convolution operations.

[0033] Further, the modified vector bit is obtained as follows:

[0034] The weight data is adjusted to obtain a calculation flag bit and a vector bit;

[0035] The sign bit data is used to adjust the positive and negative of the vector bit data, and the modified vector bit is obtained.

[0036] The method and system have the following advantages: the weight parameters are quantified from floating-point numbers to fixed-point numbers by using the weight quantization algorithm, and each weight data can be quantized to different bits according to the size of the weight data, thereby greatly reducing the required storage space of the weight data; all weight data in each layer of the convolutional neural network is quantized to different bits by quantizing the weight data into a base two-dimensional tensor and a second three-dimensional tensor, thereby greatly reducing the required storage space of the weight parameters while the loss of the accuracy of the network is small; the multiplication-accumulation module is designed, thereby improving the calculation efficiency and performance of the hardware accelerator; the convolution calculation unit designed by the parallel processing array unit can simultaneously perform multiple convolution operations, thereby improving the calculation efficiency of the entire convolution calculation unit. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a step flow chart of an AI accelerator hybrid quantization method based on deep learning of the application;

[0038] Figure 2 is a weight data diagram of the hardware design of an AI accelerator based on deep learning of the application;

[0039] Figure 3 is a multiplication-accumulation module schematic diagram of the hardware design of an AI accelerator based on deep learning of the application;

[0040] Figure 4 is a multiplication-accumulation operation schematic diagram of the hardware design of an AI accelerator based on deep learning of the application;

[0041] Figure 5 is a complete flow schematic diagram of the AI accelerator hybrid quantization method and hardware design based on deep learning of the application. DETAILED DESCRIPTION

[0042] The application will be further described in detail below with reference to the drawings and specific embodiments. For the step numbers in the following embodiments, only the setting is for the convenience of description and explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0043] Referring to Figure 1 The application provides an AI accelerator hybrid quantization method based on deep learning, which comprises the following steps:

[0044] The embodiment of the application is implemented to deploy a target detection model on an edge end, selects YOLOv7, EfficientDet and R-CNN as a framework of the target detection model, trains the target detection model through training set data, obtains a trained target detection model, acquires convolution kernel parameters and weight parameters of the trained target detection model, and then quantizes the extracted convolution kernel parameters and weight parameters. The essence of a weight quantization algorithm is to quantize weight data into an inner product of a basis two-dimensional tensor Basis and a second three-dimensional tensor V according to a bit number of quantization, wherein the basis two-dimensional tensor Basis is composed of multiple fixed-point numbers with a bit width of 16 bits, and the second three-dimensional tensor V is a vector composed of 0 and 1, and the weight parameters between all layers and within a single layer of a convolutional neural network can be quantized arbitrarily.

[0045] Firstly, an initial basis two-dimensional tensor Basis is determined according to an experience value, a size parameter of a convolution kernel and a bit number to be quantized; specifically, if 32-bit floating-point numbers are to be quantized into 8-bit fixed-point integers, the 32-bit floating-point numbers are expressed by K basis vectors, that is:

[0046]

[0047] wherein, represents a fixed-point number with a bit width of 16.

[0048] Then, all possible combinations of 0 and 1 are determined according to the bit number to be quantized, to obtain a first two-dimensional tensor Quan_Mult, which has a size of [out_channels,bits]; Specifically, if the bit number to be quantized is 4 bits, there are possible combinations of 0 and 1.

[0049] Then, a threshold value of quantization is set and saved in a second two-dimensional tensor Threshold, which has a size of [out_channels,bits];

[0050] Specifically, the threshold value adopted in the embodiment is 0.5, and when bits=2, the second two-dimensional tensor Threshold is specifically as follows:

[0051]

[0052] Then, based on a matrix product of the first two-dimensional tensor Quan_Mult and the initial basis two-dimensional tensor Basis, a third two-dimensional tensor Level is obtained, which has a size of [out_channels,bits], wherein out_channels is a channel number of output activation of a layer of a convolutional neural network, that is, a number of convolution kernels.

[0053] ​Then sort the third two-dimensional tensor Level in ascending order to obtain an ordered fourth two-dimensional tensor Level_New and a serial number tensor Ser;

[0054] Then perform matrix multiplication based on the fourth two-dimensional tensor Level_New and the second two-dimensional tensor Threshold to obtain a fifth two-dimensional tensor Level_Threshold, which has a size of [out_channels, Each value in the fifth two-dimensional tensor Level_Threshold is the middle value of two adjacent values in the fourth two-dimensional tensor Level_New.

[0055] Then convert the original weights that need to be quantized to obtain a sixth two-dimensional tensor Init_Weights, which has a size of Where k is the size of the convolution kernel and in_channels is the number of channels of the convolution kernel.

[0056] Then extract the sign value of each original weight and save it in a first three-dimensional tensor Weights_Sign, which has a size of The saving rule is as follows:

[0057] When the original weight value of the sixth two-dimensional tensor Init_Weights is negative, the saved sign value is 1; when the original weight value of the sixth two-dimensional tensor Init_Weights is positive, the saved sign value is 0.

[0058] Then take the absolute value of the original weight, i.e. the absolute value of the sixth two-dimensional tensor Init_Weights, to obtain an absolute value Weights. Arrange the absolute value Weights in ascending order from smallest to largest, and simultaneously traverse all values of the fifth two-dimensional tensor Level_Threshold. Compare the absolute value Weights with all values of the fifth two-dimensional tensor Level_Threshold. When the value of the absolute value Weights is greater than the value at the corresponding position of the fifth two-dimensional tensor Level_Threshold, it indicates that the quantized weight is greater than the quantization threshold, and the corresponding quantized weight value is the value on the right side of two adjacent values in the fourth two-dimensional tensor Level_New, i.e. the larger value. When the value of the absolute value Weights is less than the value at the corresponding position of the fifth two-dimensional tensor Level_Threshold, it indicates that the quantized weight is less than the quantization threshold, and the corresponding quantized weight value is the value on the left side of two adjacent values in the fourth two-dimensional tensor Level_New, i.e. the smaller value. Then save the quantized weight to a seventh two-dimensional tensor Quan_Weights, which has a size of The specific evolution example is as follows:

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065] wherein, , , are randomly selected parameters.

[0066] Then according to the value of the fourth two-dimensional tensor Level_New taken by the quantized weight in the seventh two-dimensional tensor Quan_Weights, the value of the corresponding position in the first two-dimensional tensor Quan_Mult is found and saved to the second three-dimensional tensor V, and the size is .

[0067] Then the first three-dimensional tensor Weights_Sign is reduced in dimension to obtain the eighth two-dimensional tensor Reshape_Weights_Sign, and then the eighth two-dimensional tensor Reshape_Weights_Sign and the seventh two-dimensional tensor , are compared. When the value corresponding to the eighth two-dimensional tensor Reshape_Weights_Sign is 1, the quantized weight at the corresponding position of the seventh two-dimensional tensor is adjusted to a negative number; when the value corresponding to the eighth two-dimensional tensor Reshape_Weights_Sign is 0, the quantized weight y at the corresponding position of the seventh two-dimensional tensor remains a positive number, obtaining the modified seventh two-dimensional tensor.

[0068] According to the above steps, the original weight is quantized to the corresponding bit width, and the two tensors Basis and V representing the quantized weight and the sign bit tensor Weights_Sign required for hardware calculation are obtained, the quantized data is implemented in hardware, the corresponding target detection function is realized, the quantized Basis, V and sign bit tensor Weights_Sign can be flexibly applied to various network structures and optimization of specific networks, this quantization method will not bring excessive precision loss and greatly reduce the required memory space for weight parameter storage, then according to the quantized weight data format, the multiply-accumulate calculation module of the AI accelerator is designed, which can realize efficient convolution calculation on hardware with effective resources, improve the performance and efficiency of running convolutional neural network on the accelerator. Through the quantization method of the application, the weight parameters of the network can be quantized to any bit from 1 to 8, effectively reducing the size of the model and the storage space requirement of the network parameters, then according to the quantized data format, the multiply-accumulate module is designed on hardware to realize efficient and low-resource-consumption accelerator convolution operation.

[0069] The application provides a hardware device of an AI accelerator based on deep learning, and a convolution calculation unit is designed to include a plurality of processing array units, wherein:

[0070] The processing array unit includes a plurality of multiply-accumulate modules;

[0071] The multiply-accumulate module includes an addition tree module;

[0072] The addition tree module is used for accumulating the input activation and the modified vector bit;

[0073] The multiply-accumulate module performs multiplication operation on the output of the addition tree module and the basis data, and accumulates the multiplication result to output the intermediate value result of the multiply-accumulate;

[0074] The processing array unit accumulates the intermediate value result output by the multiply-accumulate module until the convolution operation of one convolution kernel and the input activation in the corresponding region is completed;

[0075] The convolution calculation unit is used for simultaneously performing a plurality of convolution operations.

[0076] The modified vector bit is obtained in the following way:

[0077] The weight data is adjusted to obtain a calculation flag bit and a vector bit;

[0078] The data of the vector bit is adjusted based on the sign bit data to obtain the modified vector bit.

[0079] For the weight data format quantized by the weight quantization algorithm, a reasonable convolution calculation unit is designed to complete the efficient mixed-precision convolution operation. In the above quantization process, after the original 32-bit floating-point original weight is quantized by the quantization algorithm, three data will be obtained. The first is the data representing the sign bit, Weights_Sign, wherein 1 represents that the corresponding weight is negative, and 0 represents that the corresponding weight is positive. The second is a set of bases Basis shared by all weights of a convolution kernel, and Basis contains 8 values, each of which has a bit width of 16 bits. The third is a linear vector V unique to each weight data in the convolution kernel, which is composed of 0 and 1, and each V contains 8 0s or 1s.

[0080] With reference to Figure 2 and Figure 5 First, the weight data is adjusted. In order to more efficiently perform convolution calculation, hardware calculates the multiplication and accumulation operation of 8 channel input activations and weight data at a time. First, the V corresponding to the 8 weight data is processed. From the high bit, it is counted whether the 8 V has all 0 data. When the highest bit of the 8 V is 0 and the remaining bits have at least one 1, it is counted that the 8 V needs to be calculated 7 times. The data 7 is stored in the calculation flag bit Flag. Then the lowest bit of the 8 V is stored in the vector bit Vector0, and the process is repeated until the 7th bit of the 8 V is stored in the vector bit Vector6. When the highest bit and the second highest bit of the 8 V are 0 and the remaining bits have at least one 1, it is counted that the 8 V needs to be calculated 6 times. The data 8 is stored in the calculation flag bit Flag. Then the lowest bit of the 8 V is stored in the vector bit Vector0, and the process is repeated until the 6th bit of the 8 V is stored in the vector bit Vector6. The calculation flag bit Flag and the vector bit Vector of the weight V of all 8 channels are counted in turn.

[0081] Then, the adjusted weight data (calculation flag bit Flag and vector bit Vector) is preprocessed. According to the size of the calculation flag bit Flag, it is determined how many data in each Basis need to be used. When the calculation flag bit Flag is 4, the data in the low 4 bits of Basis will be taken out for calculation. Then the vector bit Vector is placed in the cache module. Through the size of the calculation flag bit Flag, the vector bit Vector is sequentially transmitted into the multiplication and accumulation module and the 8-channel input activation for accumulation calculation. In addition, according to the sign bit Weights_Sign corresponding to the weight data of the 8 channels, it is determined whether the data 1 in the vector bit Vector is positive or negative.

[0082] With reference to Figure 3The 8-channel input activation and 8 data of each vector bit Vector are subjected to AND operation, and then the results after AND are accumulated to obtain an addition tree module; the results output by the addition tree are multiplied by the data of Basis once, and the product results are accumulated, and finally the intermediate value results of 8-channel weight and activation multiplication and accumulation are output to obtain a multiplication and accumulation module (MAC); then the accumulation of the intermediate values is performed inside the multiplication and accumulation module until the convolution operation of one convolution kernel and the input activation inside the corresponding region is completed, and a complete output activation result is output to obtain a processing array unit (PE). Therefore, one multiplication and accumulation module can complete one complete convolution operation and output the result in different clock cycles according to the size of the calculation flag bit Flag.

[0083] With reference to Figure 4 In order to more efficiently calculate, the designed convolution calculation unit contains 16 processing array units (PE), and each processing array unit has 10 multiplication and accumulation modules (MAC) inside, therefore, one convolution calculation unit contains 160 multiplication and accumulation modules MAC, and multiple convolution operations can be simultaneously performed at a time to obtain 160 output activation results.

[0084] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A hardware device of an AI accelerator based on deep learning, characterized in that: the weight quantization method comprises the following steps: obtaining the convolution kernel parameters of the neural network model according to the scene task; determining the initial base two-dimensional tensor according to the experience value, the size parameter of the convolution kernel and the number of bits to be quantized; determining all possible combinations of 0 and 1 according to the number of bits to be quantized to obtain a first two-dimensional tensor; setting the quantization threshold and saving it in a second two-dimensional tensor; obtaining a third two-dimensional tensor based on the matrix product of the first two-dimensional tensor and the initial base two-dimensional tensor; sorting the third two-dimensional tensor to obtain a fourth two-dimensional tensor and a serial number tensor; obtaining a fifth two-dimensional tensor based on the matrix multiplication of the fourth two-dimensional tensor and the second two-dimensional tensor; converting the original weight to be quantized to obtain a sixth two-dimensional tensor; extracting the sign value of the original weight and saving it in a first three-dimensional tensor; taking the absolute value of the original weight, combining all values of the fifth two-dimensional tensor and selecting the quantized weight in the fourth two-dimensional tensor to save it to a seventh two-dimensional tensor; finding the value of the corresponding position in the first two-dimensional tensor based on the selected quantized weight value to obtain a second three-dimensional tensor; reducing the dimension of the first three-dimensional tensor and modifying the weight sign of the seventh two-dimensional tensor based on the obtained eighth two-dimensional tensor to obtain a modified seventh two-dimensional tensor; designing a convolution calculation unit for the weight data format quantized by the weight quantization method, which includes a plurality of processing array units, wherein: the processing array unit includes a plurality of multiply-accumulate modules; the multiply-accumulate module includes an addition tree module; the addition tree module is used for cumulative operation on the input activation and the modified vector bit; the multiply-accumulate module performs multiplication operation on the output of the addition tree module and the base data, and performs accumulation on the multiplication result to output the intermediate value result of the multiply-accumulate; the processing array unit accumulates the intermediate value result output by the multiply-accumulate module until the convolution operation of one convolution kernel and the input activation in the corresponding region is completed; the convolution calculation unit is used for simultaneously performing multiple convolution operations.

2. The hardware apparatus of an AI accelerator based on deep learning according to claim 1, wherein, The base two-dimensional tensor represents a set of bases shared by all weights of the convolution kernel; the first three-dimensional tensor represents the data of the sign bit; and the second three-dimensional tensor is a linear tensor unique to each weight data in the convolution kernel.

3. The hardware apparatus of an AI accelerator based on deep learning according to claim 1, wherein, The step of taking the absolute value of the original weight, combining all values of the fifth two-dimensional tensor and selecting the quantized weight in the fourth two-dimensional tensor to save it to the seventh two-dimensional tensor specifically includes: taking the absolute value of the quantized weight, sorting from small to large and comparing with all values of the fifth two-dimensional tensor to obtain a comparison result; selecting the quantized weight value in the corresponding position of the fourth two-dimensional tensor based on the comparison result and saving the quantized weight value to the seventh two-dimensional tensor.

4. The hardware apparatus of an AI accelerator based on deep learning according to claim 1, wherein, The step of reducing the dimension of the first three-dimensional tensor and modifying the weight sign of the seventh two-dimensional tensor based on the obtained eighth two-dimensional tensor to obtain a modified seventh two-dimensional tensor specifically includes: reducing the dimension of the first three-dimensional tensor to obtain an eighth two-dimensional tensor; The weight sign of the seventh two-dimensional tensor is modified based on the sign value in the eighth two-dimensional tensor, to obtain a modified seventh two-dimensional tensor.

5. The hardware apparatus of an AI accelerator based on deep learning according to claim 1, wherein, The modified vector bit is obtained in the following way: The weight data is adjusted to obtain a calculation flag bit and a vector bit; The data positive and negative of the vector bit is adjusted based on the sign bit data, to obtain a modified vector bit.

Images