Neural network accelerator, acceleration implementation method and device thereof and storage medium

Abstract

The invention discloses an acceleration implementation method and device of a neural network accelerator, the neural network accelerator, a storage medium and a computer program product, the neural network accelerator comprises an FPGA, a neural network model is deployed on the FPGA, and the neural network model comprises a convolutional layer; the method comprises the following steps: quantizing weight data to reduce the bit width of the weight data; according to the number of the convolution windows of the convolution layer, the number of the input channels, the number of the output channels and the number of the DSP units of the FPGA, determining a parallel mode of the convolution layer; and performing reasoning calculation by using the quantized weight data and the determined parallel mode of the convolutional layer. According to the scheme, by quantifying the weight data and determining the parallel mode of the convolutional layer, consumption of FPGA on-chip resources is reduced, the reasoning speed is increased, the on-chip resources are fully utilized, and the parallel performance is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of neural network accelerators, and specifically relates to a neural network accelerator and an acceleration implementation method, device, storage medium, and computer program product thereof. Background Art

[0002] FPGA stands for Field Programmable Gate Array, which is a hardware accelerator that can be customized for various neural networks. It consists of programmable logic elements, programmable connections, on-chip memory, clock management resources, input / output (I / O) resources, configuration memory, etc. Neural networks are a computing model designed based on human biology and are widely used in the field of artificial intelligence. With the development of deep learning, there is an increasing demand for the deployment of neural networks at the edge and for inference acceleration.

[0003] There are three common ways to deploy hardware acceleration at the edge: graphics inference accelerator (GPU), application-specific integrated circuit (ASIC), and field-programmable gate array (FPGA). Among them, GPU has large-scale parallel acceleration capabilities, but has the disadvantages of high power consumption and high cost, making it difficult to deploy in embedded devices. ASIC is designed for specific tasks. Different ASICs may be required for different tasks, and its design and manufacturing require a lot of money and time, which makes it less flexible in small-scale applications. The advantages of FPGA include: (1) programmability, which can be programmed and configured according to different needs; (2) parallelism, the structure of FPGA runs parallel processing, and there are a large number of structures in neural networks that can be processed in parallel; (3) low energy consumption, FPGA can be used for hardware acceleration in embedded devices with energy consumption constraints. Therefore, FPGA is more suitable for hardware acceleration in embedded devices.

[0004] However, since the on-chip resources of FPGA are not abundant, the calculation speed is not high and the acceleration performance is low during the hardware acceleration process, and the parallelism of FPGA cannot be fully utilized.

[0005] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The purpose of the present invention is to provide an acceleration implementation method for a neural network accelerator to solve the problem of tight resources on the FPGA chip in related solutions. In the process of reasoning acceleration, there are problems such as low computing speed, low acceleration performance, and inability to fully utilize the parallelism of the FPGA. The method can reduce the consumption of FPGA on-chip resources by quantizing weight data and determining the parallel mode of the convolution layer, speed up the reasoning speed, make full use of on-chip resources, and improve the effect of parallel performance.

[0007] The present invention provides an acceleration implementation method for a neural network accelerator, the neural network accelerator comprising an FPGA; a neural network model is deployed on the FPGA; the neural network model comprises a convolution layer; the method comprises: obtaining weight data of the neural network model, the number of convolution windows of the convolution layer, the number of input channels of the convolution layer, the number of output channels of the convolution layer, and the number of DSP units of the FPGA; quantizing the weight data to reduce the bit width of the weight data; determining the parallel mode of the convolution layer according to the number of convolution windows, the number of input channels, the number of output channels, and the number of DSP units of the FPGA; and performing inference calculation using the quantized weight data and the determined parallel mode of the convolution layer.

[0008] In some embodiments, the neural network model also includes a fully connected layer; the weight data includes the weight data of the convolutional layer and the bit width data of the fully connected layer; the weight data is quantized to reduce the bit width of the weight data, including: obtaining the weight range of the weight data of the convolutional layer and the weight range of the weight data of the fully connected layer; quantizing the bit width of the layer with the smallest weight range among the weight data of the convolutional layer and the weight data of the fully connected layer to the first bit width, and quantizing the bit width of other layers to the second bit width; the precision of the first bit width is less than the precision of the second bit width.

[0009] In some embodiments, it also includes: using a preset dynamic quantization technology to quantize the bit width of the layer with the smallest weight range in the weight data of the convolutional layer and the weight data of the fully connected layer to the first bit width, and quantize the bit widths of other layers to the second bit width.

[0010] In some embodiments, the parallel mode of the convolution layer includes: convolution kernel parallel or convolution kernel serial, input channel parallel or input channel serial, output channel parallel or output channel serial; the preset parallel mode of the convolution layer is: convolution kernel parallel, input channel parallel, output channel parallel; according to the number of the convolution windows, the number of the input channels, the number of the output channels, and the number of DSP units of the FPGA, the parallel mode of the convolution layer is determined, including: recording the product of the number of the convolution windows, the number of the input channels, and the number of the output channels as the DSP unit consumption; judging the relationship between the DSP unit consumption and the number of DSP units of the FPGA; if the DSP unit consumption is less than or equal to the number of DSP units of the FPGA, determining that the parallel mode of the convolution layer is the preset parallel mode of the convolution layer; if the DSP unit consumption is less than or equal to the number of DSP units of the FPGA, determining that the parallel mode of the convolution layer is the preset parallel mode of the convolution layer; If the consumption is greater than the number of DSP units of the FPGA, the number of the convolution windows, the number of input channels, and the number of output channels are replaced with 1 respectively, and the new DSP unit consumption is calculated until the new DSP unit consumption is less than or equal to the number of DSP units of the FPGA; after the new DSP unit consumption is less than or equal to the number of DSP units of the FPGA, if the number of convolution windows is 1, the convolution kernel parallel in the preset parallel mode of the convolution layer is adjusted to the convolution kernel serial; if the number of input channels is 1, the input channel parallel in the preset parallel mode of the convolution layer is adjusted to the input channel serial; if the number of output channels is 1, the output channel parallel in the preset parallel mode of the convolution layer is adjusted to the output channel serial, and then the parallel mode of the convolution layer is determined to be the preset parallel mode of the convolution layer.

[0011] In some embodiments, the convolutional layer in the neural network model is at least two layers; the method also includes: adjusting the parallel mode of the convolutional layer so that the sum of the DSP unit consumption of at least two layers of the convolutional layer is less than or equal to the number of DSP units of the FPGA.

[0012] Matching the above method, another aspect of the present invention provides an acceleration implementation device for a neural network accelerator, the neural network accelerator comprising an FPGA; a neural network model is deployed on the FPGA; the neural network model comprises a convolution layer; the device comprises: an acquisition unit, configured to acquire weight data of the neural network model, the number of convolution windows of the convolution layer, the number of input channels of the convolution layer, the number of output channels of the convolution layer, and the number of DSP units of the FPGA; a quantization unit, configured to quantize the weight data to reduce the bit width of the weight data; a convolution parallel planning unit, configured to determine the parallel mode of the convolution layer according to the number of convolution windows, the number of input channels, the number of output channels, and the number of DSP units of the FPGA; an inference calculation unit, configured to perform inference calculation using the quantized weight data and the determined parallel mode of the convolution layer.

[0013] In some embodiments, the weight data includes the weight data of the convolutional layer and the bit width data of the fully connected layer; the quantization unit quantizes the weight data to reduce the bit width of the weight data, including: obtaining the weight range of the weight data of the convolutional layer and the weight range of the weight data of the fully connected layer; quantizing the bit width of the layer with the smallest weight range among the weight data of the convolutional layer and the weight data of the fully connected layer to the first bit width, and quantizing the bit widths of other layers to the second bit width; the precision of the first bit width is less than the precision of the second bit width.

[0014] In some embodiments, the quantization unit is further configured to use a preset dynamic quantization technology to quantize the bit width of the layer with the smallest weight range in the weight data of the convolutional layer and the weight data of the fully connected layer to the first bit width, and quantize the bit widths of other layers to the second bit width.

[0015] In some embodiments, the parallel mode of the convolution layer includes: convolution kernel parallel or convolution kernel serial, input channel parallel or input channel serial, output channel parallel or output channel serial; the preset parallel mode of the convolution layer is: convolution kernel parallel, input channel parallel, output channel parallel; a convolution parallel planning unit determines the parallel mode of the convolution layer according to the number of the convolution windows, the number of the input channels, the number of the output channels, and the number of DSP units of the FPGA, including: recording the product of the number of the convolution windows, the number of the input channels, and the number of the output channels as the DSP unit consumption; judging the relationship between the DSP unit consumption and the number of DSP units of the FPGA; if the DSP unit consumption is less than or equal to the number of DSP units of the FPGA, determining that the parallel mode of the convolution layer is the preset parallel mode of the convolution layer; if the D If the SP unit consumption is greater than the number of DSP units of the FPGA, the number of the convolution windows, the number of input channels, and the number of output channels are replaced with 1 in sequence, and the new DSP unit consumption is calculated until the new DSP unit consumption is less than or equal to the number of DSP units of the FPGA; after the new DSP unit consumption is less than or equal to the number of DSP units of the FPGA, if the number of convolution windows is 1, the convolution kernel parallel in the preset parallel mode of the convolution layer is adjusted to the convolution kernel serial; if the number of input channels is 1, the input channel parallel in the preset parallel mode of the convolution layer is adjusted to the input channel serial; if the number of output channels is 1, the output channel parallel in the preset parallel mode of the convolution layer is adjusted to the output channel serial, and then the parallel mode of the convolution layer is determined to be the preset parallel mode of the convolution layer.

[0016] In some embodiments, the convolution layer in the neural network model is at least two layers; the convolution parallel planning unit is also configured to adjust the parallel mode of the convolution layer so that the sum of the DSP unit consumption of at least two layers of the convolution layer is less than or equal to the number of DSP units of the FPGA.

[0017] Matching the above-mentioned device, the present invention provides a neural network accelerator on the other hand, including: an acceleration implementation device of the neural network accelerator described above.

[0018] In accordance with the above method, the present invention provides a storage medium on the other hand, wherein the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the acceleration implementation method of the neural network accelerator described above.

[0019] In accordance with the above method, the present invention provides a computer program product on another aspect, wherein the computer program product includes a computer program, and when the computer program product is processed and executed, the steps of the acceleration implementation method of the above neural network accelerator are implemented.

[0020] The solution of the present invention quantizes the weights, determines the parallel mode of the convolution layer according to the number of convolution windows, the number of input channels, and the number of output channels of the convolution layer, and performs reasoning calculation using the quantized weight data and the determined parallel mode of the convolution layer. This reduces the consumption of FPGA on-chip resources, speeds up reasoning, makes full use of on-chip resources, and improves parallel performance.

[0021] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the present invention.

[0022] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of a flow chart of an embodiment of a method for implementing acceleration of a neural network accelerator of the present invention;

[0024] Figure 2 It is a structural schematic diagram of an embodiment of an acceleration implementation device of a neural network accelerator of the present invention;

[0025] Figure 3 is a hardware block diagram of a neural network accelerator of the present invention;

[0026] Figure 4 is the weight distribution diagram of convolutional layer 1;

[0027] Figure 5 It is the weight distribution diagram of convolutional layer 2;

[0028] Figure 6 is the weight distribution diagram of the fully connected layer 1;

[0029] Figure 7 It is the weight distribution diagram of the fully connected layer 2;

[0030] Figure 8 A flowchart of another embodiment of a method for implementing acceleration of a neural network accelerator;

[0031] Fig. 9 Schematic diagram of the process of quantizing weight data.

[0032] In conjunction with the accompanying drawings, the reference numerals in the embodiments of the present invention are as follows:

[0033] 101 - acquisition unit; 102 - quantization unit; 103 - convolution parallel planning unit; 104 - inference calculation unit. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0035] According to an embodiment of the present invention, a method for accelerating a neural network accelerator is provided, wherein the neural network accelerator includes an FPGA; a neural network model is deployed on the FPGA; and the neural network model includes a convolutional layer.

[0036] The hardware structure of the neural network accelerator is as follows: Figure 3 As shown in the figure, the accelerator in the figure is the Xilinx Zynq 7020 series. First, the trained weight data and image data are downloaded to the SD card on the PC side, and then transferred to the onboard DDR, and the data is controlled by ARM and transferred to the FPGA on the PL side for logical reasoning. The data interaction between the PS and PL sides is carried out through the AXI interface protocol for data transmission. Finally, the probability of the image is inferred and output to the Vitis SDK on the PC side for display. Among them, due to the limited resources on the FPGA chip, the neural network model selected in this scheme is Lenet-5, which includes 3 convolutional layers, 2 pooling layers, and 1 fully connected layer. The order of the network structure is input layer, convolutional layer 1, pooling layer 1, convolutional layer 2, pooling layer 2, convolutional layer 3, fully connected layer, and output layer.

[0037] In order to improve the inference calculation speed of the neural network accelerator and the parallelism of the FPGA, this solution quantizes the weight data on the one hand to reduce the bit width of the weight data, thereby reducing the complexity of the inference calculation and memory usage; on the other hand, it performs inference calculations on the data through a hybrid parallel method to reduce the resource consumption on the FPGA chip and improve the parallelism of the FPGA.

[0038] like Figure 1 The flowchart of an embodiment of the method of the present invention is shown in FIG. The acceleration implementation method of the neural network accelerator may include: steps S110 to S140.

[0039] At step S110, the weight data of the neural network model, the number of convolution windows of the convolution layer, the number of input channels of the convolution layer, the number of output channels of the convolution layer, and the number of DSP units of the FPGA are obtained.

[0040] In step S120, the weight data is quantized to reduce the bit width of the weight data.

[0041] The bit width of the trained weight data is FP32 type. Although the data has high precision, the amount of calculation and storage resource consumption are large. For example, the DSP consumption of FP32 type weight data is 4, the LUT consumption is 426, and the DSP consumption of INT8 type weight data is 2, and the LUT consumption is 258. Therefore, it is necessary to quantize the weight data to reduce the bit width, reduce the amount of calculation and resource occupation.

[0042] In some embodiments, the neural network model also includes a fully connected layer; the weight data includes the weight data of the convolutional layer and the bit width data of the fully connected layer. In step S120, the specific process of quantizing the weight data to reduce the bit width of the weight data includes: obtaining the weight range of the weight data of the convolutional layer and the weight range of the weight data of the fully connected layer; quantizing the bit width of the layer with the smallest weight range in the weight data of the convolutional layer and the weight data of the fully connected layer to the first bit width, such as INT4 type, and quantizing the bit width of other layers to the second bit width, such as INT8 type; the precision of the first bit width is less than the precision of the second bit width.

[0043] In the neural network model, the range of weight data in different layers is different. Take two convolutional layers and two fully connected layers as an example. Figures 4 to 7 As shown in the figure, the weight range of convolution layer 1 is [-0.5, 0.5], the weight range of convolution layer 2 is [-0.96, 0.5], the weight range of fully connected layer 1 is [-0.91, 0.8], and the weight range of fully connected layer 2 is [-1.11, 0.2]. When quantizing, if the weight range is large but a smaller precision is used for quantization, the accuracy of the data will be affected; if the weight range is small but a larger precision is used for quantization, the consumption of resources will be increased. Therefore, when quantizing the weight data, the dynamic quantization process is used to quantize the four-layer weights into different bit widths, thereby reducing the demand for model storage and calculation and improving the efficiency of the model in resource-constrained environments such as embedded devices. Specifically, the weight data with the smallest weight range is quantized to a smaller INT4 type, and the other weight data is quantized to a larger INT8 type. For example, if the weight range of convolution layer 1 is the smallest, then convolution layer 1 is quantized to INT4, and convolution layer 2, fully connected layer 1, and fully connected layer 2 are quantized to INT8.

[0044] In some embodiments, it also includes using a preset dynamic quantization technology to quantize the bit width of the layer with the smallest weight range in the weight data of the convolutional layer and the weight data of the fully connected layer to the first bit width, and quantize the bit widths of other layers to the second bit width.

[0045] The preset dynamic quantization technology can be the dynamic quantization technology in the Pytorch deep learning framework. The process of dynamic quantization is as follows: Fig. 9 As shown in the figure, dynamic quantization can re-evaluate the weight range of weight data and select appropriate quantization parameters every time the neural network model processes new input weight data. For example, after the weight data is input for the first time, the weight data of convolution layer 1 is quantized to INT4 type, and after the weight data is input for the second time, the weight data of convolution layer 1 is quantized to INT8 type. The process of dynamic quantization includes: determining the weight range, calculating the quantization parameters, and performing weight quantization. Compared with other quantization methods, the implementation process of dynamic quantization is simpler, and it can quickly realize quantization and model deployment, reduce computing resource consumption, and improve hardware acceleration effects.

[0046] At step S130, the parallel mode of the convolution layer is determined according to the number of the convolution windows, the number of the input channels, the number of the output channels, and the number of DSP units of the FPGA.

[0047] In some embodiments, the parallel mode of the convolution layer includes: parallel convolution kernels or serial convolution kernels, parallel input channels or serial input channels, and parallel output channels or serial output channels; the preset parallel mode of the convolution layer is: parallel convolution kernels, parallel input channels, and parallel output channels.

[0048] The convolution layer usually performs data calculations in four loops, including output channel loop, input channel loop, height loop, and width loop. These four loops are nested with each other to complete the convolution calculation of the convolution layer, extract the required features from the input data, and generate the output feature map.

[0049] The output channel loop is the outermost loop, which determines how many different output feature maps are generated. The feature map corresponding to each output channel is obtained by convolving a set of independent convolution kernels with the input data. In the calculation process of the entire convolution layer, it is first necessary to determine how many output channels are to be generated.

[0050] The input channel loop is the loop of the middle layer. For each output channel, all input channels are traversed, the data of each input channel is convolved with the corresponding convolution kernel channel part, and then these results are added as part of the convolution result of the output channel.

[0051] The width loop and height loop are the innermost loops. After considering the output channels and input channels, for each input and output channel combination, a convolution operation needs to be performed on the spatial dimensions (height and width) of the input data. The convolution kernel slides in the height and width directions of the input data, performing multiplication and accumulation operations element by element.

[0052] When the convolution layer calculates data, the outermost loop (output channel), the middle loop (input channel), and the innermost loop (convolution kernel) can all be calculated in parallel or serially. However, the amount of DSP consumed by parallel calculation is greater than that consumed by serial calculation. Therefore, it is necessary to use parallel calculation as much as possible to improve parallel performance when the amount of DSP consumed is not greater than the DSP amount of the hardware itself.

[0053] In the preset parallel mode of the convolution layer, the convolution layer consumes the highest amount of DSP. For example, in the case of parallel convolution kernels, parallel input channels, and parallel output channels, DSP1 = 25*1*16*6 = 2400, which is much larger than the number of DSPs in the hardware itself (220 DSP units), where 25, 1, 16, and 6 are the number of convolution windows, the number of input channels, the number of convolution groups, and the number of output channels, respectively. Therefore, based on this parallel mode, this solution uses serial computing to replace the parallel computing in the preset parallel mode, so that the amount of DSP consumed does not exceed the amount of DSP in the hardware itself, while ensuring parallel performance.

[0054] In some embodiments, in step S130, the specific process of determining the parallel mode of the convolution layer according to the number of the convolution windows, the number of the input channels, the number of the output channels, and the number of DSP units of the FPGA includes: steps S210 to S250.

[0055] Step S210: The product of the number of the convolution windows, the number of the input channels, and the number of the output channels is recorded as the DSP unit consumption.

[0056] When calculating the DSP unit consumption, it is assumed that one convolution window uses one DSP unit, that is, 1*the number of convolution windows*the number of input channels*the number of output channels=the DSP unit consumption.

[0057] Step S220, determining the relationship between the consumption of the DSP unit and the number of DSP units of the FPGA.

[0058] Step S230: If the DSP unit consumption is less than or equal to the number of DSP units of the FPGA, determine that the parallel mode of the convolution layer is a preset parallel mode of the convolution layer.

[0059] When determining the parallel method, if the preset parallel method of the convolution layer can ensure that the DSP consumption is ≤ the number of DSP units of the FPGA, the full parallel method is directly used to calculate the convolution layer.

[0060] Step S240, if the DSP unit consumption is greater than the number of DSP units of the FPGA, the number of the convolution windows, the number of the input channels, and the number of the output channels are replaced with 1 respectively, and the new DSP unit consumption is calculated until the new DSP unit consumption is less than or equal to the number of DSP units of the FPGA.

[0061] Step S250, after the new DSP unit consumption is less than or equal to the number of DSP units of the FPGA, if the number of convolution windows is 1, the convolution kernel parallel in the preset parallel mode of the convolution layer is adjusted to the convolution kernel serial; if the number of input channels is 1, the input channel parallel in the preset parallel mode of the convolution layer is adjusted to the input channel serial; if the number of output channels is 1, the output channel parallel in the preset parallel mode of the convolution layer is adjusted to the output channel serial, and then the parallel mode of the convolution layer is determined to be the preset parallel mode of the convolution layer.

[0062] When determining the parallel mode, if the DSP consumption is greater than the number of DSP units of the FPGA after adopting the preset parallel mode of the convolution layer, it is necessary to adjust the parallel mode to serial mode to reduce the DSP consumption. Optionally, the parallel mode of the preset convolution layer can be adjusted to serial mode in the order of convolution kernel, input channel, and output channel, or in other orders.

[0063] Taking the convolution kernel as an example, after adjusting the convolution kernel parallel to the convolution kernel serial, the number of convolution windows will be replaced by 1 when calculating the DSP consumption. For example, the total number of convolution windows is 16. Under the convolution kernel parallel, 16 convolution windows are calculated at the same time, and under the convolution kernel serial, only one convolution window is calculated at the same time. After replacing the number of convolution windows with 1, the DSP consumption will be reduced.

[0064] After adjusting the convolution kernel parallel to the convolution kernel serial, if the DSP consumption is ≤ the number of DSP units of the FPGA, the convolution kernel serial, input channel parallel, and output channel parallel mode are determined as the convolution kernel parallel mode; if the DSP consumption is greater than the number of DSP units of the FPGA, the input channel parallel is adjusted to the input channel serial to further reduce the DSP consumption; if the reduced DSP consumption is greater than the number of DSP units of the FPGA, the output channel parallel is adjusted to the output channel serial.

[0065] By adopting a hybrid parallel approach for the convolution kernel, input channel, and output channel, the DSP consumption does not exceed the hardware DSP amount, reducing the cache resources on the FPGA chip and improving the parallel capability of inference calculation.

[0066] In some implementations, the convolutional layers in the neural network model are at least two layers. The control method further includes: adjusting the parallel mode of the convolutional layers so that the sum of the DSP unit consumption of at least two layers of the convolutional layers is less than or equal to the number of DSP units of the FPGA.

[0067] In order to improve the utilization of hardware resources and computing efficiency, when the neural network model includes multiple convolutional layers, it is necessary to enable as many convolutional layers as possible to perform inference calculations simultaneously.

[0068] Taking three convolutional layers as an example, the DSP consumption of convolutional layer one = 1*16*1*16 = 96, where the order of 1, 16, 1, and 16 respectively indicates that one convolution window uses 1 DSP, 16 groups of convolution windows, the number of input channels is 1, and the number of output channels is 16, that is, the convolution kernels are parallel, the input channels are serial, and the output channels are parallel; the DSP consumption of convolutional layer two = 1*1*16*16 = 96, where the order of 1, 1, 16, and 16 respectively indicates that one convolution window uses 1 DSP, 1 group of convolution windows, the number of input channels is 16, and the number of output channels is 16, that is, the convolution kernels are serial, the input channels are parallel, and the output channels are parallel; the DSP consumption of convolutional layer three = 1*1*16*1 = 16, where the order of 1, 1, 16, and 1 respectively indicates that one convolution window uses 1 DSP, 1 group of convolution windows, the number of input channels is 16, and the number of output channels is 1, that is, the convolution kernels are serial, the input channels are parallel, and the output channels are serial. The total DSP consumption of the three convolutional layers is 208, which is less than the number of DSPs in the hardware itself (220 DSP units), so the three convolutional layers can perform inference calculations simultaneously.

[0069] At step S140, the quantized weight data and the determined convolutional layer are used in parallel to perform inference calculations to accelerate the inference on the FPGA. The inference calculations include calculations of the neural network model during training and inferences during prediction.

[0070] Figure 8 FIG. 1 is a flow chart of another embodiment of a method for implementing acceleration of a neural network accelerator. Figure 8 As shown, including:

[0071] Step 1: Input the trained model parameters into the neural network model, including image data and weight data, and dynamically quantize the type of weight data according to the parameter size of different layers.

[0072] Step 2: Evaluate the parallel mode based on the resources on the chip and configure the parallelism parameters.

[0073] Step 3: Design each module in the neural network model and write the driver software. After the inference calculation is completed, print the inference result.

[0074] By adopting the technical solution of this embodiment, the weights are quantized, the parallel mode of the convolution layer is determined according to the number of convolution windows, the number of input channels, and the number of output channels of the convolution layer, and the inference calculation is performed using the quantized weight data and the determined parallel mode of the convolution layer. This reduces the consumption of FPGA on-chip resources, speeds up the inference speed, makes full use of on-chip resources, and improves parallel performance.

[0075] According to an embodiment of the present invention, a neural network accelerator acceleration implementation device corresponding to the neural network accelerator acceleration implementation method is also provided. The neural network accelerator includes an FPGA; a neural network model is deployed on the FPGA; and the neural network model includes a convolution layer.

[0076] The hardware structure of the neural network accelerator is as follows: Figure 3 As shown in the figure, the accelerator in the figure is the Xilinx Zynq 7020 series. First, the trained weight data and image data are downloaded to the SD card on the PC side, and then transferred to the onboard DDR, and the data is controlled by ARM and transferred to the FPGA on the PL side for logical reasoning. The data interaction between the PS and PL sides is carried out through the AXI interface protocol for data transmission. Finally, the probability of the image is inferred and output to the Vitis SDK on the PC side for display. Among them, due to the limited resources on the FPGA chip, the neural network model selected in this scheme is Lenet-5, which includes 3 convolutional layers, 2 pooling layers, and 1 fully connected layer. The order of the network structure is input layer, convolutional layer 1, pooling layer 1, convolutional layer 2, pooling layer 2, convolutional layer 3, fully connected layer, and output layer.

[0077] In order to improve the inference calculation speed of the neural network accelerator and the parallelism of the FPGA, this solution quantizes the weight data on the one hand to reduce the bit width of the weight data, thereby reducing the complexity of the inference calculation and memory usage; on the other hand, it performs inference calculations on the data through a hybrid parallel method to reduce the resource consumption on the FPGA chip and improve the parallelism of the FPGA.

[0078] See also Figure 2 The schematic diagram of the structure of an embodiment of the device of the present invention is shown. The acceleration implementation device of the neural network accelerator may include: an acquisition unit 101, a quantization unit 102, a convolution parallel planning unit 103, and an inference calculation unit 104.

[0079] The acquisition unit 101 is configured to acquire weight data of the neural network model, the number of convolution windows of the convolution layer, the number of input channels of the convolution layer, the number of output channels of the convolution layer, and the number of DSP units of the FPGA.

[0080] The quantization unit 102 is configured to quantize the weight data to reduce the bit width of the weight data.

[0081] The bit width of the trained weight data is FP32 type. Although the data has high precision, the amount of calculation and storage resource consumption are large. For example, the DSP consumption of FP32 type weight data is 4, the LUT consumption is 426, and the DSP consumption of INT8 type weight data is 2, and the LUT consumption is 258. Therefore, it is necessary to quantize the weight data to reduce the bit width, reduce the amount of calculation and resource occupation.

[0082] In some embodiments, the weight data includes the weight data of the convolutional layer and the bit width data of the fully connected layer. The quantization unit 102 quantizes the weight data to reduce the specific process of the bit width of the weight data, including: obtaining the weight range of the weight data of the convolutional layer and the weight range of the weight data of the fully connected layer; quantizing the bit width of the layer with the smallest weight range in the weight data of the convolutional layer and the weight data of the fully connected layer to the first bit width, such as INT4 type, and quantizing the bit width of other layers to the second bit width, such as INT8 type; the precision of the first bit width is less than the precision of the second bit width.

[0083] In the neural network model, the range of weight data in different layers is different. Take two convolutional layers and two fully connected layers as an example. Figures 4 to 7 As shown in the figure, the weight range of convolution layer 1 is [-0.5, 0.5], the weight range of convolution layer 2 is [-0.96, 0.5], the weight range of fully connected layer 1 is [-0.91, 0.8], and the weight range of fully connected layer 2 is [-1.11, 0.2]. When quantizing, if the weight range is large but a smaller precision is used for quantization, the accuracy of the data will be affected; if the weight range is small but a larger precision is used for quantization, the consumption of resources will be increased. Therefore, when quantizing the weight data, the dynamic quantization process is used to quantize the four-layer weights into different bit widths, thereby reducing the demand for model storage and calculation and improving the efficiency of the model in resource-constrained environments such as embedded devices. Specifically, the weight data with the smallest weight range is quantized to a smaller INT4 type, and the other weight data is quantized to a larger INT8 type. For example, if the weight range of convolution layer 1 is the smallest, then convolution layer 1 is quantized to INT4, and convolution layer 2, fully connected layer 1, and fully connected layer 2 are quantized to INT8.

[0084] In some embodiments, the quantization unit 102 is further configured to use a preset dynamic quantization technology to quantize the bit width of the layer with the smallest weight range in the weight data of the convolutional layer and the weight data of the fully connected layer to the first bit width, and quantize the bit widths of other layers to the second bit width.

[0085] The preset dynamic quantization technology can be the dynamic quantization technology in the Pytorch deep learning framework. The process of dynamic quantization is as follows: Fig. 9 As shown in the figure, dynamic quantization can re-evaluate the weight range of weight data and select appropriate quantization parameters every time the neural network model processes new input weight data. For example, after the weight data is input for the first time, the weight data of convolution layer 1 is quantized to INT4 type, and after the weight data is input for the second time, the weight data of convolution layer 1 is quantized to INT8 type. The process of dynamic quantization includes: determining the weight range, calculating the quantization parameters, and performing weight quantization. Compared with other quantization methods, the implementation process of dynamic quantization is simpler, and it can quickly realize quantization and model deployment, reduce computing resource consumption, and improve hardware acceleration effects.

[0086] The convolution parallel planning unit 103 is configured to determine the parallel mode of the convolution layer according to the number of the convolution windows, the number of the input channels, the number of the output channels, and the number of DSP units of the FPGA.

[0087] In some embodiments, the parallel mode of the convolution layer includes: parallel convolution kernels or serial convolution kernels, parallel input channels or serial input channels, and parallel output channels or serial output channels; the preset parallel mode of the convolution layer is: parallel convolution kernels, parallel input channels, and parallel output channels.

[0088] The convolution layer usually performs data calculations in four loops, including output channel loop, input channel loop, height loop, and width loop. These four loops are nested with each other to complete the convolution calculation of the convolution layer, extract the required features from the input data, and generate the output feature map.

[0089] The output channel loop is the outermost loop, which determines how many different output feature maps are generated. The feature map corresponding to each output channel is obtained by convolving a set of independent convolution kernels with the input data. In the calculation process of the entire convolution layer, it is first necessary to determine how many output channels are to be generated.

[0090] The input channel loop is the loop of the middle layer. For each output channel, all input channels are traversed, the data of each input channel is convolved with the corresponding convolution kernel channel part, and then these results are added as part of the convolution result of the output channel.

[0091] The width loop and height loop are the innermost loops. After considering the output channels and input channels, for each input and output channel combination, a convolution operation needs to be performed on the spatial dimensions (height and width) of the input data. The convolution kernel slides in the height and width directions of the input data, performing multiplication and accumulation operations element by element.

[0092] When the convolution layer calculates data, the outermost loop (output channel), the middle loop (input channel), and the innermost loop (convolution kernel) can all be calculated in parallel or serially. However, the amount of DSP consumed by parallel calculation is greater than that consumed by serial calculation. Therefore, it is necessary to use parallel calculation as much as possible to improve parallel performance when the amount of DSP consumed is not greater than the DSP amount of the hardware itself.

[0093] In the preset parallel mode of the convolution layer, the convolution layer consumes the highest amount of DSP. For example, in the case of parallel convolution kernels, parallel input channels, and parallel output channels, DSP1 = 25*1*16*6 = 2400, which is much larger than the number of DSPs in the hardware itself (220 DSP units), where 25, 1, 16, and 6 are the number of convolution windows, the number of input channels, the number of convolution groups, and the number of output channels, respectively. Therefore, based on this parallel mode, this solution uses serial computing to replace the parallel computing in the preset parallel mode, so that the amount of DSP consumed does not exceed the amount of DSP in the hardware itself, while ensuring parallel performance.

[0094] In some implementations, the convolution parallel planning unit 103 determines the specific process of the parallel mode of the convolution layer according to the number of the convolution windows, the number of the input channels, the number of the output channels, and the number of DSP units of the FPGA, including:

[0095] The convolution parallel planning unit 103 is further specifically configured to record the product of the number of the convolution windows, the number of the input channels, and the number of the output channels as the DSP unit consumption.

[0096] When calculating the DSP unit consumption, it is assumed that one convolution window uses one DSP unit, that is, 1*the number of convolution windows*the number of input channels*the number of output channels=the DSP unit consumption.

[0097] The convolution parallel planning unit 103 is further configured to determine the relationship between the consumption of the DSP unit and the number of DSP units of the FPGA.

[0098] The convolution parallel planning unit 103 is further specifically configured to determine that the parallel mode of the convolution layer is a preset parallel mode of the convolution layer if the consumption of the DSP unit is less than or equal to the number of DSP units of the FPGA.

[0099] When determining the parallel method, if the preset parallel method of the convolution layer can ensure that the DSP consumption is ≤ the number of DSP units of the FPGA, the full parallel method is directly used to calculate the convolution layer.

[0100] The convolution parallel planning unit 103 is further specifically configured to replace the number of the convolution windows, the number of the input channels, and the number of the output channels with 1 respectively if the DSP unit consumption is greater than the number of DSP units of the FPGA, and calculate the new DSP unit consumption until the new DSP unit consumption is less than or equal to the number of DSP units of the FPGA.

[0101] The convolution parallel planning unit 103 is further specifically configured to, after the new DSP unit consumption is less than or equal to the number of DSP units of the FPGA, if the number of convolution windows is 1, adjust the convolution kernel parallelism in the preset parallel mode of the convolution layer to the convolution kernel serialism; if the number of input channels is 1, adjust the input channel parallelism in the preset parallel mode of the convolution layer to the input channel serialism; if the number of output channels is 1, adjust the output channel parallelism in the preset parallel mode of the convolution layer to the output channel serialism, and then determine that the parallel mode of the convolution layer is the preset parallel mode of the convolution layer.

[0102] When determining the parallel mode, if the DSP consumption is greater than the number of DSP units of the FPGA after adopting the preset parallel mode of the convolution layer, it is necessary to adjust the parallel mode to serial mode to reduce the DSP consumption. Optionally, the parallel mode of the preset convolution layer can be adjusted to serial mode in the order of convolution kernel, input channel, and output channel, or in other orders.

[0103] Taking the convolution kernel as an example, after adjusting the convolution kernel parallel to the convolution kernel serial, the number of convolution windows will be replaced by 1 when calculating the DSP consumption. For example, the total number of convolution windows is 16. Under the convolution kernel parallel, 16 convolution windows are calculated at the same time, and under the convolution kernel serial, only one convolution window is calculated at the same time. After replacing the number of convolution windows with 1, the DSP consumption will be reduced.

[0104] After adjusting the convolution kernel parallel to the convolution kernel serial, if the DSP consumption is ≤ the number of DSP units of the FPGA, the convolution kernel serial, input channel parallel, and output channel parallel mode are determined as the convolution kernel parallel mode; if the DSP consumption is greater than the number of DSP units of the FPGA, the input channel parallel is adjusted to the input channel serial to further reduce the DSP consumption; if the reduced DSP consumption is greater than the number of DSP units of the FPGA, the output channel parallel is adjusted to the output channel serial.

[0105] By adopting a hybrid parallel approach for the convolution kernel, input channel, and output channel, the DSP consumption does not exceed the hardware DSP amount, reducing the cache resources on the FPGA chip and improving the parallel capability of inference calculation.

[0106] In some embodiments, the convolution layer in the neural network model is at least two layers. The convolution parallel planning unit 103 is further configured to adjust the parallel mode of the convolution layer so that the sum of the DSP unit consumption of at least two layers of the convolution layer is less than or equal to the number of DSP units of the FPGA.

[0107] In order to improve the utilization of hardware resources and computing efficiency, when the neural network model includes multiple convolutional layers, it is necessary to enable as many convolutional layers as possible to perform inference calculations simultaneously.

[0108] Taking three convolutional layers as an example, the DSP consumption of convolutional layer one = 1*16*1*16 = 96, where the order of 1, 16, 1, and 16 respectively indicates that one convolution window uses 1 DSP, 16 groups of convolution windows, the number of input channels is 1, and the number of output channels is 16, that is, the convolution kernels are parallel, the input channels are serial, and the output channels are parallel; the DSP consumption of convolutional layer two = 1*1*16*16 = 96, where the order of 1, 1, 16, and 16 respectively indicates that one convolution window uses 1 DSP, 1 group of convolution windows, the number of input channels is 16, and the number of output channels is 16, that is, the convolution kernels are serial, the input channels are parallel, and the output channels are parallel; the DSP consumption of convolutional layer three = 1*1*16*1 = 16, where the order of 1, 1, 16, and 1 respectively indicates that one convolution window uses 1 DSP, 1 group of convolution windows, the number of input channels is 16, and the number of output channels is 1, that is, the convolution kernels are serial, the input channels are parallel, and the output channels are serial. The total DSP consumption of the three convolutional layers is 208, which is less than the number of DSPs in the hardware itself (220 DSP units), so the three convolutional layers can perform inference calculations simultaneously.

[0109] The inference calculation unit 104 is configured as a calculation unit to perform inference calculation in a parallel manner using the quantized weight data and the determined convolution layer to accelerate the inference on the FPGA. The inference calculation includes the calculation of the neural network model during the training process and the inference during the prediction process.

[0110] Figure 8 FIG. 1 is a flow chart of another embodiment of a method for implementing acceleration of a neural network accelerator. Figure 8 As shown, including:

[0111] Step 1: Input the trained model parameters into the neural network model, including image data and weight data, and dynamically quantize the type of weight data according to the parameter size of different layers.

[0112] Step 2: Evaluate the parallel mode based on the resources on the chip and configure the parallelism parameters.

[0113] Step 3: Design each module in the neural network model and write the driver software. After the inference calculation is completed, print the inference result.

[0114] Since the processing and functions implemented by the device of this embodiment basically correspond to the embodiments, principles and examples of the aforementioned method, for the details not fully described in this embodiment, please refer to the relevant descriptions in the aforementioned embodiments, and no further elaboration will be made here.

[0115] By adopting the technical solution of the present invention, the weights are quantized, the parallel mode of the convolution layer is determined according to the number of convolution windows, the number of input channels, and the number of output channels of the convolution layer, and the inference calculation is performed using the quantized weight data and the determined parallel mode of the convolution layer. This reduces the consumption of FPGA on-chip resources, speeds up the inference speed, makes full use of on-chip resources, and improves the parallel performance.

[0116] According to an embodiment of the present invention, a neural network accelerator corresponding to the acceleration implementation device of the neural network accelerator is also provided. The neural network accelerator may include: the acceleration implementation device of the neural network accelerator described above.

[0117] Since the processing and functions implemented by the neural network accelerator of this embodiment basically correspond to the embodiments, principles and examples of the aforementioned device, for the details not fully described in this embodiment, please refer to the relevant descriptions in the aforementioned embodiments and will not be repeated here.

[0118] By adopting the technical solution of the present invention, the weights are quantized, the parallel mode of the convolution layer is determined according to the number of convolution windows, the number of input channels, and the number of output channels of the convolution layer, and the inference calculation is performed using the quantized weight data and the determined parallel mode of the convolution layer. This reduces the consumption of FPGA on-chip resources, speeds up the inference speed, makes full use of on-chip resources, and improves the parallel performance.

[0119] According to an embodiment of the present invention, a storage medium corresponding to the acceleration implementation method of a neural network accelerator is also provided, wherein the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the acceleration implementation method of the neural network accelerator described above.

[0120] Since the processing and functions implemented by the storage medium of this embodiment basically correspond to the embodiments, principles and examples of the aforementioned method, for the details not fully described in this embodiment, please refer to the relevant descriptions in the aforementioned embodiments, and no further elaboration will be made here.

[0121] By adopting the technical solution of the present invention, the weights are quantized, the parallel mode of the convolution layer is determined according to the number of convolution windows, the number of input channels, and the number of output channels of the convolution layer, and the inference calculation is performed using the quantized weight data and the determined parallel mode of the convolution layer. This reduces the consumption of FPGA on-chip resources, speeds up the inference speed, makes full use of on-chip resources, and improves the parallel performance.

[0122] According to an embodiment of the present invention, a computer program product corresponding to the acceleration implementation method of a neural network accelerator is also provided. The computer program product includes a computer program, and when the computer program product is processed and executed, the steps of the acceleration implementation method of the neural network accelerator are implemented.

[0123] Since the processing and functions implemented by the computer program product of this embodiment basically correspond to the embodiments, principles and examples of the aforementioned method, for the details not fully described in this embodiment, please refer to the relevant descriptions in the aforementioned embodiments and will not be repeated here.

[0124] By adopting the technical solution of the present invention, the weights are quantized, the parallel mode of the convolution layer is determined according to the number of convolution windows, the number of input channels, and the number of output channels of the convolution layer, and the inference calculation is performed using the quantized weight data and the determined parallel mode of the convolution layer. This reduces the consumption of FPGA on-chip resources, speeds up the inference speed, makes full use of on-chip resources, and improves the parallel performance.

[0125] In summary, it is easy for those skilled in the art to understand that, under the premise of no conflict, the above-mentioned advantageous methods can be freely combined and superimposed.

[0126] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the claims of the present invention.

Claims

1. A method for accelerating a neural network accelerator, characterized in that: The neural network accelerator includes an FPGA; a neural network model is deployed on the FPGA; the neural network model includes a convolutional layer; The method comprises: Obtain weight data of the neural network model, the number of convolution windows of the convolution layer, the number of input channels of the convolution layer, the number of output channels of the convolution layer, and the number of DSP units of the FPGA; quantizing the weight data to reduce the bit width of the weight data; Determine the parallel mode of the convolution layer according to the number of the convolution windows, the number of the input channels, the number of the output channels, and the number of DSP units of the FPGA; Inference calculation is performed in a parallel manner using the quantized weight data and the determined convolutional layer.

2. The acceleration implementation method of the neural network accelerator according to claim 1, characterized in that: The neural network model also includes a fully connected layer; the weight data includes weight data of the convolutional layer and bit width data of the fully connected layer; Quantizing the weight data to reduce the bit width of the weight data includes: Obtaining a weight range of the weight data of the convolutional layer and a weight range of the weight data of the fully connected layer; Among the weight data of the convolutional layer and the weight data of the fully connected layer, the bit width of the layer with the smallest weight range is quantized as the first bit width, and the bit widths of the other layers are quantized as the second bit width; the precision of the first bit width is less than the precision of the second bit width.

3. The acceleration implementation method of the neural network accelerator according to claim 2, characterized in that: Also includes: Using a preset dynamic quantization technology, the bit width of the layer with the smallest weight range in the weight data of the convolutional layer and the weight data of the fully connected layer is quantized to the first bit width, and the bit widths of other layers are quantized to the second bit width.

4. The acceleration implementation method of the neural network accelerator according to claim 1, characterized in that: The parallel mode of the convolution layer includes: parallel convolution kernels or serial convolution kernels, parallel input channels or serial input channels, and parallel output channels or serial output channels; the preset parallel mode of the convolution layer is: parallel convolution kernels, parallel input channels, and parallel output channels; Determining the parallel mode of the convolution layer according to the number of the convolution windows, the number of the input channels, the number of the output channels, and the number of DSP units of the FPGA includes: The product of the number of the convolution windows, the number of the input channels, and the number of the output channels is recorded as the DSP unit consumption; Determine the relationship between the consumption of the DSP unit and the number of DSP units of the FPGA; If the consumption of the DSP unit is less than or equal to the number of DSP units of the FPGA, determining that the parallel mode of the convolution layer is a preset parallel mode of the convolution layer; If the DSP unit consumption is greater than the number of DSP units of the FPGA, the number of convolution windows, the number of input channels, and the number of output channels are replaced with 1 in sequence, and a new DSP unit consumption is calculated until the new DSP unit consumption is less than or equal to the number of DSP units of the FPGA; After the new DSP unit consumption is less than or equal to the number of DSP units of the FPGA, if the number of convolution windows is 1, the convolution kernel parallelism in the preset parallel mode of the convolution layer is adjusted to the convolution kernel serialism; if the number of input channels is 1, the input channel parallelism in the preset parallel mode of the convolution layer is adjusted to the input channel serialism; if the number of output channels is 1, the output channel parallelism in the preset parallel mode of the convolution layer is adjusted to the output channel serialism, and then the parallel mode of the convolution layer is determined to be the preset parallel mode of the convolution layer.

5. The acceleration implementation method of the neural network accelerator according to claim 4, characterized in that: The convolutional layer in the neural network model is at least two layers; the method further includes: The parallel mode of the convolutional layers is adjusted so that the sum of the DSP unit consumption of at least two layers of the convolutional layers is less than or equal to the number of DSP units of the FPGA.

6. A neural network accelerator acceleration implementation device, characterized in that: The neural network accelerator includes an FPGA; a neural network model is deployed on the FPGA; the neural network model includes a convolutional layer; The device comprises: An acquisition unit is configured to acquire weight data of the neural network model, the number of convolution windows of the convolution layer, the number of input channels of the convolution layer, the number of output channels of the convolution layer, and the number of DSP units of the FPGA; a quantization unit, configured to quantize the weight data to reduce the bit width of the weight data; A convolution parallel planning unit, configured to determine a parallel mode of the convolution layer according to the number of the convolution windows, the number of the input channels, the number of the output channels, and the number of DSP units of the FPGA; The inference calculation unit is configured to perform inference calculation in a parallel manner using the quantized weight data and the determined convolutional layer.

7. The acceleration implementation device of the neural network accelerator according to claim 6, characterized in that: The neural network model also includes a fully connected layer; the weight data includes weight data of the convolutional layer and bit width data of the fully connected layer; The quantization unit quantizes the weight data to reduce the bit width of the weight data, including: Obtaining a weight range of the weight data of the convolutional layer and a weight range of the weight data of the fully connected layer; Among the weight data of the convolutional layer and the weight data of the fully connected layer, the bit width of the layer with the smallest weight range is quantized as the first bit width, and the bit widths of the other layers are quantized as the second bit width; the precision of the first bit width is less than the precision of the second bit width.

8. A neural network accelerator, characterized in that: include: An acceleration implementation device for a neural network accelerator as claimed in claim 6 or 7.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the acceleration implementation method of the neural network accelerator described in any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images