Multi-DNN reasoning-oriented mixed precision FPGA (Field Programmable Gate Array) hardware accelerator and quantification method

Abstract

The invention discloses a Multi-DNN reasoning-oriented mixed precision FPGA (Field Programmable Gate Array) hardware accelerator and a quantification method, and relates to the technical field of neural network accelerators. The multi-DNN reasoning-oriented mixed precision FPGA hardware accelerator comprises a processor, a plurality of calculation core modules, a DDR (Double Data Rate) module and an intermediate data cache module; each computing core module supports one of high, medium and low computing precision, and a computing unit array module in each computing core module consists of computing units for performing three bit width multiply-accumulate operations; the computing unit is realized by different resources in the FPGA; according to the quantization method, weight data of the neural network are quantized by taking a layer as a unit by adopting three bit widths, namely a high bit width, a middle bit width and a low bit width, and intermediate feature map data of a neural network layer are quantized by adopting one bit width. According to the hardware accelerator and the quantification method provided by the invention, the hardware utilization rate, the throughput and the energy efficiency ratio of the accelerator can be improved, the neural network load of mixed precision quantification can be processed in parallel, and the memory access bandwidth and the energy efficiency of Multi-DNN reasoning are optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of FPGA and neural network accelerators, and more specifically, to a mixed-precision FPGA hardware accelerator and quantization method for Multi-DNN reasoning. Background Art

[0002] In recent years, the development of deep neural networks has greatly enhanced the performance of various artificial intelligence systems, such as image processing, speech recognition, natural language processing, etc. With the advancement of algorithm models, the complexity of deep neural networks has become increasingly higher. They have performed well in various tasks, surpassing the performance of traditional machine learning methods. Therefore, deep neural networks have become one of the core technologies in new applications in various related fields such as embedded, mobile devices, and cloud.

[0003] As the application scenarios of deep neural networks increase, researchers further integrate multiple deep neural networks into the same application to meet more complex needs. In modern embedded systems, new complex applications such as autonomous driving, augmented reality (AR), etc. require running multiple different deep neural networks at the same time. In particular, in cloud data centers, processing a large number of deep neural network inference requests from users also brings the need to run multiple different deep neural networks on hardware at the same time. This deep neural network inference scenario for multi-tenant request processing is called a Multi-DNN scenario.

[0004] However, there are few studies on using new quantization methods to enhance Multi-DNN accelerators. This is because each neural network has a different tolerance for quantization technology. The accuracy degradation caused by a fixed single quantization technology in different neural networks can vary greatly. Using hardware to support flexible quantization technology is a great challenge.

[0005] The paper Spantidi O, Zervakis G, Alsalamin S, et al. Targeting DNN inference via efficient utilization of heterogeneous precision DNN accelerators [J]. IEEE Transactions on Emerging Topics in Computing, 2022, 11 (1): 112-125 proposed a Multi-DNN accelerator system, in which quantization technology is applied to enhance the energy efficiency of the accelerator. However, the application of quantization technology in its design is relatively simple, and the neural network is quantized by only using a uniform quantization bit width for all layers, which greatly limits the accuracy of inference. The accelerator hardware architecture also only supports the processing of the entire network on the same computing core, and the flexibility and resource utilization are relatively low.

[0006] Therefore, for Multi-DNN accelerators, we need to design more flexible hardware with higher resource utilization to support the use of more novel quantization schemes in Multi-DNN scenarios. Summary of the invention

[0007] The purpose of the present invention is to overcome the deficiencies in the above-mentioned background technology and provide a hardware accelerator and a corresponding quantization method for accelerating Multi-DNN reasoning on FPGA using hybrid quantization technology to achieve better hardware utilization, throughput and energy efficiency.

[0008] To achieve the above objectives, the present invention provides a mixed-precision FPGA hardware accelerator for Multi-DNN reasoning.

[0009] The technical solution of the present invention is as follows:

[0010] A mixed-precision FPGA hardware accelerator for Multi-DNN reasoning, including a processor, an intermediate data cache module, a DDR module, and several computing core modules;

[0011] The plurality of computing core modules are used to perform convolution operations of multiple specific precisions, and the number of the computing core modules used to perform convolution operations of each specific precision is at least one;

[0012] The intermediate data cache module is used to cache the feature graph data generated by the intermediate calculation of the neural network in the computing core module;

[0013] The DDR module is used to store other data except the feature map data generated by the intermediate calculation of the neural network;

[0014] The processor is used to control the operation of each module in the hardware accelerator.

[0015] As a preferred solution, the computing core modules are divided into three types: high-precision, medium-precision and low-precision computing core modules according to the supported convolution operation precision. Different types of computing core modules have the same sub-module types and data path structures, wherein the sub-modules of different types of computing core modules have different operation bit widths and data path bit widths, corresponding to high bit width, medium bit width and low bit width respectively.

[0016] As a preferred solution, the computing core module includes a computing unit array module, a weight data cache module, an input data cache module, an output data accumulation module and a control module;

[0017] The computing unit array module is used to calculate multiplication and accumulation operations with high parallelism, and the bit width of the computing unit array module determines the bit width of the multiplication and accumulation operations calculated by it;

[0018] The weight data cache module is used to cache the weight data of the neural network and provide input to the computing unit array module. The bit width of the weight data cache module determines the bit width of the weight data cached therein;

[0019] The input data cache module is used to cache the input feature map data of the neural network and provide input to the computing unit array module. The bit width of the input data cache module determines the bit width of the input feature map data cached by it;

[0020] The output data accumulation module is used to accumulate the convolution part and data generated by the computing unit array module, and the bit width of the output data accumulation module determines the bit width of the accumulated convolution part and data;

[0021] The control module is used to receive control information from the processor, and control the operation of the computing unit array module, the weight data cache module, the input data cache module, the output part and the accumulation module according to the control information.

[0022] As a preferred embodiment, the computing unit array includes computing units that perform multiplication and accumulation operations. The size of the computing unit array is determined by three dimensional parameters, namely, the feature map size dimension of the convolution calculation, the input channel dimension, and the output channel dimension. The bit width of the computing unit array module determines the bit width of the computing unit.

[0023] As a preferred solution, the computing units are divided into three types: high bit width, medium bit width and low bit width according to the bit width of the multiplication and accumulation operations performed. The computing units with high bit width and medium bit width are implemented by DSP resources in the FPGA, and the computing units with low bit width are implemented by logic resources in the FPGA including a lookup table.

[0024] As a preferred solution, when designing the computing unit array, the accumulation cascade of high and medium bit-width computing units along the input channel dimension is implemented using fixed cascade connections of DSP resources, and register interrupt logic technology is inserted between the accumulation cascades of low bit-width computing units along the input channel dimension.

[0025] To achieve the purpose of the invention, the present invention also provides a mixed precision quantization method for Multi-DNN reasoning, which is applied to a mixed precision FPGA hardware accelerator for Multi-DNN reasoning described in the above technical solution. The quantization method includes:

[0026] Obtaining a neural network that needs mixed precision quantization and a training data set of the neural network;

[0027] Calculating the quantized sensitivity of the parameters of each layer of the neural network;

[0028] According to the training data set, simulating the delay of each layer of the neural network running on the corresponding computing core in the hardware accelerator according to different quantization bit widths;

[0029] According to the quantization sensitivity and the latency of the neural network layer running on the accelerator, the neural network layer is divided into three groups, and three weight quantization bit widths of high, medium and low are set for the three groups respectively to quantize the weight data of the neural network layer;

[0030] A unified quantization bit width is set for the output feature map data of all layers of the neural network for quantization.

[0031] As a preferred solution, the three groups are respectively set with three kinds of weight quantization bit widths of high, medium and low to quantize the weight data of the neural network layer, and the following quantization bit width allocation rules are adopted: the delay of each layer of the neural network running on the hardware accelerator with different precision and the parallelism parameters of the hardware computing core module are used as constraints, and the quantization bit width of each layer of the neural network is allocated from high, medium and low bit widths according to the quantization sensitivity.

[0032] As a preferred solution, the constraints for allocating quantization bit widths are specifically as follows: calculating the sum of the running delays of various layers of the neural network that are assigned the same weight quantization bit width on the hardware accelerator, and at the same time calculating the sum of the parallelism of computing cores of different precisions in the hardware accelerator, constraining the ratio of the sum of the running delays of layers of various quantization bit widths to be similar to the ratio of the sum of the parallelism of computing cores of various bit widths.

[0033] As a preferred solution, after completing the steps of quantizing the neural network, quantization-aware training is used to fine-tune the quantized neural network to restore the accuracy of neural network reasoning.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. The computing core module in the hardware accelerator of the present invention adopts multiplication and accumulation operations to achieve compatibility with high, medium and low bit width convolution operations, so that a computing core module can support calculations of different bit widths, improve the flexibility of the accelerator, and solve the problem of low resource utilization when the mixed precision quantization model is deployed on the hardware accelerator.

[0036] 2. The hardware accelerator of the present invention adopts a structure in which a computing core module only supports a specific bit-width convolution operation. Compared with the structure in which the same computing core module supports operations of different bit widths through reconstructed logic, it can eliminate the hardware resource overhead increased by switching the computing core module between different bit widths, while simplifying the data path and improving the operating frequency of the hardware.

[0037] 3. The design of the high-bit-width and medium-bit-width computing unit arrays in the present invention can achieve a higher operating frequency by fully utilizing the pipeline registers in the DSP and the cascade fixed connections between the DSPs. Since the low-bit-width multiplication has a lower logic level, it can achieve a similarly high operating frequency by effectively utilizing the configurable logic resource blocks including the lookup table.

[0038] 4. The quantization method of the present invention balances the computational load of each computing core module when the hardware accelerator is running by controlling the ratio of the three quantization bit widths of the neural network layer, thereby improving the hardware utilization of the computing core module.

[0039] 5. The quantization method of the present invention uses three quantization bit widths of high, medium and low to quantize the weight data of the neural network, taking into account the accuracy of the neural network model after quantization and the throughput and energy efficiency of the neural network deployed on the hardware accelerator for reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is an overall structural framework diagram of a mixed-precision FPGA hardware accelerator for Multi-DNN reasoning according to an embodiment of the present application;

[0041] Figure 2 A schematic diagram of the structure of a computing unit array in a hardware accelerator according to an embodiment of the present application;

[0042] Figure 3 A schematic diagram of implementing multiplication of two 8-bit signed numbers by a 5-bit unsigned number in a DSP according to an embodiment of the present application;

[0043] Figure 4 It is a schematic diagram of implementing the multiplication of four 4-bit signed numbers by 5-bit unsigned numbers in DSP according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0045] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0046] Example 1

[0047] See also Figure 1 , Embodiment 1 of the present application provides a mixed-precision FPGA hardware accelerator for Multi-DNN reasoning, including: a plurality of computing core modules with different computing precisions, an intermediate data cache module, a DDR module, and a processor for controlling the operation of the hardware accelerator. The computing core module is used to perform convolution operations of multiple specific precisions, and the number of the computing core modules used to perform convolution operations of each specific precision is at least one; the intermediate data cache module is used to cache the feature map data generated by the intermediate calculation of the neural network in the computing core module; the DDR module is used to store other data except the feature map data generated by the intermediate calculation of the neural network; the processor is used to control the operation of each module in the hardware accelerator, and the processor sends operation instructions to the computing core module via the PCIe bus to the AXI bus, and the computing core module communicates with the intermediate data cache module and the DDR module via the AXI bus.

[0048] The computing core module includes the following modules: computing unit array module, weight data cache module, input data cache module, output data accumulation module and control module.

[0049] The computing unit array module is used to calculate multiplication and accumulation operations with high parallelism. The bit width of the computing unit array module determines the bit width of the computing unit and the multiplication and accumulation operations calculated by it. The weight data cache module is used to cache the weight data of the neural network and provide input to the computing unit array module. The bit width of the weight data cache module determines the bit width of the weight data cached by it. The input data cache module is used to cache the input feature map data of the neural network and provide input to the computing unit array module. The bit width of the input data cache module determines the bit width of the input feature map data cached by it. The output data accumulation module is used to accumulate the convolution part and data generated by the computing unit array module. The bit width of the output data accumulation module determines the bit width of the accumulated convolution part and data. The control module is used to receive the control information of the processor and control the operation of the computing unit array module, the weight data cache module, the input data cache module, and the output data accumulation module according to the control information.

[0050] To support mixed precision quantized neural network reasoning, the computing core module described in this embodiment 1 has the following characteristics: the computing core modules included in the accelerator have three types of computing precision: high, medium and low. There can be one or more computing core modules of each precision. One computing core module supports a convolution operation of a specific precision, and the three types support high, medium and low convolution operation precisions respectively. The submodule types and data flow path structures in the computing core modules of different precisions are the same, and each computing core module only has a different computing bit width of the submodule and the bit width of the data path.

[0051] The convolution operation precision supported by the computing core module determines the bit width of the module. The computing units in the computing unit array support multiplication and accumulation operations of this specific bit width, the weight data cache module supports weight data cache of this specific bit width, the input data cache module supports input data cache of this specific bit width, and the output data accumulation module supports partial and data accumulation of this specific bit width.

[0052] Since padding lower bit-width data with zeros can increase the data bit width, by padding the high bits of data with zeros, the high bit-width computing units can be compatible with low bit-width multiplication and accumulation operations, thereby using high-precision computing core modules to run low-precision convolution operations, providing compatibility for high-precision computing cores to improve hardware utilization.

[0053] The workflow of the calculation core module is as follows:

[0054] 1. The control module receives the neural network layer parameters and data storage address that need to be calculated from the processor configuration.

[0055] 2. The control module notifies the weight data cache module to take out the neural network layer weight data that needs to be calculated from the DDR module for caching, and notifies the input data cache module to take out the neural network layer input data that needs to be calculated from the intermediate data cache module for caching.

[0056] 3. The control module notifies the computing unit array to start computing.

[0057] 4. The control module notifies the output data accumulation module to accumulate the partial sum data output by the computing unit array, and after the accumulation is completed, outputs it to the intermediate data cache module for storage.

[0058] The computing units in the computing unit array have the following characteristics: the computing units have three bit widths: high, medium and low, which can respectively perform three multiplication and accumulation operations. In order to evenly utilize different logic resources in the FPGA, the high-bit-width computing unit array module uses the DSP resources in the FPGA to implement high-bit-width multiplication and accumulation computing units, the medium-bit-width computing unit array module uses the DSP resources in the FPGA to implement medium-bit-width multiplication and accumulation computing units, and the low-bit-width computing unit array module uses logic resources such as lookup tables in the FPGA to implement low-bit-width multiplication and accumulation computing units.

[0059] The high-bitwidth and medium-bitwidth computing units are implemented using DSP resources in FPGA. When designing the computing unit array, the accumulated cascade of the computing units along the input channel dimension is implemented using the fixed cascade connection of DSP resources, which can achieve a higher operating frequency. The low-bitwidth computing units are implemented using the lookup table resources in FPGA. When designing the computing unit array, the register interrupt logic technology is inserted between the accumulated cascade of the computing units along the input channel dimension, which can achieve a higher operating frequency.

[0060] The size of the computing unit array is determined by three dimensional parameters: Uhw, Uic, and Uoc, which correspond to the feature map size dimension, input channel dimension, and output channel dimension of the convolution operation, respectively.

[0061] The working process of the computing unit array is as follows: after receiving the start operation signal, the weight data is first preset into the computing unit array through the shift register, and then the input data enters the computing unit array to start the calculation. The input data is delayed by 0-Uic clock cycles along the Uic dimension and enters the computing unit. The multiplication and accumulation operations are completed unit by unit along the Uic dimension, and the accumulated partial sum is output by the last level computing unit. In this way, the data calculation of a part of the neural network layer is completed, and the operation is repeated until the single-layer network calculation is completed.

[0062] When the hardware accelerator receives a set of neural network inference requests, the processor allocates each computing core to complete the calculation of a certain neural network layer according to the bit width of each neural network layer that needs to be inferred and the computing bit width supported by each computing core until all neural network inference requests are processed.

[0063] Example 2

[0064] This embodiment 2 provides a mixed precision quantization method for Multi-DNN reasoning based on a mixed precision FPGA hardware accelerator for Multi-DNN reasoning in embodiment 1, which is as follows:

[0065] A mixed precision quantization method for Multi-DNN reasoning provided in this embodiment 2 has the following characteristics: the weight data of the neural network is quantized in units of layers using three bit widths of high, medium and low, and the intermediate feature map data of all layers of the neural network is quantized using one bit width. The selection of the quantization bit width of the weight data of each layer of the neural network is based on the delay of each layer of the neural network running on the hardware accelerator at different precisions and the hardware computing core parallelism parameter as constraints, and the quantization bit width of each layer of the neural network is allocated from the three bit widths of high, medium and low according to the quantization sensitivity obtained by analyzing each layer of the neural network.

[0066] The constraints on the selection of the quantization bit width of each layer of the neural network are calculated by assigning the same weight quantization bit width to each layer of the neural network, and the sum of the running delays on the hardware accelerator, and the sum of the parallelism of the computing cores with different precisions in the hardware accelerator. The total ratio of the running delays of layers with various quantization bit widths is constrained to be similar to the total ratio of the parallelism of the computing cores with various bit widths.

[0067] After obtaining the quantization bit width of the weight data of each layer of the neural network through the above method, the network is quantized using the obtained quantization bit width, and quantization-aware training is used to fine-tune the quantized network to restore the network reasoning accuracy.

[0068] The mixed precision quantization method for Multi-DNN reasoning is as follows:

[0069] 1. Obtain the neural network that needs mixed precision quantization and the training data set of the neural network;

[0070] 2. Calculate the quantitative sensitivity of the parameters of each layer of the neural network;

[0071] 3. Based on the training data set, simulate the latency of each layer of the neural network running on the corresponding computing core in the hardware accelerator according to different quantization bit widths;

[0072] 4. According to the quantization sensitivity and the latency of the neural network layer running on the accelerator, the neural network layer is divided into three groups, and the weight data of the neural network layer is quantized with three weight quantization bit widths of high, medium and low, so that the ratio of the total latency of each group is close to the throughput ratio of each type of computing core of the hardware accelerator;

[0073] 5. Set a unified quantization bit width for the output feature map data of each layer of the neural network for quantization;

[0074] 6. After quantizing the network according to the quantization bit width set above, perform quantization-aware training on the neural network to fine-tune the quantized model parameters and restore the neural network reasoning accuracy.

[0075] The other technical details and implementation steps of this embodiment 2 are the same as those of embodiment 1 and will not be repeated here.

[0076] Example 3

[0077] Based on Example 1 and Example 2, this Example 3 further describes the specific implementation of a mixed-precision FPGA hardware accelerator and quantization method for Multi-DNN reasoning of the present invention, as follows:

[0078] Figure 2 This is a hardware structure diagram of the computing unit array. The computing unit in the figure is represented by PE. The figure does not indicate the actual number of units, but only the connection relationship. The weight data is input by the weight data cache module, preset to the computing unit array through the shift register, and input to each computing unit, where the computing units with the same input channel subscript and the same output channel subscript share one weight data. The feature map data is input by the input data cache module, and multiple input feature map data along the input channel dimension are delayed by zero to Uic (input channel dimension) clock cycles through the shift register to enter the computing unit array, and input to each computing unit, where the computing units with the same input feature map size subscript and the same input channel subscript share one input feature map data.

[0079] The computing unit array is composed of computing units that perform multiplication and accumulation operations and data flow paths. The data flow path structure is the same in computing arrays of different bit widths, and only the computing unit bit width is different from the data path bit width. In the present embodiment 3, the high, medium and low weight bit widths of the convolution operation are 8 bits, 4 bits and 2 bits respectively, and the output feature map bit width of the convolution operation is 5 bits. The computing unit array utilizes the structure of the logic resources and hardware cores in the Ultrascale+ series FPGAs, so that the accelerator can achieve a higher operating frequency while achieving high parallelism. When implementing the multiplication and accumulation operation of 8-bit high-bit-width weight data by 5-bit input feature map data, based on the above-mentioned shared characteristics of weight data, two adjacent multiplication and accumulation units can be mapped to one DSP. Such as Figure 3 As shown in the schematic diagram of implementing the multiplication of two 8-bit signed numbers by 5-bit unsigned numbers in DSP, the DSP is configured to implement the operation function of (A+D)*B+P. Two signed weight data are spliced ​​through (A+D) pre-addition, an unsigned input feature map data is input into the B operand, and the two spliced ​​partial sum data can be obtained at the multiplication output P. Since there is a five-bit interval between the two partial sum data, the accumulation operation of the two sets of partial sum data can be completed by cascading the DSP's accumulation input Pi and output Po. When implementing the multiplication and accumulation operation of 4-bit medium-width weight data by 5-bit input feature map data, according to the above-mentioned characteristics of weight data sharing and input feature map data sharing, four adjacent multiplication and accumulation units can be mapped to one DSP. As shown Figure 4 As shown in the schematic diagram of implementing the multiplication of four 4-bit signed numbers by 5-bit unsigned numbers in DSP, the DSP is configured to implement the operation function of (A+D)*B+P. Two signed weight data are spliced ​​through (A+D) pre-addition, and the two spliced ​​unsigned input feature map data are input into the B operand. The two spliced ​​partial sum data can be obtained in the multiplication output M. Since there is a two-bit interval between the two partial sum data, the accumulation operation of the two sets of partial sum data can be completed by cascading the accumulation input Pi and output Po of the DSP. When implementing the multiplication of 2-bit low-width weight data by 5-bit input feature map data, the calculation unit is implemented by the resources in the configurable logic block. At this time, the multiplication is decomposed into a logical AND operation and an addition operation of two 6-bit partial sum data, which can be mapped to the lookup table and carry chain in the configurable logic resource block. The accumulation between partial sums can be efficiently implemented through the addition tree of the cascaded carry chain.

[0080] By making full use of the pipeline registers in the DSP and the cascade fixed connections between DSPs, the high-bitwidth and medium-bitwidth computing unit arrays can reach an operating frequency of 300MHz. The low-bitwidth multiply-accumulate array utilizes the pipeline with low logic levels after register segmentation to achieve an operating frequency of 300MHz.

[0081] In the workflow of a mixed precision quantization method for Multi-DNN reasoning in Example 2, the quantization sensitivity of the weight data of each layer of the neural network calculated in step 2 can be obtained using the second-order partial derivative information of the weight parameters of each layer, that is, the maximum eigenvector of the Hessian matrix of the layer weight parameters is used as an indicator of quantization sensitivity. In addition, the quantization sensitivity can also be obtained using a method that is more convenient to calculate, such as calculating the KL divergence of the output of each layer after quantization and the output of each layer before quantization, and using it as a sensitivity indicator of the quantization of the layer.

[0082] In the workflow of a mixed precision quantization method for Multi-DNN reasoning in Example 2, the division of groups in step 4 sorts the neural network layers from large to small according to the quantization sensitivity, and then divides them into three consecutive groups. The group with high quantization sensitivity is assigned the largest bit width, the group with medium quantization sensitivity is assigned the medium bit width, and the group with low quantization sensitivity is assigned the lowest bit width. The division of groups is determined by calculating the sum of the delays of each group of neural network layers running on the hardware accelerator with the corresponding quantization bit width, and the division of quantization groups is obtained by constraining the ratio of the sum of the running delays of each group to be similar to the ratio of the sum of the parallelism of various computing core modules of the hardware accelerator. Thus, when the hardware accelerator platform runs multiple neural networks in parallel, the running time of each computing core module of the accelerator tends to be balanced, avoiding the situation where some computing core modules remain running for a long time while some computing modules are idle and inefficient. In addition, when performing mixed precision quantization of neural networks, different neural networks have different tolerances for quantization, and the quantization strategy can be changed by adjusting the ratio of each group. For example, for networks whose accuracy drops significantly after quantization, the proportion of low-bit-width groups can be reduced and the proportion of high-bit-width groups can be increased to reduce the number of low-bit-width quantized network layers and improve the accuracy of the neural network. For networks whose accuracy drops slightly after quantization, the proportion of high-bit-width groups can be reduced and the proportion of low-bit-width groups can be increased to increase the number of low-bit-width quantized network layers and improve the operating efficiency of the neural network on the hardware platform.

[0083] In the workflow of a mixed precision quantization method for Multi-DNN reasoning in Example 2, the quantization method of weight data in step 4 can use asymmetric maximum and minimum value quantization, that is, the maximum value Max and the minimum value Min of all weight data in the neural network layer are counted, and the range Max-Min of the weight data value is calculated. Scale=2^n / (Max-Min) is calculated according to the quantization bit n, Zero=Scale*Min+2^(n-1), and after the weight data is linearly transformed y=Scale*x-Zero, the range of y becomes [-2^(n-1), 2^(n-1)-1], and then y is rounded to an integer type. In addition, other more complex weight quantization methods can also be used.

[0084] The output data quantization method adopted by quantization-aware training can use saturated truncation quantization. The output data is saturated and truncated with a truncation value alpha. Data greater than alpha is assigned alpha. Then Scale=2^n / alpha is calculated according to the quantization bit n. After the output data is linearly transformed y=Scale*x, the range of y becomes [0,2^n-1]. Then y is rounded to an integer type to complete the quantization of the output data.

[0085] When processing neural network requests with different mixed precision quantization, the hardware accelerator can flexibly select the bit width configuration of each core to achieve higher processing efficiency. Taking a hardware accelerator with four computing cores as an example, when processing a group of networks with a large proportion of high bit width quantization, the bitstream in the FPGA can be updated through the partial reconfiguration function of the FPGA to configure the accelerator into two high bit width cores, one medium bit width core and one low bit width core. When it is necessary to process a group of networks with a large proportion of low bit width quantization, the accelerator can be partially reconfigured into one high bit width core, one medium bit width core and two low bit width cores.

[0086] In summary, it can be seen from the above embodiments that the present invention has the following beneficial effects:

[0087] 1. The computing core module in the hardware accelerator of the present invention adopts multiplication and accumulation operations to achieve compatibility with high, medium and low bit width convolution operations, so that a computing core module can support calculations of different bit widths, improve the flexibility of the accelerator, and solve the problem of low resource utilization when the mixed precision quantization model is deployed on the hardware accelerator.

[0088] 2. The hardware accelerator of the present invention adopts a structure in which a computing core module only supports a specific bit-width convolution operation. Compared with the structure in which the same computing core module supports operations of different bit widths through reconstructed logic, it can eliminate the hardware resource overhead increased by switching the computing core module between different bit widths, while simplifying the data path and improving the operating frequency of the hardware.

[0089] 3. The design of the high-bit-width and medium-bit-width computing unit arrays in the present invention can achieve a higher operating frequency by fully utilizing the pipeline registers in the DSP and the cascade fixed connections between DSPs. Since the low-bit-width multiplication has a lower logic level, the same higher operating frequency can be achieved by effectively utilizing the lookup table, carry chain and registers in the configurable logic resource block.

[0090] 4. The quantization method of the present invention balances the computational load of each computing core module when the hardware accelerator is running by controlling the ratio of the three quantization bit widths of the neural network layer, thereby improving the hardware utilization of the computing core module.

[0091] 5. The quantization method of the present invention uses three quantization bit widths of high, medium and low to quantize the weight data of the neural network, taking into account the accuracy of the neural network model after quantization and the throughput and energy efficiency of the neural network deployed on the hardware accelerator for reasoning.

[0092] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A mixed-precision FPGA hardware accelerator for Multi-DNN reasoning, characterized in that: It includes a processor, an intermediate data cache module, a DDR module, and several computing core modules; The plurality of computing core modules are used to perform convolution operations of multiple specific precisions, and the number of the computing core modules used to perform convolution operations of each specific precision is at least one; The intermediate data cache module is used to cache the feature map data generated by the intermediate calculation of the neural network in the computing core module; The DDR module is used to store other data except the feature map data generated by the intermediate calculation of the neural network; The processor is used to control the operation of each module in the hardware accelerator.

2. A mixed-precision FPGA hardware accelerator for Multi-DNN reasoning according to claim 1, characterized in that: The computing core modules are divided into three types: high-precision, medium-precision and low-precision computing core modules according to the supported convolution operation precision. Different types of computing core modules have the same sub-module types and data path structures. The computing bit widths of the sub-modules and the bit widths of the data paths of different types of computing core modules are different, corresponding to high bit width, medium bit width and low bit width respectively.

3. The mixed-precision FPGA hardware accelerator for Multi-DNN reasoning according to claim 1, characterized in that: The computing core module includes a computing unit array module, a weight data cache module, an input data cache module, an output data accumulation module and a control module; The computing unit array module is used to calculate multiplication and accumulation operations with high parallelism, and the bit width of the computing unit array module determines the bit width of the multiplication and accumulation operations calculated by it; The weight data cache module is used to cache the weight data of the neural network and provide input to the computing unit array module. The bit width of the weight data cache module determines the bit width of the weight data cached therein; The input data cache module is used to cache the input feature map data of the neural network and provide input to the computing unit array module. The bit width of the input data cache module determines the bit width of the input feature map data cached by it; The output data accumulation module is used to accumulate the convolution part and data generated by the computing unit array module, and the bit width of the output data accumulation module determines the bit width of the accumulated convolution part and data; The control module is used to receive control information from the processor, and control the operation of the computing unit array module, the weight data cache module, the input data cache module, the output part and the accumulation module according to the control information.

4. A mixed-precision FPGA hardware accelerator for Multi-DNN reasoning according to claim 3, characterized in that: The computing unit array includes computing units that perform multiplication and accumulation operations. The size of the computing unit array is determined by three dimensional parameters, namely, the feature map size dimension of the convolution operation, the input channel dimension, and the output channel dimension. The bit width of the computing unit array module determines the bit width of the computing unit.

5. A mixed-precision FPGA hardware accelerator for Multi-DNN reasoning according to claim 4, characterized in that: The computing units are divided into three types according to the bit width of the multiplication and accumulation operations performed, namely, high bit width, medium bit width and low bit width. The computing units with high bit width and medium bit width are implemented by DSP resources in the FPGA, and the computing units with low bit width are implemented by logic resources including a lookup table in the FPGA.

6. A mixed-precision FPGA hardware accelerator for Multi-DNN reasoning according to claim 5, characterized in that: When designing the computing unit array, the accumulation cascade of high and medium bit width computing units along the input channel dimension is implemented using fixed cascade connections of DSP resources, and register interrupt logic technology is inserted between the accumulation cascades of low bit width computing units along the input channel dimension.

7. A mixed precision quantization method for Multi-DNN reasoning, applied to a mixed precision FPGA hardware accelerator for Multi-DNN reasoning as claimed in any one of claims 1 to 6, characterized in that: The steps include: Obtaining a neural network that needs mixed precision quantization and a training data set of the neural network; Calculating the quantized sensitivity of the parameters of each layer of the neural network; According to the training data set, simulating the delay of each layer of the neural network running on the corresponding computing core in the hardware accelerator according to different quantization bit widths; According to the quantization sensitivity and the latency of the neural network layer running on the accelerator, the neural network layer is divided into three groups, and three weight quantization bit widths of high, medium and low are set for the three groups respectively to quantize the weight data of the neural network layer; A unified quantization bit width is set for the output feature map data of all layers of the neural network for quantization.

8. The mixed precision quantization method for Multi-DNN reasoning according to claim 7, characterized in that: The three groups are respectively set to three weight quantization bit widths of high, medium and low to quantize the weight data of the neural network layer, and the following quantization bit width allocation rules are adopted: the delay of each layer of the neural network running on the hardware accelerator with different precision and the parallelism parameters of the hardware computing core module are used as constraints, and according to the quantization sensitivity, the quantization bit width of each layer of the neural network is allocated from three bit widths of high, medium and low.

9. The mixed precision quantization method for Multi-DNN reasoning according to claim 8, characterized in that: The constraints for allocating quantization bit widths are specifically as follows: calculating the sum of the running delays of various layers of the neural network that are assigned the same weight quantization bit width on the hardware accelerator, and at the same time calculating the sum of the parallelism of computing cores of different precisions in the hardware accelerator, and constraining the ratio of the sum of the running delays of layers of various quantization bit widths to be similar to the ratio of the sum of the parallelism of computing cores of various bit widths.

10. A mixed precision quantization method for Multi-DNN reasoning according to claim 7, characterized in that: After completing the steps of quantizing the neural network, quantization-aware training is used to fine-tune the quantized neural network and restore the accuracy of neural network reasoning.

Images