A Mixed-Precision FPGA Hardware Accelerator and Quantization Method for Multi-DNN Inference

Abstract

This invention discloses a mixed-precision FPGA hardware accelerator and quantization method for Multi-DNN inference, relating to the field of neural network accelerator technology. It includes a processor, multiple computing core modules, a DDR module, and an intermediate data cache module. Each computing core module supports one of three computational precisions: high, medium, and low. The computing unit array module within the core module consists of computing units performing multiplication and accumulation operations with three different bit widths. The computing units are implemented using different resources within the FPGA. The quantization method uses three bit widths (high, medium, and low) to quantize the neural network weight data layer by layer, and uses one bit width to quantize the intermediate feature map data of the neural network layer. The hardware accelerator and quantization method proposed in this invention can improve the hardware utilization, throughput, and energy efficiency of the accelerator, enabling parallel processing of mixed-precision quantized neural network loads and optimizing memory access bandwidth and energy efficiency for Multi-DNN inference.

Description

Technical Field

[0001] This invention relates to the fields of FPGA and neural network accelerator technology, and more specifically, to a mixed-precision FPGA hardware accelerator and quantization method for Multi-DNN inference. Background Technology

[0002] In recent years, the development of deep neural networks has greatly enhanced the performance of various artificial intelligence systems, such as image processing, speech recognition, and natural language processing. With advancements in algorithm models, the complexity of deep neural networks has also increased, resulting in outstanding performance across various tasks, surpassing the performance of traditional machine learning methods. Therefore, deep neural networks have become one of the core technologies in emerging applications across various related fields, including embedded systems, mobile devices, and the cloud.

[0003] As the application scenarios for deep neural networks (DNNs) increase, researchers are further integrating multiple DNNs into the same application to meet more complex needs. In modern embedded systems, new and complex applications such as autonomous driving and augmented reality (AR) require the simultaneous operation of multiple different DNNs. In particular, in cloud data centers, handling a large number of DNN inference requests from users also necessitates the simultaneous operation of multiple different DNNs on hardware. This deep neural network inference scenario for multi-tenant request processing is called a Multi-DNN scenario.

[0004] However, there is limited research on enhancing Multi-DNN accelerators with novel quantization methods. This is because each neural network has different tolerances to quantization techniques, and the accuracy degradation caused by a fixed single quantization technique can vary greatly across different neural networks. Supporting flexible quantization techniques with hardware presents significant challenges.

[0005] The paper Spantidi O, Zervakis G, Alsalamin S, et al. Targeting dnn inference via efficient utilization of heterogeneous precision dnn accelerators[J]. IEEE Transactions on Emerging Topics in Computing, 2022, 11(1): 112-125 proposes a Multi-DNN accelerator system that employs quantization techniques to enhance the accelerator's energy efficiency. However, the application of quantization techniques in its design is relatively simple, merely using a scheme that employs a uniform quantization bit width for all layers to quantize the neural network. This significantly limits the accuracy of inference, and the accelerator hardware architecture only supports processing the entire network on the same computing core, resulting in low flexibility and resource utilization.

[0006] Therefore, for Multi-DNN accelerators, we need to design more flexible and resource-efficient hardware to support the use of more novel quantization schemes in Multi-DNN scenarios. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the above-mentioned background technology and provide a hardware accelerator and corresponding quantization method for accelerating Multi-DNN inference on FPGA using hybrid quantization technology, so as to achieve better hardware utilization, throughput and energy efficiency.

[0008] To achieve the above objectives, the present invention provides a mixed-precision FPGA hardware accelerator for Multi-DNN inference.

[0009] The technical solution of the present invention is as follows:

[0010] A mixed-precision FPGA hardware accelerator for Multi-DNN inference includes a processor, an intermediate data cache module, a DDR module, and several computing core modules.

[0011] The plurality of computing core modules are used to perform convolution operations of various specific precisions, and the number of computing core modules used to perform each specific precision convolution operation is at least one.

[0012] The intermediate data caching module is used to cache the feature map data generated by intermediate neural network computations in the computing core module.

[0013] The DDR module is used to store data other than the feature map data generated by the intermediate computation of the neural network.

[0014] The processor is used to control the operation of various modules in the hardware accelerator.

[0015] As a preferred embodiment, the computing core module is divided into three types according to the supported convolution operation precision: high precision, medium precision, and low precision computing core module. Different types of computing core modules have the same sub-module types and data path structure. However, the operation bit width and data path bit width of the sub-modules of different types of computing core modules are different, corresponding to high bit width, medium bit width, and low bit width, respectively.

[0016] As a preferred embodiment, the core computing module includes a computing unit array module, a weight data cache module, an input data cache module, an output data accumulation module, and a control module;

[0017] The computing unit array module is used to perform multiply-accumulate operations with high parallelism, and the bit width of the computing unit array module determines the bit width of its multiply-accumulate operations.

[0018] The weight data caching module is used to cache the weight data of the neural network and provide it as input to the computing unit array module. The bit width of the weight data caching module determines the bit width of the weight data it caches.

[0019] The input data caching module is used to cache the input feature map data of the neural network and provide it as input to the computing unit array module. The bit width of the input data caching module determines the bit width of the input feature map data it caches.

[0020] The output data accumulation module is used to accumulate the convolution part and data generated by the computing unit array module. The bit width of the output data accumulation module determines the bit width of the convolution part and data it accumulates.

[0021] The control module is used to receive control information from the processor and control the operation of the computing unit array module, weight data cache module, input data cache module, output section and accumulation module according to the control information.

[0022] As a preferred embodiment, the computational unit array includes computational units for performing multiplication-accumulation operations. The size of the computational unit array is determined by three dimensional parameters: the feature map size dimension for convolution computation, the input channel dimension, and the output channel dimension. The bit width of the computational unit array module determines the bit width of the computational unit.

[0023] As a preferred embodiment, the computing units are divided into three types according to the bit width of the multiply-accumulate operation: high bit width, medium bit width, and low bit width. The high bit width and medium bit width computing units are implemented by DSP resources in the FPGA, while the low bit width computing units are implemented by logic resources in the FPGA including lookup tables.

[0024] As a preferred solution, when designing the computing unit array, the accumulation cascade of high and medium bit-width computing units along the input channel dimension is implemented using fixed cascade connections of DSP resources, and register interruption logic is inserted between the accumulation cascades of low bit-width computing units along the input channel dimension.

[0025] To achieve the objectives of the invention, this invention also provides a mixed-precision quantization method for Multi-DNN inference, applied to a mixed-precision FPGA hardware accelerator for Multi-DNN inference as described in the above technical solution. The quantization method includes:

[0026] Obtain the neural network that needs to be quantized with mixed precision and the training dataset of the neural network;

[0027] Calculate the quantization sensitivity of the parameters of each layer of the neural network;

[0028] Based on the training dataset, the latency of each layer of the simulated neural network running on the corresponding computing core in the hardware accelerator with different quantization bit widths is simulated.

[0029] Based on the quantization sensitivity and the latency of the neural network layer running on the accelerator, the neural network layer is divided into three groups, and high, medium and low weight quantization bit widths are set for the three groups respectively to quantize the weight data of the neural network layer.

[0030] Quantize the output feature map data of all layers of the neural network by setting a uniform quantization bit width.

[0031] As a preferred embodiment, the three groups are respectively set with high, medium and low weight quantization bit widths to quantize the weight data of the neural network layers. The following quantization bit width allocation rule is adopted: using the latency of each layer of the neural network running on the hardware accelerator with different precision and the parallelism parameter of the hardware computing core module as constraints, the quantization bit width of each layer of the neural network is allocated from the three bit widths of high, medium and low according to the quantization sensitivity.

[0032] As a preferred embodiment, the constraint on allocating quantization bit width is specifically as follows: calculate the sum of the running delays of each layer of the neural network with the same weight quantization bit width on the hardware accelerator, and at the same time calculate the sum of the parallelism of the computing cores with different precision in the hardware accelerator, so as to constrain the ratio of the sum of the running delays of layers with various quantization bit widths to be similar to the ratio of the sum of the parallelism of the computing cores with various bit widths.

[0033] As a preferred approach, after completing the quantization of the neural network, quantization perception is used to train and fine-tune the quantized neural network to restore the accuracy of neural network inference.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] 1. The computing core module of the hardware accelerator of the present invention adopts multiply-accumulate operation to achieve compatibility with high, medium and low bit width convolution operations, so that one computing core module can support computing with different bit widths, improve the flexibility of the accelerator, and solve the problem of low resource utilization when deploying mixed precision quantization models on hardware accelerators.

[0036] 2. The hardware accelerator of the present invention adopts a structure in which one computing core module supports only one specific bit-width convolution operation. Compared with the structure in which the same computing core module supports different bit-width operations through reconfiguration logic, it can eliminate the hardware resource overhead caused by switching computing core modules between different bit widths, while simplifying the data path and improving the operating frequency of the hardware.

[0037] 3. The design of the high-bit-width and medium-bit-width computing unit arrays in this invention can achieve a higher operating frequency by making full use of the pipeline registers in the DSP and the cascaded fixed connections between DSPs. The low-bit-width multiplication, due to its lower logic level, can achieve the same high operating frequency by effectively utilizing configurable logic resource blocks, including lookup tables.

[0038] 4. The quantization method of the present invention balances the computational load of each computing core module during the operation of the hardware accelerator by controlling the ratio of the three quantization bit widths of the neural network layer, thereby improving the hardware utilization of the computing core module.

[0039] 5. The quantization method of the present invention uses three quantization bit widths (high, medium, and low) to quantize the weight data of the neural network, which takes into account both the accuracy of the quantized neural network model and the throughput and energy efficiency of the neural network deployed on the hardware accelerator for inference. Attached Figure Description

[0040] Figure 1 This is an overall structural framework diagram of a mixed-precision FPGA hardware accelerator for Multi-DNN inference, according to an embodiment of this application.

[0041] Figure 2 This is a schematic diagram of the computing unit array in the hardware accelerator of an embodiment of this application;

[0042] Figure 3 This is a schematic diagram illustrating the implementation of multiplication of two 8-bit signed numbers by a 5-bit unsigned number in a DSP, as an embodiment of this application.

[0043] Figure 4 This is a schematic diagram illustrating the implementation of a DSP in which four 4-bit signed numbers are multiplied by 5-bit unsigned numbers, according to an embodiment of this application. Detailed Implementation

[0044] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0045] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0046] Example 1

[0047] Please see Figure 1 Embodiment 1 of this application provides a mixed-precision FPGA hardware accelerator for Multi-DNN inference, comprising: multiple computational core modules with different computational precisions, an intermediate data cache module, a DDR module, and a processor for controlling the operation of the hardware accelerator. The computational core modules are used to perform convolution operations of various specific precisions, and at least one computational core module is used for each specific precision convolution operation. The intermediate data cache module is used to cache feature map data generated by intermediate computations of the neural network in the computational core modules. The DDR module is used to store other data besides the feature map data generated by intermediate computations of the neural network. The processor is used to control the operation of each module in the hardware accelerator. The processor issues execution instructions to the computational core modules via a PCIe bus connected to an AXI bus. The computational core modules communicate with the intermediate data cache module and the DDR module via the AXI bus.

[0048] The core computation module includes the following modules: computation unit array module, weight data cache module, input data cache module, output data accumulation module, and control module.

[0049] The computation unit array module is used to perform multiply-accumulate operations with high parallelism. The bit width of the computation unit array module determines the bit width of the computation unit and its multiply-accumulate operations. The weight data cache module caches the weight data of the neural network, providing input to the computation unit array module. The bit width of the weight data cache module determines the bit width of the cached weight data. The input data cache module caches the input feature map data of the neural network, providing input to the computation unit array module. The bit width of the input data cache module determines the bit width of the cached input feature map data. The output data accumulation module accumulates the convolutional parts and data generated by the computation unit array module. The bit width of the output data accumulation module determines the bit width of the accumulated convolutional parts and data. The control module receives control information from the processor and controls the operation of the computation unit array module, weight data cache module, input data cache module, and output data accumulation module according to the control information.

[0050] To support mixed-precision quantization for neural network inference, the computational core module described in Embodiment 1 has the following characteristics: the accelerator contains computational core modules with three operational precisions: high, medium, and low. Each precision can have one or more computational core modules. Each computational core module supports a specific precision of convolution operation, with the three types supporting high, medium, and low convolution operation precisions respectively. The sub-module types and data path structures within the computational core modules of different precisions are identical; the only difference between each computational core module is the bit width of its sub-modules and the bit width of its data path.

[0051] The precision of convolution operations supported by the core computation module determines the bit width of its modules. Specifically, the computation units in the computation unit array support multiply-accumulate operations with this specific bit width, the weight data cache module supports weight data caching with this specific bit width, the input data cache module supports input data caching with this specific bit width, and the output data accumulation module supports partial sum accumulation with this specific bit width.

[0052] Since padding lower-bit-width data with zeros can increase the data bit width, by using high-bit-width zero-padding, high-bit-width computational units can be compatible with low-bit-width multiplication and accumulation operations. This allows high-precision computational core modules to run low-precision convolution operations, providing compatibility for high-precision computational cores and improving hardware utilization.

[0053] The workflow of the core computing module is as follows:

[0054] 1. The control module receives the neural network layer parameters to be calculated and the data storage address configured by the processor.

[0055] 2. The control module notifies the weight data caching module to retrieve the neural network layer weight data to be calculated from the DDR module for caching, and notifies the input data caching module to retrieve the neural network layer input data to be calculated from the intermediate data caching module for caching.

[0056] 3. The control module notifies the computing unit array to start computing.

[0057] 4. The control module instructs the output data accumulation module to accumulate the output data of the computing unit array, and after the accumulation is completed, the data is output to the intermediate data cache module for storage.

[0058] The computing units in the computing unit array have the following characteristics: They have three bit widths: high, medium, and low, each capable of performing multiplication and accumulation operations. To make balanced use of the different logic resources in the FPGA, the high-bit-width computing unit array module utilizes the FPGA's DSP resources to implement the high-bit-width multiplication and accumulation computing unit; the medium-bit-width computing unit array module utilizes the FPGA's DSP resources to implement the medium-bit-width multiplication and accumulation computing unit; and the low-bit-width computing unit array module utilizes the FPGA's lookup table and other logic resources to implement the low-bit-width multiplication and accumulation computing unit.

[0059] High- and mid-width computing units are implemented using DSP resources within the FPGA. When designing the computing unit array, the accumulation cascade of computing units along the input channel dimension utilizes fixed cascade connections of the DSP resources, achieving a higher operating frequency. Low-width computing units are implemented using lookup table resources within the FPGA. When designing the computing unit array, register interruption logic is inserted between the accumulation cascades of computing units along the input channel dimension, achieving a higher operating frequency.

[0060] The size of the computational unit array is determined by three parameters: Uhw, Uic, and Uoc, which correspond to the feature map size dimension, input channel dimension, and output channel dimension of the convolution operation, respectively.

[0061] The workflow of the computation unit array is as follows: Upon receiving the start signal, the weight data is first preset into the computation unit array via a shift register. Then, the input data enters the array to begin computation. The input data enters the computation unit array with a delay of 0-Uic clock cycles along the Uic dimension. Multiplication and accumulation operations are performed unit by unit along the Uic dimension, and the last stage computation unit outputs the accumulated partial sum. This process completes a portion of the computation for this neural network layer, repeating until the computation of a single layer is complete.

[0062] When the hardware accelerator receives a set of neural network inference requests, the processor allocates each computing core to complete the computation of a certain neural network layer according to the bit width of each neural network layer to be inferred and the computation bit width supported by each computing core, until all neural network inference requests have been processed.

[0063] Example 2

[0064] This embodiment 2, based on the mixed-precision FPGA hardware accelerator for Multi-DNN inference in embodiment 1, provides a mixed-precision quantization method for Multi-DNN inference, as follows:

[0065] The mixed-precision quantization method for Multi-DNN inference provided in Embodiment 2 has the following characteristics: it uses three bit widths (high, medium, and low) to quantize the weight data of the neural network on a layer-by-layer basis, and uses one bit width to quantize the intermediate feature map data of all layers of the neural network. The selection of the quantization bit width for the weight data of each layer of the neural network is constrained by the latency of each layer of the neural network running on the hardware accelerator at different precisions and the parallelism parameter of the hardware computing core. Based on the quantization sensitivity obtained from the analysis of each layer of the neural network, the quantization bit width of each layer of the neural network is allocated from the three bit widths: high, medium, and low.

[0066] The constraint on the selection of quantization bit width for each layer of a neural network is to calculate the sum of the running latency of each layer with the same weight and quantization bit width on the hardware accelerator, and the sum of the parallelism of the computing cores with different precision in the hardware accelerator. The constraint is to ensure that the ratio of the total running latency of layers with various quantization bit widths is close to the ratio of the total parallelism of computing cores with various bit widths.

[0067] After obtaining the quantization bit width of the weight data of each layer of the neural network using the above method, the network is quantized using the obtained quantization bit width, and the quantized network is fine-tuned by quantization-aware training to restore the network's inference accuracy.

[0068] The process of the mixed-precision quantization method for Multi-DNN inference is as follows:

[0069] 1. Obtain the neural network that needs to be quantized with mixed precision and the training dataset of the neural network;

[0070] 2. Calculate the quantization sensitivity of the parameters of each layer in the neural network;

[0071] 3. Based on the training dataset, simulate the latency of each layer of the neural network running on the corresponding computing core in the hardware accelerator with different quantization bit widths;

[0072] 4. Based on the quantization sensitivity and the latency of the neural network layer running on the accelerator, the neural network layer is divided into three groups, and three weight quantization bit widths (high, medium, and low) are set to quantize the weight data of the neural network layer, so that the ratio of the total latency of each group is similar to the throughput ratio of each type of computing core of the hardware accelerator.

[0073] 5. Set a uniform quantization bit width for the output feature map data of each layer of the neural network;

[0074] 6. After quantizing the network according to the quantization bit width set above, perform quantization-aware training on the neural network to fine-tune the quantized model parameters and restore the inference accuracy of the neural network.

[0075] Other technical details and implementation steps of this embodiment 2 are the same as those of embodiment 1, and will not be repeated here.

[0076] Example 3

[0077] This embodiment 3, based on embodiments 1 and 2, further illustrates the specific implementation of a mixed-precision FPGA hardware accelerator and quantization method for Multi-DNN inference according to the present invention, as follows:

[0078] Figure 2 This is a schematic diagram of the hardware structure of the computing unit array. The computing units are represented by PE in the diagram. The diagram does not show the actual number of units, only the connection relationships. Weight data is input from the weight data buffer module and pre-loaded into the computing unit array via shift registers. Computing units with the same input channel index and the same output channel index share the same weight data. Feature map data is input from the input data buffer module. Multiple input feature map data along the input channel dimension are delayed by zero to Uic (input channel dimension) clock cycles via shift registers before entering the computing unit array. Computing units with the same input feature map size index and the same input channel index share the same input feature map data.

[0079] The computational unit array consists of computational units performing multiply-accumulate operations and data paths. The data path structure is identical across computational arrays of different bit widths; only the bit width of the computational units and the bit width of the data paths differ. In this embodiment 3, the high, medium, and low weight bit widths of the convolution operation are 8 bits, 4 bits, and 2 bits, respectively, and the output feature map bit width of the convolution operation is 5 bits. The computational unit array utilizes the logic resources and hardware core structure of the Ultrascale+ series FPGA, enabling the accelerator to achieve high parallelism while maintaining a high operating frequency. When implementing the multiply-accumulate operation of 8-bit high-bit-width weight data multiplied by 5-bit input feature map data, based on the aforementioned characteristic of shared weight data, two adjacent multiply-accumulate units can be mapped to a single DSP. For example... Figure 3 The diagram illustrates the implementation of multiplication of two 8-bit signed numbers by a 5-bit unsigned number in a DSP. The DSP is configured to perform the operation (A+D)*B+P. Two signed weight data are concatenated using (A+D) pre-addition. The B operand receives an unsigned input feature map data. The multiplication output P yields the concatenated sum of the two parts. Since there is a five-bit interval between the two sums, the two sets of sums can be accumulated by cascading the DSP's accumulation input Pi and output Po. When implementing the multiplication and accumulation operation of 4-bit weight data multiplied by 5-bit input feature map data, based on the shared weight data and input feature map data characteristics, four adjacent multiplication and accumulation units can be mapped to a single DSP. For example... Figure 4 The diagram illustrates the implementation of multiplication of four 4-bit signed numbers by 5-bit unsigned numbers in a DSP. The DSP is configured to perform the operation (A+D)*B+P. Two signed weighted data are concatenated using a (A+D) pre-addition. The B operand receives the two concatenated unsigned input feature map data. The multiplication output M yields the two concatenated partial sums. Since there is a two-bit interval between the two partial sums, the accumulation operation of the two sets of partial sums can be completed by cascading the DSP's accumulation input Pi and output Po. When multiplying 2-bit low-width weighted data by 5-bit input feature map data, the computation unit is implemented using resources in a configurable logic block. In this case, the multiplication is decomposed into a logical AND operation and the addition of two 6-bit partial sums, which can be mapped to a lookup table and carry chain in the configurable logic resource block. The accumulation between partial sums can be efficiently implemented using an addition tree with cascaded carry chains.

[0080] By fully utilizing the pipelined registers in the DSP and the cascaded fixed connections between DSPs, high-bit-width and medium-bit-width computational unit arrays can achieve an operating frequency of 300MHz. The low-bit-width multiply-accumulate array utilizes the low-logic-level pipeline after register partitioning to achieve an operating frequency of 300MHz.

[0081] In the workflow of a mixed-precision quantization method for Multi-DNN inference in Example 2, the quantization sensitivity of the weight data of each layer of the neural network in step 2 can be obtained using the second-order partial derivative information of the weight parameters of each layer, that is, the largest eigenvector of the Hessian matrix of the layer weight parameters is used as the index of quantization sensitivity. Alternatively, quantization sensitivity can also be obtained using a simpler computational method, such as calculating the KL divergence between the output of each layer after quantization and the output of each layer before quantization, and using this as the quantization sensitivity index for that layer.

[0082] In the workflow of a mixed-precision quantization method for Multi-DNN inference in Example 2, step 4 involves grouping the neural network layers according to their quantization sensitivity from highest to lowest, and then dividing them into three consecutive groups. The group with high quantization sensitivity is assigned the largest bit width, the group with medium quantization sensitivity is assigned a medium bit width, and the group with low quantization sensitivity is assigned the smallest bit width. The group division is determined by calculating the total latency of each group's neural network layers running on the hardware accelerator at the corresponding quantization bit width. The quantization group division is obtained by constraining the ratio of the total latency of each group to be similar to the ratio of the total parallelism of various computing core modules in the hardware accelerator. Therefore, when the hardware accelerator platform runs multiple neural networks in parallel, the running time of each computing core module of the accelerator tends to be balanced, avoiding the inefficient situation where some computing core modules run for a long time while others are idle. Furthermore, when performing mixed-precision quantization of neural networks, different neural networks have different tolerances to quantization; the quantization strategy can be changed by adjusting the ratio of each group. For example, for networks whose accuracy drops significantly after quantization, the number of low-bit-width quantized network layers can be reduced by decreasing the proportion of low-bit-width groups and increasing the proportion of high-bit-width groups, thereby improving the accuracy of the neural network. For networks whose accuracy drops only slightly after quantization, the number of low-bit-width quantized network layers can be increased by decreasing the proportion of high-bit-width groups and increasing the proportion of low-bit-width groups, thereby improving the operating efficiency of the neural network on the hardware platform.

[0083] In the workflow of a mixed-precision quantization method for Multi-DNN inference in Example 2, the quantization method for weight data in step 4 can use asymmetric maximum-minimum quantization. This involves statistically analyzing the maximum and minimum values ​​(Max and Min) of all weight data in the neural network layer, calculating the range of weight data values ​​(Max-Min). Based on the quantization bits n, we calculate Scale = 2^n / (Max-Min) and Zero = Scale*Min + 2^(n-1). After performing a linear transformation y = Scale*x - Zero on the weight data, the range of y becomes [-2^(n-1), 2^(n-1)-1]. Then, y is rounded to an integer. Besides this, other more complex weight quantization methods can also be used.

[0084] The output data quantization method used in quantization-aware training can use saturated truncation quantization. The output data is saturated and truncated with a cutoff value alpha. Data greater than alpha is assigned the value alpha. Then, Scale = 2^n / alpha is calculated based on the quantization bits n. After performing a linear transformation y = Scale * x on the output data, the range of y becomes [0, 2^n-1]. Then, y is rounded to an integer type to complete the quantization of the output data.

[0085] When processing neural network requests with varying mixed-precision quantization, hardware accelerators can flexibly select the bit width configuration of each core to achieve higher processing efficiency. Taking a hardware accelerator with four computing cores as an example, when processing a group of networks with a high bit-width quantization ratio, the bitstream in the FPGA can be updated using the FPGA's partial reconfiguration function, configuring the accelerator with two high-bit-width cores, one medium-bit-width core, and one low-bit-width core. When processing a group of networks with a high low-bit-width quantization ratio, the accelerator can be partially reconfigured to one high-bit-width core, one medium-bit-width core, and two low-bit-width cores.

[0086] In summary, as can be seen from the above embodiments, the present invention has the following beneficial effects:

[0087] 1. The computing core module of the hardware accelerator of the present invention adopts multiply-accumulate operation to achieve compatibility with high, medium and low bit width convolution operations, so that one computing core module can support computing with different bit widths, improve the flexibility of the accelerator, and solve the problem of low resource utilization when deploying mixed precision quantization models on hardware accelerators.

[0088] 2. The hardware accelerator of the present invention adopts a structure in which one computing core module supports only one specific bit-width convolution operation. Compared with the structure in which the same computing core module supports different bit-width operations through reconfiguration logic, it can eliminate the hardware resource overhead caused by switching computing core modules between different bit widths, while simplifying the data path and improving the operating frequency of the hardware.

[0089] 3. The design of the high-bit-width and medium-bit-width computing unit arrays in this invention can achieve a higher operating frequency by making full use of the pipeline registers in the DSP and the cascaded fixed connections between DSPs. The low-bit-width multiplication, due to its lower logic level, achieves the same high operating frequency by effectively utilizing the lookup table, carry chain and registers in the configurable logic resource block.

[0090] 4. The quantization method of the present invention balances the computational load of each computing core module during the operation of the hardware accelerator by controlling the ratio of the three quantization bit widths of the neural network layer, thereby improving the hardware utilization of the computing core module.

[0091] 5. The quantization method of the present invention uses three quantization bit widths (high, medium, and low) to quantize the weight data of the neural network, which takes into account both the accuracy of the quantized neural network model and the throughput and energy efficiency of the neural network deployed on the hardware accelerator for inference.

[0092] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A mixed-precision FPGA hardware accelerator for Multi-DNN inference, characterized in that, It includes a processor, an intermediate data cache module, a DDR module, and several computing core modules; The plurality of computing core modules are used to perform convolution operations of various specific precisions, and the number of computing core modules used to perform each specific precision convolution operation is at least one. The intermediate data caching module is used to cache the feature map data generated by intermediate neural network computations in the computing core module. The DDR module is used to store data other than the feature map data generated by the intermediate computation of the neural network. The processor is used to control the operation of various modules in the hardware accelerator; The computing core module is divided into three types according to the supported convolution operation precision: high precision, medium precision, and low precision computing core module. Different types of computing core modules have the same sub-module types and data path structure. However, the operation bit width and data path bit width of the sub-modules of different types of computing core modules are different, corresponding to high bit width, medium bit width, and low bit width, respectively. The core computing module includes a computing unit array module, a weight data cache module, an input data cache module, an output data accumulation module, and a control module. The computing unit array module is used to perform multiply-accumulate operations with high parallelism, and the bit width of the computing unit array module determines the bit width of its multiply-accumulate operations. The weight data caching module is used to cache the weight data of the neural network and provide it as input to the computing unit array module. The bit width of the weight data caching module determines the bit width of the weight data it caches. The input data caching module is used to cache the input feature map data of the neural network and provide it as input to the computing unit array module. The bit width of the input data caching module determines the bit width of the input feature map data it caches. The output data accumulation module is used to accumulate the convolution part and data generated by the computing unit array module. The bit width of the output data accumulation module determines the bit width of the convolution part and data it accumulates. The control module is used to receive control information from the processor and control the operation of the computing unit array module, weight data cache module, input data cache module, output section and accumulation module according to the control information.

2. The mixed-precision FPGA hardware accelerator for Multi-DNN inference according to claim 1, characterized in that, The computational unit array includes computational units that perform multiplication and accumulation operations. The size of the computational unit array is determined by three dimensional parameters: the feature map size dimension of the convolution operation, the input channel dimension, and the output channel dimension. The bit width of the computational unit array module determines the bit width of the computational unit.

3. A mixed-precision FPGA hardware accelerator for Multi-DNN inference according to claim 2, characterized in that, The computing units are divided into three types based on the bit width of the multiply-accumulate operations they perform: high bit width, medium bit width, and low bit width. The high bit width and medium bit width computing units are implemented by DSP resources in the FPGA, while the low bit width computing units are implemented by logic resources in the FPGA that include lookup tables.

4. A mixed-precision FPGA hardware accelerator for Multi-DNN inference according to claim 3, characterized in that, When designing the computing unit array, the high and medium bit-width computing units are cascaded along the input channel dimension using fixed cascaded connections of DSP resources, and register interruption logic is inserted between the low bit-width computing units cascaded along the input channel dimension.

5. A mixed-precision quantization method for Multi-DNN inference, applied to a mixed-precision FPGA hardware accelerator for Multi-DNN inference as described in any one of claims 1-4, characterized in that, Includes the following steps: Obtain the neural network that needs to be quantized with mixed precision and the training dataset of the neural network; Calculate the quantization sensitivity of the parameters of each layer of the neural network; Based on the training dataset, the latency of each layer of the simulated neural network running on the corresponding computing core in the hardware accelerator with different quantization bit widths is simulated. Based on the quantization sensitivity and the latency of the neural network layer running on the accelerator, the neural network layer is divided into three groups, and high, medium and low weight quantization bit widths are set for the three groups respectively to quantize the weight data of the neural network layer. Quantize the output feature map data of all layers of the neural network by setting a uniform quantization bit width.

6. A hybrid precision quantization method for Multi-DNN inference according to claim 5, characterized in that, The method involves setting high, medium, and low weight quantization bit widths for the three groups to quantize the weight data of the neural network layers, using the following quantization bit width allocation rule: using the latency of each layer of the neural network running on the hardware accelerator at different precisions and the parallelism parameter of the hardware computing core module as constraints, and allocating the quantization bit width of each layer of the neural network from the three bit widths of high, medium, and low according to the quantization sensitivity.

7. A hybrid precision quantization method for Multi-DNN inference according to claim 6, characterized in that, The constraint on allocating quantization bit width is specifically as follows: calculate the sum of the running latency of each layer of the neural network with the same weight quantization bit width on the hardware accelerator, and at the same time calculate the sum of the parallelism of the computing cores with different precision in the hardware accelerator, so that the ratio of the sum of the running latency of layers with various quantization bit widths is similar to the ratio of the sum of the parallelism of computing cores with various bit widths.

8. A hybrid precision quantization method for Multi-DNN inference according to claim 5, characterized in that, After completing the quantization of the neural network, quantization perception is used to train and fine-tune the quantized neural network to restore the accuracy of neural network inference.

Images