A neural network acceleration hardware architecture and method with quantization bit width dynamic selection

Abstract

The application discloses a kind of quantization bit width dynamic selection neural network acceleration hardware architecture and method, hardware architecture includes global storage module, data scheduling module, local storage module, dynamic quantization prediction controller and computing unit array;The feature map in network is divided into neurons with block as unit, in each block, continue to be divided into group with space dimension as unit, define group as the minimum unit of executing dynamic quantization operation;Each block in all feature maps of target network is configured with a trainable threshold parameter, and according to the given basic quantization bit width and target bit width total quantity constraint, the upper and lower bounds of the sparsity selectable by each block are determined;Establish the training and inference model of dynamic quantization neural network, divide inference into high-precision and low-precision calculation two parts, whether the low-precision calculation is executed is judged by the result of high-precision calculation;On the premise of ensuring that the accuracy is not lost, the inference time in actual hardware is reduced as much as possible.

Description

Technical Field

[0001] The present invention relates to the field of embedded data intelligent processing and artificial intelligence technology; in particular, to a neural network acceleration hardware architecture and method for dynamically selecting quantization bit width. Background Art

[0002] In recent years, deep learning has been widely applied to data processing applications such as image and speech processing. Its powerful data analysis, prediction, modeling, and classification capabilities offer unique advantages in addressing complex problems in the digital power grid. These include image and speech recognition technologies for transmission line inspection image analysis, intelligent monitoring and alarming of distribution rooms and cable corridors, and more accurate assessments of safety risks surrounding power grid equipment. Furthermore, the intelligent data analysis enabled by deep learning can be applied to scenarios such as extracting knowledge from text for power grid regulation and electricity price forecasting. In modern power systems, characterized by the widespread integration of low-inertia renewable energy and a high proportion of power electronic equipment, deep learning can be used to address multi-objective control problems such as voltage control, power flow control, frequency dynamic trend perception, assessment, and control, cascading fault prevention, and security defense. This allows for precise analysis, judgment, and optimization of all aspects of generation, transmission, distribution, and utilization, thereby improving power grid stability. However, the superior performance of neural networks often comes with a significant number of parameters and computational complexity, requiring significant storage space, data bandwidth, and long inference times. This makes deep neural networks difficult to deploy directly in resource-constrained power embedded devices.

[0003] Neural network compression is a major approach to addressing these issues. Quantization is an effective means of compressing neural networks, converting 32-bit floating-point operations in the network into low-precision fixed-point operations, thereby reducing data storage space and computational complexity. Currently, most quantization methods are performed before the network is deployed on the terminal. Specifically, the network's quantization bit width is determined through training, and this quantization bit width calculation is then performed consistently during network inference on the terminal. These methods, such as those described in patents CN110188880A, CN112085181A, CN111563589A, and CN109754074A, are examples. However, given the varying importance of different layers, channels within a layer, and even neurons, it is clearly unreasonable to use a uniform quantization bit width for all data. Patent CN111931906A employs a search approach to determine different quantization bit widths for different layers in the network, allowing important layers to retain relatively large bit widths and less important layers to use smaller quantization bit widths. However, this approach still fails to account for the differences between channels and neurons within the same layer. Furthermore, different network input vectors also vary. For simpler tasks, neural networks can use fewer quantization bit widths to reduce inference time. However, for more complex tasks, a wider quantization bit width should be used to maintain high performance, achieving a better trade-off between network performance and real-time inference time. Therefore, how to dynamically adjust the data bit width in real time during the quantization process while minimizing accuracy degradation is an urgent problem to be solved. Researching a neural network acceleration technology for power edge intelligent computing with dynamically selectable quantization bit width is a key challenge. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a neural network acceleration hardware architecture and method with dynamic selection of quantization bit width, so as to minimize the inference time in actual hardware while ensuring no loss of accuracy.

[0005] Technical solution of the present invention:

[0006] A neural network acceleration hardware architecture with dynamic selection of quantization bit width, comprising:

[0007] Global storage module: used to store data information of each link of the neural network;

[0008] Data scheduling module: used to convert the read data into a bit stream. If the read data width is P bits, it is output in P clock cycles, with 1 bit output per cycle.

[0009] Local storage module: used to store input and output buffer data, also using double buffering to avoid reading and writing blocking;

[0010] Computational unit array: uses bit-serial-based calculations, with an array size of 16x16. The entire array performs calculations for 16 output channels and 4 output neurons in parallel per clock cycle. Each computational unit performs 1 bit of data from 16 input channels in parallel per clock cycle, and operations include location and accumulation.

[0011] Dynamic quantization prediction controller: used to determine whether each block performs low-precision calculations and the sparsity of the execution.

[0012] The operations of the dynamic quantized predictive controller include:

[0013] First, after the high-precision calculation of the current group is completed, the maximum value of the results in each group is taken and stored as the importance value of the group;

[0014] Secondly, after all high-precision parts of the current block are calculated, the L1 norm is calculated according to the maximum value of each group, and the threshold parameters of the corresponding block are read for comparison to determine the sparsity of the current block;

[0015] Finally, the four groups of each spatial dimension in the block are sorted based on the importance value of each group. The number of groups that need to continue to perform low-precision operations is determined according to the sparsity of the block, and the group with the highest sorting order will perform low-precision operations; the predicted execution information is stored in the prediction table, which is used to read the group to be judged during low-precision calculations; the controller predicts the execution status of the current block before the high-precision part of the next block is calculated to ensure that, except for the first block, the calculations of the high-precision and low-precision parts can be executed at intervals of blocks.

[0016] The block structure uses a multi-bank storage method so that adjacent blocks are stored in different banks.

[0017] The global storage module includes weight storage, neuron storage and threshold storage; the weight storage is used to store the weights used in a single layer; the neuron storage module is divided into input neurons and output neurons, and the sign bit, high-precision part and low-precision part of each neuron are stored separately; the threshold storage module is used to store the threshold parameters for judging the sparsity of each block.

[0018] A neural network acceleration method for dynamically selecting quantization bit width, comprising:

[0019] Step 1: Based on the parallel computing characteristics of the deployed hardware architecture, the feature map in the network is divided into neurons in blocks. Within each block, the neurons are further divided into groups in the spatial dimension, and the group is defined as the smallest unit for performing dynamic quantization operations.

[0020] Step 2: Configure a trainable threshold parameter for each block in all feature maps of the target network, and determine the upper and lower bounds of the sparsity that can be selected for each block based on the given basic quantization bit width and the target bit width total constraint;

[0021] Step 3: Establish a dynamic quantized neural network training and inference model, divide the inference into high-precision and low-precision calculations, and determine whether to perform low-precision calculations based on the results of high-precision calculations. If the current block is identified as a non-important block, skip the low-precision calculation; otherwise, perform low-precision calculations.

[0022] Step 3 also includes: determining the number and position of the groups executed in the block according to the high-precision calculation result; and obtaining an optimized model through training according to a given target bit width constraint.

[0023] In step 1, the sizes of the blocks and groups are adjusted according to the hardware computing characteristics.

[0024] The selectable sparsity in step 2 is [0, 0.25, 0.5, 0.75, 1].

[0025] The specific implementation method of step 3 includes: establishing a dynamic quantization inference model of a neural network based on the feature maps obtained from the defined groups and blocks, dividing the input neurons into a high-precision part and a low-precision part, first performing the calculation of the high-precision part, and then judging whether to perform the calculation of the low-precision part for each block based on the calculation result of the high-precision part; the judgment process includes:

[0026] Step 3.1: Take the maximum value of the results in each group as the importance value representative of the group;

[0027] Step 3.2: Calculate the L1 norm of the group maximum value of the block, that is, the sum of the absolute values ​​of the maximum values ​​of each group, and compare it with the corresponding threshold parameter. If it is greater than the threshold, the sparsity of the block is the lower bound, otherwise the upper bound sparsity is selected;

[0028] Step 3.3: Sort the four groups in each spatial dimension within the block. The sorting is based on the importance value of each group. The number of groups that need to continue to perform low-precision operations is determined based on the sparsity of the block. The group with the highest sorting order will perform low-precision operations.

[0029] Step 3.4: Add the results of the low-precision operation and the high-precision operation to obtain the final operation result of the layer.

[0030] The dynamic quantization neural network training and inference model trains threshold parameters and network quantization data under a given target computational constraint, and obtains an optimized network model while meeting the target computational constraint. The objective function of the training is defined as:

[0031] min L+lambda R(theta)

[0032] Wherein, L is the training loss function of the original target network, theta is the threshold parameter, R(theta) is the hardware constraint function, lambda is the parameter balancing the two, and R(theta) is expressed as:

[0033] R(theta) = log((f(theta)-T*r) / T+1)

[0034] The formula adopts the log form, and the target is to make the output of the prediction network, i.e. the predicted hardware performance, close to the preset hardware performance T*r, wherein T is the original network calculation amount, r is the preset target ratio, and the input f(theta) is the current network calculation amount related to the threshold parameter.

[0035] The beneficial effects of the present application are:

[0036] The quantization method of the present application considers the importance difference between different neurons and automatically learns the real-time quantization bit width allocation of different neurons through training, so that the performance of the quantization network can be improved under the given calculation amount or inference time constraint compared with the static quantization method.

[0037] The optimization in the present application considers the parallel computing characteristics of the hardware, performs dynamic quantization selection in units of blocks, avoids the problem of low utilization rate of calculation units caused by irregular dynamic quantization, and realizes real acceleration effect through the design of a corresponding hardware accelerator.

[0038] Under the premise of ensuring the accuracy, the inference time in the actual hardware is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 It is a general schematic diagram of the hardware architecture of the present application;

[0040] Figure 2 It is a structural schematic diagram of the block and group of the present application;

[0041] Figure 3 It is a network inference model schematic diagram of the dynamic selection of quantization bit width of the present application;

[0042] Figure 4 It is a schematic diagram of the bit serial calculation unit of the present application;

[0043] Figure 5 It is a schematic diagram of the execution order of high and low precision calculation. DETAILED DESCRIPTION

[0044] To achieve the purpose of the present application, the present application includes the following operation steps:

[0045] Step 1: Based on the parallel computing characteristics of the deployed hardware, divide the neurons in the feature map of the network into blocks. In each block, continue to divide the neurons into groups in the spatial dimension, and define the group as the smallest unit for performing dynamic quantization operations.

[0046] Step 2: Configure a trainable threshold parameter for each block in all feature maps of the target network, and determine the upper and lower bounds of the sparsity that can be selected for each block based on the given basic quantization bit width and the total target bit width constraints.

[0047] Step 3: Build a dynamically quantized neural network training and inference model. This model divides inference into high-precision and low-precision calculations. The high-precision calculation results determine whether to execute the low-precision calculations. If the current block is deemed unimportant, the low-precision calculations are skipped. Otherwise, the low-precision calculations are executed, and the number and location of groups executed within the block are determined based on the high-precision calculation results. The optimized model is obtained through training based on the given target bit width constraints.

[0048] Step 4. Design a neural network hardware accelerator based on bit-serial computing units, which mainly includes a global storage module, a data scheduling module, a local storage module, a dynamic quantization prediction controller, and a computing unit array. A high- and low-precision distributed storage method for neurons is designed to support the dynamic selection of quantization bit width to achieve acceleration of real-time inference time.

[0049] Software model:

[0050] First, according to Figure 2 As shown in the structure, each layer of the feature map in the network is divided into blocks. Taking a 4x4x16 block as an example, the spatial dimension of each block is further divided into groups. The size of each group is 2x2. The size of the block and group can be adjusted accordingly based on the hardware computing characteristics.

[0051] A trainable threshold parameter is configured for each block in all feature maps of the target network. Based on the given base quantization bit width and the total target bit width constraint, the upper and lower bounds of the selectable sparsity of each block are determined. The selectable sparsity is [0, 0.25, 0.5, 0.75, 1].

[0052] Based on the feature maps obtained from the defined groups and blocks, a dynamic quantization inference model of the neural network is established, such as Figure 3 As shown. Taking convolution as an example, the input neurons are divided into high-precision and low-precision parts. The high-precision part is calculated first, and for each block, it is determined whether to perform the low-precision part calculation based on the calculation result of the high-precision part. The judgment process is as follows:

[0053] 1) Take the maximum value of the results in each group as the importance value of the group

[0054] 2) Calculate the L1 norm value of the group maximum value of the block (i.e. the sum of the absolute values of each group maximum value), compare it with the corresponding threshold value parameter, if greater than the threshold value, the sparsity of the block is the lower bound, otherwise select the upper bound sparsity.

[0055] 3) Sort the 4 groups of each spatial dimension in the block according to the importance value of each group, determine the number of groups that need to continue to perform low-precision operation according to the sparsity of the block, and the group with the highest sorting order will perform low-precision operation.

[0056] Add the results of the executed low-precision operation and the high-precision operation, which is the final operation result of the layer.

[0057] According to the above forward inference process, a network operation model is established, and under the given target calculation amount constraint, the threshold value parameter and the network quantization data are trained, and under the condition of meeting the target calculation amount, an optimized network model is obtained. The objective function of the training is defined as:

[0058] min L+λR(θ)

[0059] Where L is the training loss function of the original target network, θ is the threshold value parameter, R(θ) is the hardware constraint function, and λ is a parameter balancing the two. R(θ) is expressed as:

[0060] R(θ)=log((f(θ)-T*r) / T+1)

[0061] The formula adopts the log form, and the goal is to make the output of the prediction network, i.e. the predicted hardware performance (inference time and power consumption, etc.), close to the preset hardware performance T*r, where T is the original network calculation amount, r is the preset target ratio, and the input f(θ) is the current network calculation amount related to the threshold value parameter.

[0062] Hardware design:

[0063] According to the above software model, a corresponding hardware accelerator is designed to support dynamic quantization selection, and the overall architecture is as shown in Figure 1 The main modules include global storage module, data scheduling module, local storage module, dynamic quantization prediction controller, and calculation unit array.

[0064] The global storage module is divided into weight storage, neuron storage and threshold storage. The weight storage is used to store the weights used by a single layer, which is written by the off-chip memory. In order to avoid read-write blocking, double buffering is adopted. The neuron storage module is divided into input neurons and output neurons. In order to improve the utilization rate of reading neurons, the sign bit, high-precision part and low-precision part of each neuron are stored in three storage areas respectively. The threshold storage module is used to store the threshold value parameter for judging the sparsity of each block.

[0065] The data scheduling module is used to convert the read data into a bit stream. If the read data bit width is P bits, it is output in P clock cycles, with 1 bit output in each cycle.

[0066] The local storage module is used to store input and output buffer data, and also adopts double buffering to avoid reading and writing blocking.

[0067] The computing unit array uses bit serial-based computing, and its structure is as follows: Figure 4 As shown, the array size is 16x16. The entire array performs calculations on 16 output channels and 4 output neurons in parallel per clock cycle. Each computing unit performs 1-bit data of 16 input channels in parallel per clock cycle. Operations include location and accumulation.

[0068] The dynamic quantization prediction controller is used to determine whether each block performs low-precision calculations and the sparsity of the execution. Its main operations include first taking the maximum value of the result in each group after the high-precision calculation of the current group is completed, representing the importance value of the group and storing it. Secondly, after all the high-precision parts of the current block are calculated, the L1 norm is calculated based on the maximum value of each group, and the threshold parameters of the corresponding block are read for comparison to determine the sparsity of the current block. Finally, the four groups of each spatial dimension in the block are sorted, and the sorting is based on the importance value of each group. The number of groups that need to continue to perform low-precision operations is determined according to the sparsity of the block, and the group with the highest sorting order will perform low-precision operations. The information of the predicted execution is stored in the prediction table, which is used to read the group to be judged during low-precision calculations. Figure 5 As shown, the controller needs to predict the execution status of the current block before the high-precision part of the next block is calculated to ensure that the calculation of the high-precision and low-precision parts can be executed at intervals of blocks except for the first block.

[0069] To avoid the situation where data is stored at adjacent addresses each time a neuron is read, which would require multiple clock cycles to complete the read and block the calculation, a multi-bank storage method is adopted in the block structure so that adjacent blocks are stored in different banks, ensuring that data reading can be completed within a single clock cycle.

[0070] During the implementation process, the software experimental environment for verifying this method is Pytorch, the datasets used include the CIFAR-10 database and the ImageNet database, and the deep neural network models used include ResNet-18 / 20 / 56 and vgg16.

Claims

1. A neural network acceleration hardware architecture with dynamic selection of quantization bit width, characterized by: It includes: Global storage module: used to store data information of each link of the neural network; Data scheduling module: used to convert the read data into a bit stream. If the read data width is P bits, it is output in P clock cycles, with 1 bit output in each cycle. Local storage module: used to store input and output buffer data, also using double buffering to avoid reading and writing blocking; Computational unit array: uses bit-serial-based calculations, with an array size of 16x16. The entire array performs calculations for 16 output channels and 4 output neurons in parallel per clock cycle. Each computational unit performs 1 bit of data from 16 input channels in parallel per clock cycle, and operations include location and accumulation. Dynamic quantization prediction controller: used to determine whether each block performs low-precision calculations and the sparsity of the execution; wherein, the operation of the dynamic quantization prediction controller includes: first, after the high-precision calculation of the current group is completed, the maximum value of the result in each group is taken, and stored as the importance value of the group; secondly, after all the high-precision parts of the current block are calculated, the L1 norm is calculated according to the maximum value of each group, and the threshold parameters of the corresponding block are read for comparison to determine the sparsity of the current block; finally, the four groups of each spatial dimension in the block are sorted, and the sorting is based on the importance value of each group. The number of groups that need to continue to perform low-precision operations is determined according to the sparsity of the block, and the group with the highest sorting order will perform low-precision operations; the information of predicted execution is stored in the prediction table, which is used to read the group to be executed during low-precision calculation; the controller predicts the execution status of the current block before the high-precision part of the next block is calculated to ensure that the calculation of the high-precision and low-precision parts can be executed at intervals of blocks except for the first block.

2. The neural network acceleration hardware architecture with dynamic selection of quantization bit width according to claim 1, characterized in that: The block structure uses a multi-bank storage method so that adjacent blocks are stored in different banks.

3. The neural network acceleration hardware architecture with dynamic selection of quantization bit width according to claim 1, characterized in that: The global storage module includes weight storage, neuron storage and threshold storage; the weight storage is used to store the weights used in a single layer; the neuron storage module is divided into input neurons and output neurons, and the sign bit, high-precision part and low-precision part of each neuron are stored separately; the threshold storage module is used to store the threshold parameters for judging the sparsity of each block.

4. A neural network acceleration method with dynamic selection of quantization bit width, characterized by: A neural network acceleration hardware architecture for dynamic selection of quantization bit width according to any one of claims 1 to 3; comprising: Step 1: Based on the parallel computing characteristics of the deployed hardware architecture, the feature map in the network is divided into neurons in blocks. Within each block, the neurons are further divided into groups in the spatial dimension, and the group is defined as the smallest unit for performing dynamic quantization operations. Step 2: Configure a trainable threshold parameter for each block in all feature maps of the target network, and determine the upper and lower bounds of the sparsity that can be selected for each block based on the given basic quantization bit width and the target bit width total constraint; Step 3: Establish a dynamic quantized neural network training and inference model, divide the inference into high-precision and low-precision calculations, and determine whether to perform low-precision calculations based on the results of high-precision calculations. If the current block is identified as a non-important block, skip the low-precision calculation; otherwise, perform low-precision calculations.

5. The neural network acceleration method for dynamically selecting quantization bit width according to claim 4, characterized in that: Step 3 also includes: determining the number and position of the groups executed in the block according to the high-precision calculation result; and obtaining an optimized model through training according to a given target bit width constraint.

6. The neural network acceleration method for dynamically selecting quantization bit width according to claim 4, characterized in that: In step 1, the sizes of the blocks and groups are adjusted according to the hardware computing characteristics.

7. The neural network acceleration method for dynamically selecting quantization bit width according to claim 4, characterized in that: The selectable sparsity in step 2 is [0, 0.25, 0.5, 0.75, 1].

8. The neural network acceleration method for dynamically selecting quantization bit width according to claim 4, characterized in that: The specific implementation method of step 3 includes: establishing a dynamic quantization inference model of a neural network based on the feature maps obtained from the defined groups and blocks, dividing the input neurons into a high-precision part and a low-precision part, first performing the calculation of the high-precision part, and then judging whether to perform the calculation of the low-precision part for each block based on the calculation result of the high-precision part; the judgment process includes: Step 3.1: Take the maximum value of the results in each group as the importance value representative of the group; Step 3.2: Calculate the L1 norm of the group maximum value of the block, that is, the sum of the absolute values ​​of the maximum values ​​of each group, and compare it with the corresponding threshold parameter. If it is greater than the threshold, the sparsity of the block is the lower bound, otherwise the upper bound sparsity is selected; Step 3.3: Sort the four groups in each spatial dimension within the block. The sorting is based on the importance value of each group. The number of groups that need to continue to perform low-precision operations is determined based on the sparsity of the block. The group with the highest sorting order will perform low-precision operations. Step 3.4: Add the results of the low-precision operation and the high-precision operation to obtain the final operation result of the layer.

9. The neural network acceleration method for dynamically selecting quantization bit width according to claim 4, characterized in that: The dynamic quantization neural network training and inference model trains threshold parameters and network quantization data under a given target computational constraint, and obtains an optimized network model while meeting the target computational constraint. The objective function of the training is defined as: min L+λR(θ) Where L is the training loss function of the original target network, θ is the threshold parameter, R(θ) is the hardware constraint function, and λ is the parameter that balances the two. R(θ) is expressed as: R(θ)=log((f(θ)-T*r) / T+1) This formula uses log form, and the goal is to make the output of the prediction network, that is, the predicted hardware performance, close to the preset hardware performance T*r, where T is the original network computation amount, r is the preset target ratio, and the input f(θ) is the current network computation amount related to the threshold parameter.

Images