Deep neural network quantization acceleration method based on RISC-V vector
By expanding multiplication and division instructions under the RISC-V architecture and optimizing them in the compiler, the problems of low parallelism and large instruction overhead in the quantization and inverse quantization process are solved, and efficient quantization operations and large-scale data processing are achieved.
Patent Information
- Application Number
- CN202410831667.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-25
- Publication Date
- 2025-04-29
AI Technical Summary
Under the RISC-V architecture, during the quantization and inverse quantization of neural networks, the parallelism is low when performing the division and multiplication operations, and the instruction overhead is high, resulting in additional power consumption and inefficiency.
By expanding the multiplication and division instructions of V expansion, new multiplication and division instructions are designed, and corresponding optimizations are performed in the compiler, so that parallel computing accelerates quantization and inverse quantization operations can be performed under the vector architecture.
Quantitative operations of different quantization scales are realized, which reduces the number of instructions, reduces resource overhead, and improves the efficiency of processing large-scale data.
Smart Images
Figure CN119903880A_ABST
Abstract
Description
Technical Field
[0001] The present invention mainly relates to the fields of computer architecture and microprocessor architecture, and deep learning, and in particular to a deep neural network quantization acceleration method based on RISC-V vectors. Background Art
[0002] Traditionally, complex deep neural network calculations are usually performed in cloud data centers. However, with the increasing popularity of deep learning applications on edge devices, due to the low computing power of edge devices and limitations in memory and power consumption, there is an urgent need to reduce the size of deep neural networks, speed up inference, and reduce power consumption while ensuring computing accuracy. Currently, combining neural network quantization methods with the emerging RISC-V instruction set architecture can effectively address these challenges.
[0003] Quantization technology reduces the requirements for weight and activation accuracy and is one of the effective ways to simplify model complexity. We can directly quantize from an existing floating-point model to map all weights into a range. For example, 8-bit quantization quantizes weights into the range of [0,255] without retraining or rebuilding the model. At the same time, low-precision data can move faster in the processor when participating in activation, increasing cache reuse, improving computing speed, and reducing the memory required to store this data.
[0004] RISC-V is an open source instruction set architecture (ISA). Unlike previous ISAs, RISC-V is modularly designed. At its core is a basic ISA called RV32I, and extended instruction sets are defined for different scenarios. Among them, the V extension (RVV, RISC-V Vector Extension) is a variant of single instruction multiple data (SIMD, Single Instruction Multiple Data) - the vector architecture. When calculating, the vector architecture does not embed opcodes, but separates the length of the vector and the maximum number of operations that can be performed per clock cycle. This design allows for more flexible design of data parallel hardware and generates fewer instructions at compile time compared to traditional SIMD.
[0005] Current research on RISC-V and deep neural networks in academia includes the customized RISC-V vector processor SPARQ of the University of Montreal in Canada, an ISA extension for quantized neural network reasoning proposed by the Swiss Federal Institute of Technology in Zurich, and a barrel-type RISC-V processor designed by the University of Montreal in Canada.
[0006] Based on the design principle of RISC-V instruction set architecture, these current solutions optimize the reasoning performance of quantized deep neural networks from different angles through customized processors or vector coprocessors and customized ISA extensions. However, since these optimization solutions are targeted at different neural networks, they lack certain universality. In particular, when performing quantization and dequantization of neural network weights, they are still processed in non-SIMD form, that is, one instruction can only process one data element. When processing large-scale data, this method requires multiple instructions, which increases time overhead and leads to low efficiency. Summary of the invention
[0007] The purpose of the present invention is to provide a method for accelerating neural network quantization based on RISC-V vectors, aiming to solve the problems of low parallelism, large instruction overhead, and additional power consumption caused by frequent movement of data between memory and computing units when performing related divide-accumulate operations (DIVADD) and multiply-accumulate operations (MAC) when different neural networks are quantized and dequantized under the RISC-V architecture.
[0008] The present invention is implemented in this way: a method for accelerating neural network quantization based on RISC-V vector includes the following steps: Step 1: Divide the vector register stack into blocks according to its size so that its size is consistent with the tensor block length; Step 2: Calculate the quantization scale (△, Quantization Scale) and zero point (z, ZeroPoint) of the divided tensor and save them into a global data structure; Step 3: Expand the multiplication and division instructions of V extension; Step 4: Modify the compiler so that it can recognize the extended multiplication and division instructions and complete the compilation; Step 5: Perform parallel computing to accelerate quantization and dequantization operations under the vector architecture.
[0009] Furthermore, in step 1, the vector register stack is divided into blocks, and the size of the registers after the blocks is consistent with the size of the tensor blocks, thereby reducing the impact of abnormal values in the tensor on quantization.
[0010] Furthermore, in step 2, the maximum and minimum values in the block are calculated for the weights after block division, and then the corresponding quantization scale (△, Quantization Scale) and zero point (z, Zero Point) are calculated according to the Rang range, and indexes are established and saved in an array, which is convenient for quick search and calculation during subsequent dequantization.
[0011] Furthermore, in step 3, the multiplication and division instructions of V extension are extended, and the specific steps are as follows:
[0012] Step 1: According to the RVV 1.0 standard, design new multiplication and addition instructions and division and addition instructions with reference to the multiplication and division instructions in the V instruction, and explain how the instructions correctly complete quantization and dequantization when the weights and quantization scales are assigned to the vector registers and scalar registers;
[0013] Step 2: Describe the planned instruction information according to the compiler backend specification, and explain the instruction structure, the specific meaning of each field, and the instruction function in the description file.
[0014] Furthermore, in step 4, the compiler is expanded, and according to its layered design style, the quantization part of the program is identified from the front end to ensure the correctness of the intermediate process and that the back end can correctly compile into added multiplication and addition instructions and division and addition instructions.
[0015] Furthermore, during the calculation process of step 5, the tensor weights stored in the vector register and the quantization scale and zero point stored in the scalar register are accessed through instruction control to complete the corresponding quantization and dequantization calculations. After the calculation is completed, the weights in the vector register overwrite the original tensor weights.
[0016] The beneficial effects of the present invention are as follows: (1) The present invention supports quantization operations of different quantization scales, and the method is universal; (2) The present invention optimizes the important multiplication and addition and division and addition operations of the quantization process at the instruction level to reduce the number of instructions generated when the program is executed, thereby reducing unnecessary overhead; (3) The present invention uses vector processing to process quantization. When dealing with large-scale data processing, the specialization of the extended instructions and parallel computing make the process without the need for additional instructions, reducing resource overhead. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a flowchart of a RISC-V vector-based accelerated neural network quantization method provided in an embodiment of the present invention.
[0018] Figure 2 This is an example of vector stack segmentation provided by an embodiment of the present invention.
[0019] Figure 3 This is an example of tensor partitioning provided by an embodiment of the present invention.
[0020] Figure 4 It is a mathematical expression for calculating the quantization scale provided by an embodiment of the present invention.
[0021] Figure 5 It is a mathematical expression for calculating the quantized zero point provided by an embodiment of the present invention.
[0022] Figure 6 This is an example of vectorized quantized data provided by an embodiment of the present invention.
[0023] Figure 7 It is a mathematical expression of vectorized quantized data provided by an embodiment of the present invention.
[0024] Figure 8 This is an example of vectorized dequantized data provided by an embodiment of the present invention.
[0025] Fig. 9 It is a mathematical expression of vectorized dequantized data provided by an embodiment of the present invention.
[0026] Fig.10 It is a schematic diagram of the instruction format provided by an embodiment of the present invention.
[0027] Fig.11 This is an example of the process of the vector accelerated neural network quantization method provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0029] Figure 1 The process of the RISC-V vector-based accelerated neural network quantization method of the present invention is shown. As shown in the figure, the present invention is implemented in this way. A RISC-V vector-based accelerated neural network quantization method includes the following steps: S101: Divide the vector register stack into blocks according to its size, and the size of the blocks is consistent with the length of the tensor blocks; S102: Determine the maximum and minimum values of the tensor after block division, and calculate the quantization scale and zero point and save them into a global data structure; S103: Expand the multiplication and division instructions of V expansion; S104: Modify the compiler so that it can recognize the extended multiplication and division instructions and complete the compilation; S105: Perform parallel computing to accelerate quantization and dequantization operations under a vector architecture.
[0030] Furthermore, in step S101, the vector register stack is divided into blocks, and the size of the registers after the blocks is consistent with the size of the tensor blocks, thereby reducing the impact of abnormal values in the tensor on quantization.
[0031] Furthermore, in step S102, the maximum and minimum values in the block are calculated for the weights after block division, and then the corresponding quantization scale (△, Quantization Scale) and zero point (z, Zero Point) are calculated according to the Rang range, and indexes are established and saved in an array, which is convenient for quick search and calculation during subsequent dequantization.
[0032] Furthermore, in step S103, the multiplication and division instructions of V extension are extended:
[0033] Step 1: According to the RVV 1.0 standard, refer to the multiplication and division instructions in the V instruction to design new multiplication and addition instructions and division and addition instructions. When the weights and quantization scales in the first two steps are assigned to the vector register and scalar register, how can the instructions correctly complete quantization and dequantization?
[0034] Step 2: Describe the planned instruction information according to the compiler backend specification, and explain the instruction structure, the specific meaning of each field, and the instruction function in the description file.
[0035] Furthermore, in the step S104, the compiler is expanded, and according to its layered design style, the quantization part of the program is identified from the front end to ensure the correctness of the intermediate process and that the back end can correctly compile into added multiplication and addition instructions and division and addition instructions.
[0036] Furthermore, the step S105 completes the corresponding quantization and dequantization calculations by accessing the tensor weights stored in the vector register and the quantization scale and zero point stored in the scalar register through instruction control. After the calculation is completed, the weights in the vector register overwrite the original tensor weights.
[0037] Example: LLVM consists of a series of functional modules and corresponding tool sets. LLVM mainly involves the middle and back ends of the compiler, and the front end is processed by clang. LLVM adopts a modular design. Unlike the strong coupling of GCC, LLVM is more flexible. The steps for implementing the RISC-V vector-based deep neural network quantization acceleration method in LLVM are as follows:
[0038] Step 1: Divide the vector register stack into blocks according to its size, which is consistent with the length of the tensor block; 32 vector registers and 7 non-privileged CSRs are added in the V extension. After counting the vector register stack size, non-privileged instructions are used to complete the grouping of registers. The purpose of grouping is because there are abnormal values when quantizing tensors. If a quantization scale △ is used as a whole, it will seriously affect the accuracy. Figure 2 The 32 128-bit registers are divided into 4 512-bit registers, so that one instruction can control 4 registers, and 16 weights can be quantized at a time in a model with a weight size of float32.
[0039] Step 2: Calculate the quantization scale (△, Quantization Scale) and zero point (z, ZeroPoint) of the divided tensor and save them into a global data structure; The maximum and minimum values of the divided tensor are calculated, and the quantization scale and zero point are calculated according to different quantization ranges. During the quantization process, the quantization scale will determine how to map the dynamic range of floating-point numbers to a smaller integer range, and when the zero point is in effect, the zero point value is mapped to the quantized integer value. These two important information are saved in a global data structure, and the corresponding index is established to facilitate the search and calculation during the subsequent dequantization.
[0040] Step 3: Expand the multiplication and division instructions of V extension; According to the RVV 1.0 standard, the custom instruction format, namespace, and meaning of each field are expanded in comparison with its multiplication and division instructions to avoid instruction conflicts. The definitions of custom instructions are written using TableGen according to the LLVM framework structure, and the related files are stored in the correct directory structure. The node of the return flag is defined to return the specific identifier of the instruction.
[0041] Step 4: Modify the compiler so that it can recognize the extended multiplication and division instructions and complete the compilation; Because of the separate front-end and back-end compilation mode of LLVM, we process the quantization and dequantization related code parts in the intermediate result IR in this step so that LLVM can correctly convert them into our custom V extension instructions.
[0042] Step 5: Perform parallel computing to accelerate quantization and dequantization operations under the vector architecture; Through the vector register block in step 1, Figure 2 In this example, we use the four registers v0 to v3 as an operation object. According to this size, we load the weight vector of the corresponding size into the register during quantization and dequantization, such as Figure 3 In the float32-bit weight matrix, every 16 elements are divided into a block of 512 bits, which is the same size as the vector register group. In this step, we calculate the maximum value, minimum value, quantization scale and zero point of different blocks respectively, and save the quantization scale and zero point to the global data structure. When executing quantization, each custom instruction performs the same quantization or dequantization calculation process on each element in the register group. After the calculation is completed, the quantization value overwrites the original float32 value, such as Figure 4 Convert float32 data to int8 data to reduce the memory usage of neural network weights, and then dequantize ( Figure 5 )The process is the same.
[0043] The above proposed RISC-V vector-based deep neural network quantization acceleration method has the following execution effects: Figure 6 This method uses the characteristics of the RISC-V vector architecture to optimize and vectorize quantization and dequantization calculations at the instruction level, reducing memory and power consumption overhead, and can support quantization operations of different quantization scales, with strong versatility and memory utilization efficiency.
[0044] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative labor are still within the scope of protection of the present invention.
Claims
1. A deep neural network quantization acceleration method based on RISC-V vector, characterized in that: The RISC-V vector-based deep neural network quantization acceleration method comprises the following steps: Step 1: Divide the vector register stack into blocks according to its size so that its size is consistent with the tensor block length; Step 2: Calculate the quantization scale (Δ, Quantization Scale) and zero point (z, ZeroPoint) of the divided tensor and save them into a global data structure; Step 3: Expand the multiplication and division instructions of V extension; Step 4: Modify the compiler so that it can recognize the extended multiplication and division instructions and complete the compilation; Step 5: Perform parallel computing to accelerate quantization and dequantization operations under the vector architecture.
2. The RISC-V vector-based deep neural network quantization acceleration method according to claim 1, characterized in that: The step 1 divides the vector register stack into blocks, and the size of the registers after the blocks is consistent with the size of the tensor blocks, thereby reducing the impact of abnormal values in the tensor on quantization.
3. The RISC-V vector-based deep neural network quantization acceleration method according to claim 1, characterized in that: In step 2, the maximum and minimum values in the block are calculated for the weights after block division, and then the corresponding quantization scale (Δ, Quantization Scale) and zero point (z, Zero Point) are calculated according to the Rang range, and indexes are established and saved in an array to facilitate quick search and calculation during subsequent dequantization.
4. The RISC-V vector-based deep neural network quantization acceleration method according to claim 1, characterized in that: The step 3 described above expands the multiplication and division instructions of the V expansion, and the specific steps are as follows: Step 1: According to the RVV standard, design new multiplication and addition instructions and division and addition instructions with reference to the 32-bit multiplication and division instructions in the V instruction, keep the flag bit and mask bit unchanged, modify the register bit information, and indicate the vector register and scalar register that need to be operated; Step 2: Describe the planned custom instruction information according to the compiler backend specification, and explain the instruction structure, the specific meaning of each field, and the instruction function in the description file.
5. The RISC-V vector-based deep neural network quantization acceleration method according to claim 1, characterized in that: The step 4 described above expands the compiler and, according to its layered design style, completes the identification of the quantization part of the program from the front end, ensures the correctness of the intermediate process, and ensures that the back end can correctly compile into added multiplication and addition instructions and division and addition instructions.
6. The RISC-V vector-based deep neural network quantization acceleration method according to claim 1, characterized in that: The step 5 completes the corresponding quantization and dequantization calculations by accessing the tensor weights stored in the vector register and the quantization scale and zero point stored in the scalar register through instruction control. After the calculation is completed, the weights in the vector register overwrite the original tensor weights.