AI reasoning acceleration system based on RISC-V extension

Abstract

The invention relates to an AI reasoning acceleration system based on RISC-V extension, which realizes integrated acceleration of operations such as matrix multiplication, activation and quantization by adding a U8 matrix multiplication acceleration component and an LUT (lookup table) component on the basis of RVV and expanding related instructions of an AI special instruction set. The AI special instruction set expands a VMMA instruction and a VLMMP instruction, comprises a coding format, a register interaction rule and a hardware scheduling strategy of the instructions, cooperates with the RVV to realize higher matrix multiplication efficiency, and can complete nonlinear transformation of U8 data in a single cycle when realizing parallel table lookup operation of quantization and activation functions through the LUT lookup table component, so that the efficiency of matrix multiplication is improved. The effect of quantitatively activating zero delay is achieved, the AI reasoning process is remarkably accelerated, and finally the overall effect of high-performance AI reasoning task acceleration is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent computing acceleration technology and relates to an AI reasoning acceleration system based on RISC-V extension. Background Art

[0002] As artificial intelligence (AI) models evolve toward lightweight architectures, complex AI inference tasks are increasingly required on embedded devices, such as smart wearables, industrial control terminals, and autonomous driving ECUs. However, traditional hardware architectures face significant challenges in performance, power consumption, and flexibility. General-purpose processor efficiency bottlenecks exist. Traditional embedded CPUs (such as the ARM Cortex-M series and the RISC-V Base instruction set) lack dedicated instruction support when executing intensive AI operations like matrix operations and quantized activation functions, relying on software simulation and resulting in poor performance that fails to meet real-time requirements. Dedicated accelerators also lack flexibility. This is because fixed architectures struggle to adapt to dynamically changing models and are ecosystem-unfriendly. They require custom software stacks, and independent coprocessors communicate over buses to transfer data, introducing additional latency and power consumption, making them unsuitable for the strict power constraints of embedded devices. The existing RISC-V vector extension standard (RVV) (e.g., version 1.0) is primarily targeted at general-purpose vector computations (such as convolutional neural networks (DNNs)) and is not optimized for AI inference. Firstly, matrix multiplication is inefficient, requiring the combination of multiple VADD and VMUL instructions, resulting in high scheduling overhead. Secondly, activation functions are not sufficiently hardware-based. Common functions such as ReLU and GELU activation functions require conditional branching, impacting pipeline efficiency. Therefore, achieving high-performance acceleration for AI inference tasks has become a current technical challenge. Summary of the Invention

[0003] In response to the problems existing in the above-mentioned traditional technologies, the present invention proposes an AI reasoning acceleration system based on RISC-V extension, which can achieve high-performance AI reasoning task acceleration.

[0004] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: Provides an AI inference acceleration system based on RISC-V extensions, including a vector launch queue, scoreboard, vector decode and launch logic, vector register file read and write control, execution unit, vector register file, AI extension unit, vector load-store data first-in-first-out queue, advanced consistency interface and data cache; The vector decode and transmit logic are respectively connected to the vector transmit queue, vector register file read / write control, and execution unit. The vector transmit queue is connected to the scoreboard. The vector register file is respectively connected to the vector register file read / write control, AI extension unit, and vector load-store data first-in-first-out queue. The vector load-store data first-in-first-out queue is respectively connected to the advanced consistency interface and data cache. The execution unit is connected to the AI ​​extension unit. The vector register file is a unified vector register file with a width of 256 bits, supports mixed data type storage, and has built-in data type marking and overflow protection logic. The AI ​​extension unit consists of a U8 matrix multiplication acceleration unit and a LUT lookup table unit. The U8 matrix multiplication acceleration unit is called by the matrix multiplication and accumulation instruction based on the RISC-V extension to perform 4×4 matrix multiplication parallel operations, and the LUT lookup table unit is called by the load quantization matrix multiplication parameter instruction based on the RISC-V extension to perform element-by-element operations of the U8 data type.

[0005] In one embodiment, the U8 matrix multiplication acceleration component consists of a 4×4 matrix multiplication array, a quantization compensation module, and an INT32->U8 quantization module.

[0006] In one embodiment, the size of a single storage unit of the LUT lookup table component is 256×8 bits.

[0007] In one embodiment, the RISC-V based vector instruction extension uses 32 vector registers, the vector length is 128 bits and the instruction format is based on RVV OP-V encoding.

[0008] One of the above technical solutions has the following advantages and beneficial effects: The aforementioned RISC-V-based AI inference acceleration system achieves integrated acceleration for operations such as matrix multiplication, activation, and quantization by adding a U8 matrix multiplication acceleration component and a LUT lookup table component to RVV and expanding the relevant instructions of the AI-specific instruction set. The AI-specific instruction set extends the matrix multiplication and accumulation instructions and the load quantization matrix multiplication parameter instructions, including their instruction encoding format, register interaction rules, and hardware scheduling strategy. It collaborates with RVV to achieve higher matrix multiplication efficiency. Furthermore, when implementing parallel table lookup operations for quantization and activation functions through the LUT lookup table component, the nonlinear transformation of U8 data can be completed in a single cycle, achieving zero-latency quantization activation, significantly accelerating the AI ​​inference process, and ultimately achieving an overall high-performance acceleration of AI inference tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 Schematic diagram of the overall architecture of an AI inference acceleration system based on RISC-V extension in one embodiment; Figure 2 A schematic diagram of the structure of a U8 matrix multiplication acceleration component in one embodiment; Figure 3 FIG. 1 is a schematic structural diagram of a LUT lookup table component in an embodiment. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the description of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0012] It should be noted that the reference to "embodiment" in this document means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The presentation of this phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It will be understood by those skilled in the art that the embodiments described herein may be combined with other embodiments. The term "and / or" used in the specification of the present invention and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0013] The following describes the implementation of the present invention in detail with reference to the accompanying drawings in the embodiments of the present invention.

[0014] Traditionally, dedicated NPU accelerators use an NPU as a coprocessor attached to a processor core, with a driver driving the NPU to accelerate AI inference tasks. These NPU-specific accelerators require a bus to move data from main memory to the accelerator's local cache, resulting in high bandwidth utilization and rigid accelerator functionality, making them difficult to adapt to AI inference tasks with varying bit widths or dynamic changes.

[0015] The traditional RISC-V Vector Extension (RVV) natively supports matrix multiplication algorithms by leveraging the RVV 1.0 standard instruction set (e.g., through dynamic configuration and the vector multiply-add instruction VSETVL). This RVV-based implementation implements SIMD (single instruction, multiple data) parallel computing, performing batch operations on data using vector registers. However, RVV lacks dedicated instructions and hardware support for operations such as matrix multiplication, quantization, and activation, resulting in redundant algorithm implementations and increased instruction count and power consumption.

[0016] In the design of artificial intelligence (AI) inference acceleration for low-power, high-performance edge intelligent devices, this invention integrates AI-specific instructions, enhances the RISC-V vector extension with customized matrix multiplication, quantization, and activation instructions, and proposes a hybrid pipeline architecture. This architecture maintains backward compatibility and fully implements software reuse, accelerating the efficient execution of intensive tasks such as matrix operations and quantized activation functions in embedded systems, addressing the practical challenges of edge AI applications in terms of performance, power consumption, and flexibility. RISC-V, an open standard instruction set architecture (ISA), has led a new trend in processor design with its modular design and collaborative community. RISC-V RVV (RISC-V Vector) is a vector processing extension of the RISC-V instruction set architecture.

[0017] In one embodiment, Figure 1 As shown, a RISC-V extension-based AI inference acceleration system is provided, including a vector transmit queue, a scoreboard, vector decode and transmit logic, a vector register file read / write control, an execution unit, a vector register file, an AI extension unit, a vector load-store data first-in-first-out queue, an advanced consistency interface, and a data cache. The vector decode and transmit logic are respectively connected to the vector transmit queue, the vector register file read / write control, and the execution unit. The vector transmit queue is connected to the scoreboard. The vector register file is respectively connected to the vector register file read / write control, the AI ​​extension unit, and the vector load-store data first-in-first-out queue. The vector load-store data first-in-first-out queue is respectively connected to the advanced consistency interface and the data cache. The execution unit is connected to the AI ​​extension unit. The vector register file is a unified 256-bit vector register file that supports mixed data type storage and has built-in data type tagging and overflow protection logic. The AI ​​extension unit consists of a U8 matrix multiplication acceleration unit and a LUT lookup table unit. The U8 matrix multiplication acceleration unit is called by the matrix multiplication and accumulation instruction based on the RISC-V extension to perform 4×4 matrix multiplication parallel operations, and the LUT lookup table unit is called by the load quantization matrix multiplication parameter instruction based on the RISC-V extension to perform element-by-element operations of the U8 data type.

[0018] It can be understood that the processor architecture (hardware architecture) includes vector issue queue (VIQ), scoreboard (ScoreBoard), vector decode and issue logic (Vector Decode and Issue Logic), vector register file read / write control (VRF read / write control), execution unit (EX), vector register file (ie VRF), AI extension unit (AI EXT), vector load-store data first-in-first-out queue (Vector LS Data FIFO), advanced consistency interface (ACE) and data cache (Dcache).

[0019] The scoreboard is a hardware mechanism used to dynamically schedule instruction execution. It tracks instruction execution status and monitors operand availability to determine when instructions can be sent to execution units, avoiding data and structural conflicts and increasing parallelism. The vector issue queue (VIQ) temporarily stores vector instructions waiting to be issued. It caches vector instructions retrieved from the instruction cache and, based on specific rules and conditions, sends them to the vector decode and issue logic (VDL) at the appropriate time. The VDL decodes and issues vector instructions, analyzes information such as the instruction operation type and operands, and determines when to issue the instruction to the appropriate execution unit based on status information provided by components such as the scoreboard. EX0 to EX3 represent execution units, respectively. VRF0 to VRF3 represent vector register files, respectively.

[0020] The overall hardware architecture of the AI ​​inference acceleration system based on RISC-V extension is as follows: Figure 1 As shown in the figure, it includes two major improvements: the AI ​​extension unit and the vector instruction extension. The AI ​​extension unit is an existing patented AI module. This embodiment combines the AI ​​extension unit with RISC-V as a RISC-V computing component, and supports calling the computing component by extending the original RISC-V instruction decoder.

[0021] Furthermore, the vector instruction extension supports SIMD parallel computing, includes 32 vector registers (vector length VLEN = 128 bits), and the instruction format is based on RVV OP-V encoding, with a new AI acceleration instruction category. OP-V encoding is a key component of the RVV instruction encoding system. OP-V encoding is specifically used to encode vector operation instructions in RVV. It determines the specific operation type of the instruction, the source and destination of the operands, and the various parameters involved in the operation.

[0022] The new AI acceleration instructions added by the Vector Instruction Extension include the VMMA (Matrix Multiply Accumulate) instruction and the VLMMP (Load Quantized Matrix Multiplication Parameters) instruction. The VMMA (Matrix Multiply Accumulate) instruction is used to implement matrix multiplication and addition operations, specifically multiplying the corresponding elements of two vectors and then accumulating the multiplication results to the corresponding elements of the target vector. An example instruction for the VMMA (Matrix Multiply Accumulate) instruction is as follows: "text Copy Code vmma.vv vd, vs1, vs2" / / Multiply the corresponding elements of the two vectors, and then add the multiplication results to the corresponding elements of the target vector.

[0023] Where text represents text, Copy Code represents copy code, vd represents the destination vector register, which is used to store the final result of matrix multiplication and accumulation, and vs1 and vs2 represent two source vector registers, which are used to provide two sets of data elements respectively.

[0024] The VLMMP (load quantization matrix multiplication parameters) instruction is used to load the quantization matrix multiplication parameters. An example instruction may be as follows: "text Copy Code vlmmp v0 / / " loads the v0 parameter into the quantization matrix multiplication parameter register.

[0025] The AI ​​expansion unit consists of a U8 matrix multiplication accelerator and a LUT lookup table (LUT) component. The U8 matrix multiplication accelerator implements parallel acceleration of 4×4 matrix multiplications in the U8×U8->INT32->U8 data format. It supports asymmetric quantization (asymmetric quantization, which converts floating-point numbers into integers) and can flexibly compute either INT32 (such as the VMMA instruction, which performs vector-matrix multiplication and addition) or U8 (such as the VMMAQ instruction, which performs vector-matrix multiplication and addition followed by quantization) results, depending on the instruction. The LUT component implements element-wise operations (element-wise operations, which perform the same operation on each element in a data structure such as an array, vector, or matrix) on the U8 data type, such as quantization and activation. U8 represents an unsigned 8-bit integer with a value range of 0 to 255. "->INT32" converts the result of the "U8×U8" operation to a signed 32-bit integer. "->U8" then converts the INT32 data back to U8. All instructions in this manual are decoded by the existing instruction decoder and executed by the AI ​​expansion unit to implement the function corresponding to the instruction.

[0026] The aforementioned RISC-V extension-based AI inference acceleration system achieves integrated acceleration for operations such as matrix multiplication, activation, and quantization by adding a U8 matrix multiplication acceleration component and a LUT lookup table component to RVV, and expanding the relevant instructions of the AI-specific instruction set. The AI-specific instruction set extends the VMMA and VLMMP instructions, including their instruction encoding formats, register interaction rules, and hardware scheduling strategies. This system, in conjunction with RVV, achieves higher matrix multiplication efficiency. Furthermore, when implementing parallel table lookups for quantization and activation functions through the LUT lookup table component, it can complete nonlinear transformations of U8 data in a single cycle, achieving zero-latency quantization activation, significantly accelerating the AI ​​inference process, and ultimately achieving an overall high-performance acceleration of AI inference tasks.

[0027] In one embodiment, further, the U8 matrix multiplication acceleration component is as follows Figure 2 As shown in the figure, it consists of a 4×4 matrix multiplication array (i.e., a hardware matrix multiplication calculation unit), a quantization compensation module (i.e., an online hardware compensation unit), and an INT32->U8 quantization module (i.e., a data quantization module). This component is used to perform convolution operations in neural networks. It implements convolution operations by performing matrix multiplication operations on the input feature map and the convolution kernel. The process can be highly parallelized and combined with the quantization compensation module to eliminate low-precision accumulated errors, so it can be executed quickly and efficiently in the accelerator.

[0028] In one embodiment, further, the structure of the LUT lookup table component is as follows: Figure 3 As shown, a single storage unit measures 256×8 bits, enabling integrated parallel execution of quantization, activation functions, and dequantization operations, ensuring acceleration for U8 element-wise operations (element-wise operations, which are performed on each element in an array or tensor, with the results mapped to the element position). The LUT (Lookup Table) component pre-calculates and stores the input-to-output mapping in the form of a lookup table. When processing U8 data types, due to their limited value range, the LUT allows for extremely high computation speeds, accelerating neural network inference.

[0029] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0030] The above embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of the present invention. Therefore, the scope of the present invention shall be determined by the appended claims.

Claims

1. An AI reasoning acceleration system based on RISC-V extension, characterized in that: Includes vector issue queue, scoreboard, vector decode and issue logic, vector register file read and write control, execution unit, vector register file, AI extension unit, vector load-store data first-in-first-out queue, advanced consistency interface and data cache; The vector decode and transmit logic are respectively connected to the vector transmit queue, vector register file read / write control, and execution unit. The vector transmit queue is connected to the scoreboard. The vector register file is respectively connected to the vector register file read / write control, AI extension unit, and vector load-store data first-in-first-out queue. The vector load-store data first-in-first-out queue is respectively connected to the advanced consistency interface and data cache. The execution unit is connected to the AI ​​extension unit. The vector register file is a unified vector register file with a width of 256 bits, supports mixed data type storage, and has built-in data type marking and overflow protection logic. The AI ​​extension unit consists of a U8 matrix multiplication acceleration unit and a LUT lookup table unit. The U8 matrix multiplication acceleration unit is called by the matrix multiplication and accumulation instruction based on the RISC-V extension to perform 4×4 matrix multiplication parallel operations, and the LUT lookup table unit is called by the load quantization matrix multiplication parameter instruction based on the RISC-V extension to perform element-by-element operations of the U8 data type.

2. The RISC-V extended AI reasoning acceleration system according to claim 1, characterized in that: The U8 matrix multiplication acceleration component consists of a 4×4 matrix multiplication array, a quantization compensation module, and an INT32->U8 quantization module.

3. The AI ​​reasoning acceleration system based on RISC-V extension according to claim 1 or 2, characterized in that: The size of a single storage unit of the LUT lookup table component is 256×8 bits.

4. The RISC-V extended AI reasoning acceleration system according to claim 3, characterized in that: The RISC-V based vector instruction extension uses 32 vector registers, the vector length is 128 bits and the instruction format is based on RVV OP-V encoding.

Images