Compilation tool chain for RISC-V instruction expansion neural network accelerator
By using a compilation toolchain based on the MLIR framework and combining it with the hardware characteristics of the RISC-V neural network accelerator, multi-level optimization and operator fusion are performed to solve the problem that existing toolchains cannot fully utilize hardware resources, thus achieving efficient deployment and improved computation speed of neural network models on the RISC-V accelerator.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FUDAN UNIVERSITY
- Filing Date
- 2025-12-22
- Publication Date
- 2026-04-28
AI Technical Summary
Existing compilation toolchains cannot fully utilize the hardware characteristics of RISC-V neural network accelerators, resulting in low deployment efficiency of neural network models and difficulty in achieving efficient operation.
A compilation toolchain based on the MLIR framework was designed. Through model import, optimization and code generation modules, multi-level optimization is performed in combination with hardware characteristics, including computation graph-level operator fusion and hardware-related matrix transpose optimization, to generate efficient executable files.
It enables efficient deployment of ONNX format models on RISC-V neural network accelerators, improving hardware resource utilization and computing speed.
Smart Images
Figure CN121934845A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of compilation technology, specifically relating to a compilation toolchain for RISC-V neural network accelerators. Background Technology
[0002] RISC-V is a free and open-source reduced instruction set architecture designed according to the pure RISC philosophy, characterized by its streamlined architecture and efficient hardware implementation. This architecture not only features a concise basic instruction set but also reserves ample instruction coding space, providing users with powerful scalability. Designers can flexibly customize instruction set extensions according to specific application needs, both streamlining unnecessary functions and adding dedicated acceleration instructions. This highly customizable feature significantly lowers the hardware design threshold while enabling processors to be deeply optimized for target scenarios, exhibiting unique advantages in energy efficiency and performance. Currently, the RISC-V ecosystem is thriving globally, with innovative applications seen in everything from embedded devices to high-performance computing, making it a key force driving processor architecture transformation.
[0003] In recent years, the rapid development of neural network technology has greatly promoted progress in the field of artificial intelligence. Driven by deep learning, neural network models have achieved remarkable results in many fields, including computer vision, natural language processing, and autonomous driving. However, compilers are particularly crucial in order to fully utilize hardware platform resources and improve the running speed of neural network models during deployment. Existing compilation toolchains often fail to fully utilize hardware computing power and lack hardware-related optimizations when optimizing programs, thus hindering efficient model deployment.
[0004] Deep learning compilers are a key technology for the efficient deployment of deep learning models. They take deep learning models as input and achieve cross-platform abstraction and unified optimization of the computation graph by introducing intermediate representations. Through a series of complex graph-level and operator-level optimizations (especially operator fusion and memory planning), automatic code generation, and automatic tuning for specific hardware, they transform high-level model descriptions into efficient code adaptable to various hardware platforms. Deep learning compilers solve the optimization challenges brought about by hardware diversity, significantly improving the speed and efficiency of model inference and training, and are a core infrastructure driving the deployment of AI on a wide range of hardware platforms. TVM and MLIR-based frameworks are currently the most dynamic and representative technological directions in this field. The core idea of TVM is to build an end-to-end deep learning compiler that automatically generates high-performance code for hardware such as CPUs and GPUs by separating computation graph optimization and operator optimization. However, TVM's optimization strategies are still limited to a certain extent by its relatively fixed hardware platforms, and its flexibility and scalability face challenges. MLIR adopts a more general approach, using its modular design to make the transition from high-level IR to hardware-level instructions smoother, thereby giving the compiler framework better flexibility, scalability, and the potential to build a unified compiler ecosystem.
[0005] Therefore, extending the deep learning compilation toolchain for neural network accelerators to RISC-V instructions has become a research hotspot. Summary of the Invention
[0006] The purpose of this invention is to propose a computationally efficient, reusable, and scalable compilation toolchain for RISC-V neural network accelerators.
[0007] This invention proposes a compilation toolchain for RISC-V instruction extension neural network accelerators. Designed based on the MLIR framework, it integrates the neural network model at the top and performs compilation optimization based on hardware characteristics at the bottom. It employs multi-level optimization and efficient code generation techniques to fully utilize hardware resources, improve inference speed, and achieve efficient deployment of the neural network model. Specifically, it includes: a model import module, a high-level dialect module, a model optimization module, a code generation module, and a runtime module; wherein: The model import module uses front-end tools to convert the ONNX format neural network model into a directed acyclic graph and describes it using a custom high-level dialect; that is, it writes a model import script based on ONNX and the custom high-level dialect to convert the ONNX format model into a custom high-level dialect model; here, the high-level dialect includes operators such as matrix multiplication, matrix addition, and convolution. The advanced dialect module, through the ODS framework in the MLIR framework, utilizes computation graph-level optimization methods to achieve computation graph-level optimization based on the custom advanced dialect and its associated data types, attributes, and operations. The model optimization module divides matrix multiplication according to the on-chip cache size, systolic array size, and data flow pattern to achieve hardware-related optimizations; it also optimizes matrix transpose operations by combining the transpose unit in the hardware. The code generation module converts high-level dialects into low-level dialects, facilitating high-level dialect conversion and backend code generation; The runtime module uses supporting tools to convert low-level dialects into executable files that can actually run on hardware accelerators.
[0008] Furthermore: In the model import module, for the loaded ONNX format model structure, an MLIR context is created. Subsequently, a type mapping system is established to convert ONNX descriptions into corresponding MLIR descriptions, and the mapping relationship is maintained through a symbol table. During the conversion process, this module processes the computation graph node by node, parsing each ONNX operator into its corresponding high-level dialect. The operator definitions for the high-level dialect are shown in Table 1. Simultaneously, numerical and string attributes are converted into native MLIR attributes recursively. The front-end conversion tool dynamically constructs the IR structure using an insertion point mechanism, ultimately generating MLIR text containing hierarchical operations, type annotations, and attribute parameters.
[0009] Table 1 .
[0010] In the high-level dialect module, the high-level dialect includes the relationships between various operators in the neural network model. By combining hardware characteristics for correlation analysis, fused operators are obtained. The flowchart for operator fusion based on hardware characteristics is as follows: Figure 4 As shown, the specific process is as follows: First, a directed connection graph is built based on the MLIR-form operation list, where nodes represent operators and edges represent data volume dependencies. Next, a depth-first search algorithm is used to traverse the directed graph, identifying operators that can be fused based on preset criteria. The algorithm determines whether operators can be fused based on factors such as whether the operator itself supports fusion, whether connected operators meet the fusion conditions, and whether the operators participating in the fusion have compatible output shapes. Once fusion is confirmed, a new fusion operator is created, and the previously matched old operators are replaced with this new operator.
[0011] Repeat the above fusion process until all operators in the model have been traversed.
[0012] Finally, the merged directed graph is restored to the MLIR operation list format.
[0013] In the model optimization module, in order to make full use of hardware resources, the neural network model is optimized in combination with hardware characteristics; specifically, this includes: block optimization of matrix multiplication of the systolic array, and optimization of matrix transpose; Furthermore, when the systolic array performs matrix multiplication, it can pre-fetch data for the next round, thus enabling parallel processing of matrix multiplication and matrix transpose operations. This approach does not directly perform a traditional overall transpose, but rather achieves efficient intra-block transpose by altering the data flow of the block matrix. This avoids complex exchanges of large-scale data, reduces transpose overhead and difficulty, and is also more suited to the limitations of hardware resources and specific computational patterns.
[0014] The code generation module performs the following process for generating code from the optimized high-level dialect: First, a code generation class inheriting from `ConvertOpToLLVMPattern` is defined. The `matchAndRewrite` method extracts the operands of the high-level dialect and generates low-level dialect descriptions corresponding to the hardware atomic operations based on the high-level dialects. The high-level dialect is then removed using `replaceOpWithNewOp` in `matchAndRewrite`. Next, a type converter is used to map the high-level data types to low-level LLVM types. Finally, custom code generation classes are added to `mlir::populateCusLegalizeForLLVMExportPatterns` and `mlir::configureCusLegalizeForExportTarget` to ensure semantic correctness after code conversion.
[0015] In the runtime module, the low-level dialect is first converted to a standard LLVM dialect using `convert-to-cus` and `convert-to-llvm`. Then, the LLVM dialect is compiled into object files using the `llvm-llc` compiler. Finally, the `riscv64-unknown-linux-gnu-gcc` cross-compilation toolchain is used to link the object files and generate an executable file that can run directly on a neural network accelerator.
[0016] The technical features and functional advantages of this invention are mainly reflected in the following aspects: (1) This invention supports access to neural network models based on the ONNX format, thereby achieving the goal of deploying widely used ONNX models on hardware platforms; (2) This invention utilizes the multi-level degradation strategy of the MLIR framework to provide abstract modeling for compilation at different levels, thereby enhancing reusability and extensibility; (3) The present invention can optimize the neural network model according to the hardware characteristics, improve the utilization of hardware resources, and realize the efficient computation of the neural network model on a specific hardware platform. Attached Figure Description
[0017] Figure 1 This is a schematic diagram of the compiler toolchain structure of the present invention.
[0018] Figure 2 This is a flowchart of the RISC-V instruction-based extended neural network accelerator in this invention.
[0019] Figure 3 This is a schematic diagram of the high-level dialect representation in this invention.
[0020] Figure 4 This is a schematic diagram of operator fusion in this invention.
[0021] Figure 5 This is a diagram of the pulsating array structure of a neural network accelerator.
[0022] Figure 6 This is a diagram illustrating the filling operation in this invention.
[0023] Figure 7 This is a diagram showing the matrix loading sequence in this invention.
[0024] Figure 8 This is the matrix transpose data flow diagram in this invention. Detailed Implementation
[0025] The specific content of the present invention will be further explained in detail below with reference to the embodiments.
[0026] like Figure 1 As shown, a RISC-V instruction extension neural network accelerator compilation toolchain based on the MLIR framework mainly includes a model import module, a high-level dialect module, a model optimization module, a code generation module, and a runtime module. Among them: The model import module primarily converts ONNX format neural network models into directed acyclic graphs (DAGs) and describes them using a custom high-level dialect based on the MLIR framework, wherein: The ONNX format is Open Neural Network Exchange, which defines a standard, hardware-independent model format that breaks down barriers between frameworks and simplifies model migration.
[0027] The high-level dialect is a dialect introduced in this invention to facilitate the expression of neural network models and operator optimization. The high-level dialect mainly includes operators such as matrix multiplication, matrix addition, and convolution.
[0028] This invention uses ONNX and a custom high-level dialect to write a model import script, which converts ONNX format models into custom high-level dialect models.
[0029] The model import module first loads the ONNX format model structure and creates an MLIR context. Then, a type mapping system is established to convert ONNX descriptions into their corresponding MLIR descriptions, and a symbol table is used to maintain the mapping relationship. During the conversion process, this module processes the computation graph node by node, parsing each ONNX operator into its corresponding high-level dialect. The operator definitions for the high-level dialects are shown in Table 1. Simultaneously, numerical and string attributes are converted into native MLIR attributes recursively. The front-end conversion tool uses an insertion point mechanism to dynamically construct the IR structure, ultimately generating a structure containing hierarchical operations, type annotations, and attribute parameters, as shown in Table 1. Figure 3 The MLIR text shown.
[0030] The high-level dialect module primarily functions to implement a unified representation of neural network models within the MLIR framework using a custom high-level dialect, and to achieve computational graph-level optimization using relevant optimization methods. The advanced dialect contains the operators required in the neural network model. After the model import module converts the ONNX format model, it can use the advanced dialect to achieve a unified representation of the model.
[0031] The aforementioned computation graph-level optimization involves employing a computation graph-level operator fusion algorithm to rewrite the high-level dialect, thereby achieving computation graph-level optimization and outputting a computation graph representing the fused high-level dialect.
[0032] In the high-level dialect module, the ONNX format neural network model, after being converted by the model import module, is uniformly represented using a high-level dialect within the MLIR framework. The high-level dialect contains the relationships between various operators in the neural network model, facilitating correlation analysis based on hardware characteristics to obtain fused operators. The flowchart for operator fusion based on hardware characteristics is shown below. Figure 4 As shown, the details are as follows: First, a directed connection graph is built based on the MLIR-form operation list, where nodes represent operators and edges represent data volume dependencies.
[0033] Next, a depth-first search algorithm is used to traverse the directed graph, identifying operators that can be fused based on preset criteria. The algorithm determines whether operators can be fused based on factors such as whether the operator itself supports fusion, whether connected operators meet the fusion conditions, and whether the operators participating in the fusion have compatible output shapes. Once fusion is confirmed, a new fusion operator is created, and the previously matched old operators are replaced with this new operator.
[0034] Repeat the above fusion process until all operators in the model have been traversed.
[0035] Finally, the merged directed graph is restored to the MLIR operation list format.
[0036] The main function of the model optimization module is to optimize the model based on... Figure 2 The neural network accelerator feature implementation shown is based on transpose module optimization and block matrix multiplication optimization. The main implementation method of block matrix multiplication optimization is to divide the matrix multiplication process according to the on-chip cache size, systolic array size, and fixed output mode. The main implementation method of matrix transpose module optimization is to modify the data flow during the matrix input to the systolic array to achieve the matrix transpose operation. Specifically: In the model optimization module, to fully utilize hardware resources and optimize the neural network model based on hardware characteristics, see [link to relevant documentation]. Figure 2 .
[0037] The specific steps for optimizing block matrix multiplication are as follows: Figure 5 As shown, the systolic array is composed of two-dimensional PE cells. Each PE cell's accumulator has a label bit. When the label bit is zero, the value in the register is cleared. If it is not zero, the systolic array performs matrix multiplication and then adds the result in the register directly.
[0038] To improve data reuse, the matrix is loaded sequentially based on the buffer size and the data flow pattern of the systolic array. For example... Figure 6 As shown, in order to fully utilize the tensor computation unit, when performing matrix multiplication operations... Then, the original matrix is divided into appropriate blocks. Specifically, assuming the pulsation array size is 16*16, based on this architecture, for a matrix A of size M*K, it is divided into sub-matrix blocks with M / 16 rows and 16 columns in each sub-matrix block; Similarly, for a matrix B of size K*N, it is divided into sub-matrix blocks with K / 16 rows and 16 columns in each sub-matrix block.
[0039] When M, N, and K cannot be divided evenly by 16, padding operations need to be performed on the corresponding matrices.
[0040] To fully utilize the fixed output pattern of the systolic array, the present invention adopts the corresponding calculation order, and the specific details are as Figure 7 shown. During the calculation process: First, select the first row-direction block of the matrix A after partitioning, and perform matrix multiplication with the first column-direction block of the matrix B. Through this operation, the result in the row direction at the corresponding position of the matrix C can be obtained.
[0041] Then, keep the current row-direction block of the matrix A unchanged, and successively perform multiplication operations with the next column-direction blocks of the matrix B, so as to gradually obtain the subsequent results in the row direction of the matrix C. Continue this operation process until all column-direction blocks of the matrix B and the current row-direction block of the matrix A complete the calculation task, and the complete current row result of the matrix C is obtained.
[0042] Subsequently, take the next row-direction block of the matrix A and repeat the above calculation process until all row-direction blocks of the matrix A are traversed, complete all matrix multiplication operations, and finally obtain the complete matrix C.
[0043] To make reasonable use of the buffer size, the buffer needs to be reasonably allocated. Denote the buffer size as buf_size, the data type size as type_size, the buffer size occupied by the matrix A as A_buf, the buffer size occupied by the matrix B as B_buf, the buffer size occupied by the matrix C as C_buf, and the number of column-direction blocks of the matrix A as A_cols = ceil(K / 16), where ceil represents rounding up. The number of blocks loaded at one time is block_num. By reasonably adjusting the value of the block_num parameter, the data loading process can be flexibly adjusted, and then the optimal allocation of buffer resources can be achieved.
[0044] In the cache strategy of matrix operations, when the number of matrix blocks required to be loaded at one time is less than the number of column-direction blocks of the matrix A (i.e., block_num < A_cols), the memory occupied by the matrix A can be expressed as 16 * 16 * type_size * block_num. Since the present invention adopts a double-buffer strategy, the size of A_buf is 2 * 16 * 16 * type_size * block_num.
[0045] Furthermore, this invention employs a fixed output mode. To ensure that the number of blocks in the column direction of matrix A matches the number of blocks in the row direction of matrix B, the size of B_buf is also 2*16*16*type_size*block_num. Matrix C, as the result of multiplying block matrix A and block matrix B, has a size of 16*16 in this calculation process; therefore, the size of C_buf is 16*16*type_size. At this point, the following condition must be met: 2*2*16*16*type_size*block_num+16*16*type_size≤buf_size, And based on Figure 7 The matrix loading and calculation operations are performed sequentially.
[0046] Conversely, when the number of matrix blocks to be loaded at once is greater than the number of blocks in the column direction of matrix A (i.e., block_num > A_cols), the size of A_buf is 2 * 16 * 16 * type_size * A_cols. This is because the data in the row direction of matrix A needs to be calculated with the data in all columns of matrix B before the next row of data is needed, so caching only one row of data is sufficient. B_buf is then 2 * 16 * 16 * type_size * block_num, and the size of C_buf remains 16 * 16 * type_size. In this case, the condition 2 * 16 * 16 * type_size * (A_cols + block_num) + 16 * 16 * type_size ≤ buf_size must be met, and the size still follows the... Figure 7 The matrix loading and calculation operations are performed in the order shown.
[0047] Based on transpose module optimization, see [link / reference] Figure 8 As shown, the specific steps are as follows: Taking the transpose module-based implementation of a 2x2 matrix transpose operation as an example, in the first cycle, a11 and a12 are located in the first row of the matrix, and a21 and a22 are located in the second row. In the second cycle, the transpose module reads the first row of data. In the third cycle, the first row of data is passed upwards and the second row of data is read. After reading the complete block matrix, data is output from the left side of the transpose module. In the fourth cycle, the first column of data is read from the left side of the transpose module and passed to the systolic array, while the second column of data is passed to the left. In the fifth cycle, the second column of data of the block matrix is read and passed to the systolic array, completing the matrix transpose operation.
[0048] Furthermore, when the systolic array performs matrix multiplication, it can pre-fetch data for the next round, thus enabling parallel processing of matrix multiplication and transpose operations. This approach does not directly perform a traditional overall transpose, but rather achieves efficient intra-block transpose by altering the data flow of the block matrix. This avoids complex exchanges of large-scale data, reduces transpose overhead and difficulty, and is also more suited to the limitations of hardware resources and specific computational patterns.
[0049] The code generation module's main function is to convert the high-level dialect into a low-level dialect, using the low-level dialect to uniformly represent the neural network model. This invention abstractly describes the computational primitives and instructions specific to neural network accelerators in the low-level dialect, facilitating the conversion of the high-level dialect and backend code generation. Specifically: The process of generating code from the optimized high-level dialect in the code generation module is as follows: First, a code generation class inheriting from `ConvertOpToLLVMPattern` is defined. The `matchAndRewrite` method extracts the operands of the high-level dialect and generates low-level dialect descriptions corresponding to the hardware atomic operations based on the high-level dialects. The high-level dialect is then removed using `replaceOpWithNewOp` in `matchAndRewrite`. Next, a type converter is used to map the high-level data types to low-level LLVM types. Finally, custom code generation classes are added to `mlir::populateCusLegalizeForLLVMExportPatterns` and `mlir::configureCusLegalizeForExportTarget` to ensure semantic correctness after code conversion.
[0050] In the runtime module, the low-level dialect is first converted to a standard LLVM dialect using `convert-to-cus` and `convert-to-llvm`. Then, the LLVM dialect is compiled into object files using the `llvm-llc` compiler. Finally, the `riscv64-unknown-linux-gnu-gcc` cross-compilation toolchain is used to link the object files and generate an executable file that can run directly on a neural network accelerator.
[0051] The hardware architecture optimization method proposed in this embodiment can effectively improve the model running speed, as shown in Table 2.
[0052] Table 2 .
Claims
1. A compilation toolchain for a RISC-V instruction-extended neural network accelerator, characterized in that, Designed based on the MLIR framework, it connects to the neural network model at the top and performs compilation optimization based on hardware characteristics at the bottom. Multi-level optimization and code generation techniques are employed in between to fully utilize hardware resources and achieve efficient deployment of the neural network model. Specifically, it includes: a model import module, a high-level dialect module, a model optimization module, a code generation module, and a runtime module. Among these: The model import module uses front-end tools to convert the ONNX format neural network model into a directed acyclic graph and describes it using a custom high-level dialect; that is, it writes a model import script based on ONNX and the custom high-level dialect to convert the ONNX format model into a custom high-level dialect model; here, the high-level dialect includes operators such as matrix multiplication, matrix addition, and convolution. The advanced dialect module, through the ODS framework in the MLIR framework, utilizes computation graph-level optimization methods to achieve computation graph-level optimization based on the custom advanced dialect and its associated data types, attributes, and operations. The model optimization module divides matrix multiplication according to the on-chip cache size, systolic array size, and data flow pattern to achieve hardware-related optimizations; it also optimizes matrix transpose operations by combining the transpose unit in the hardware. The code generation module converts high-level dialects into low-level dialects, facilitating high-level dialect conversion and backend code generation; The runtime module uses supporting tools to convert low-level dialects into executable files that can actually run on hardware accelerators.
2. The compiler toolchain according to claim 1, characterized in that, In the model import module, for the loaded ONNX format model structure, an MLIR context is created; then a type mapping system is established to convert the ONNX description into the corresponding MLIR description, and the mapping relationship is maintained through a symbol table; during the conversion process, the computation graph is processed node by node, and each ONNX operator is parsed into the corresponding high-level dialect. The operator definitions of the high-level dialect are shown in Table 1; at the same time, numerical, string and other attributes are converted into MLIR native attributes through recursion; the front-end conversion tool uses an insertion point mechanism to dynamically construct the IR structure, and finally generates MLIR text containing hierarchical operations, type annotations and attribute parameters; Table 1 。 3. The compiler toolchain according to claim 2, characterized in that, In the high-level dialect module, the high-level dialect includes the relationships between various operators in the neural network model. By combining hardware characteristics for correlation analysis, fused operators are obtained. The process of operator fusion based on hardware characteristics is as follows: First, a directed connection graph is built based on the MLIR-form operation list, where nodes represent operators and edges represent data volume dependencies. Next, the directed graph is traversed using a depth-first search algorithm, and operators that can be fused are identified based on preset judgment conditions. Then, it is determined whether the operators can be fused based on whether the operators themselves support fusion, whether the connected operators meet the fusion conditions, and whether the operators participating in the fusion have compatible output shape conditions. After confirming that they can be fused, a new fusion operator is created, and the multiple old operators that were originally matched are replaced with this new operator. Repeat the above fusion process until all operators in the model have been traversed and then the process ends. Finally, the merged directed graph is restored to the MLIR operation list format.
4. The compiler toolchain according to claim 3, characterized in that, In the model optimization module, in order to make full use of hardware resources, the neural network model is optimized in combination with hardware characteristics; specifically, it includes: block optimization of matrix multiplication of the systolic array, and optimization of matrix transpose.
5. The compiler toolchain according to claim 4, characterized in that, When the systolic array performs matrix multiplication, it pre-reads the data for the next round, thereby enabling parallel processing of matrix multiplication and matrix transpose operations.
6. The compiler toolchain according to claim 4, characterized in that, In the code generation module, the optimized high-level dialect is used to generate code. The specific process is as follows: First, define a code generation class that inherits from ConvertOpToLLVMPattern. Extract the operands of the high-level dialect in the matchAndRewrite method, and generate the low-level dialect descriptions corresponding to the hardware atomic operations based on the high-level dialects. Then, delete the high-level dialects through replaceOpWithNewOp in matchAndRewrite. Next, a type converter is used to map high-level data types to low-level LLVM types; Finally, in mlir::populateCusLegalizeForLLVMExportPatterns and Add a custom code generation class to mlir::configureCusLegalizeForExportTarget to ensure the semantics are correct after code conversion.
7. The compiler toolchain according to claim 6, characterized in that, In the runtime module: First, the low-level dialect is converted into a standard LLVM dialect using convert-to-cus and convert-to-llvm. Then, the llvm-llc compiler tool is used to compile the llvm dialect and generate the object file. Finally, the target files are linked using the riscv64-unknown-linux-gnu-gcc cross-compilation toolchain, generating an executable file that can run directly on the neural network accelerator.