Design method of multi-level intermediate expression end-side heterogeneous chip AI compiler

Abstract

The invention discloses a design method of a multilevel intermediate expression end-side heterogeneous chip AI compiler, and belongs to the field of heterogeneous AI chip compiling. Based on an open-source MLIR project, two layers of intermediate expression dialects are self-defined; the high-level user-defined dialect is in butt joint with the MMIR content converted by different front-end AI framework models, a user-defined IR calculation graph composed of high-level operators is generated, and compiling information irrelevant to hardware is included; the low-layer self-defined dialect undertakes the high-layer IR calculation graph, the high-layer IR calculation graph is converted into a low-layer transition operator, and then through multiple compilation optimization PASSs, an IR calculation graph formed by self-defined heterogeneous hardware operators is generated; and finally, the low-layer IR calculation graph completes code generation through a rear-end translate tool. According to the method, the open source ecology is utilized to flexibly butt joint various different AI frames, a user-defined PASS is compiled, an advanced compiling scheme in the industry is integrated, the development efficiency is improved, end-side heterogeneous AI chips are adapted, the heterogeneous hardware compiling requirements of different differentiated designs are met, and the development efficiency of compiler adaptive hardware is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of heterogeneous AI chip compilation, and particularly relates to a multi-level intermediate expression end-side heterogeneous chip AI compiler design method. BACKGROUND

[0002] End-side heterogeneous AI chips are important infrastructure in the field of artificial intelligence, providing strong computing power and customized hardware units to accelerate AI model computing tasks, and can flexibly deploy various deep learning models to complete machine learning tasks such as CV, NLP, etc.

[0003] Current end-side AI chips are basically equipped with a set of self-developed customized compilation system, which has the following problems: the framework of the self-developed AI compiler generally has simple structure and poor scalability, and cannot adapt to the newer compilation schemes in the industry; the front end is difficult to connect to multiple different AI model frameworks, and may require a tedious model conversion process; the degree of customization is high, and is coupled with a specific chip, and cannot be reused on other chips. SUMMARY

[0004] The purpose of the present application is to provide a multi-level intermediate expression end-side heterogeneous chip AI compiler design method to solve the problems in the background art.

[0005] To solve the above technical problems, the present application provides a multi-level intermediate expression end-side heterogeneous chip AI compiler design method, comprising:

[0006] Step S1: complete operator fusion, splitting, folding and low-bit quantization operations that are independent of hardware and do not affect accuracy through the front-end AI model framework, and generate the corresponding dialect MLIR computation graph description file through the open-source MLIR conversion tool;

[0007] Step S2: write a high-level custom dialect and a converter tool through the MLIR framework, convert the MLIR computation graph description file of different dialects in the front end into a high-level IR computation graph composed of high-level operators, retain the parameter information of the original model, and remove redundant nodes and information;

[0008] Step S3: write a low-level custom dialect and an optimization compilation PASS through the MLIR framework, first convert the high-level operators to low-level transition operators, and all PASSs perform different optimization or conversion operations on each transition operator to generate a low-level IR computation graph composed of low-level hardware operators;

[0009] Step S4: submit the low-level IR computation graph to the back end through the translate tool written based on the MLIR framework and the back end, complete the final code generation, and obtain hardware executable binary machine code.

[0010] In one implementation, step S1 involves performing some hardware-independent compilation optimization operations within the front-end AI model framework, including:

[0011] Multiple operators are merged into a single equivalent operator according to matching rules; a single large operator is split into multiple sub-operators; constant data in the computation graph is folded into other operators; and low-bit quantization ensures that resource consumption is reduced within the allowable range of precision loss.

[0012] In one implementation, in step S1, the open-source MLIR conversion tool includes ONNX-MLIR, which converts the processed ONNX model into an SSA format MLIR description file, with the corresponding dialect defined by the open-source tool.

[0013] In one implementation, in step S2, a high-level operator is defined within the high-level custom dialect. The high-level operator is used to accept operators from different dialects at the front end, and includes all their parameter information to ensure that operators from different model frameworks are converted into a unified high-level operator.

[0014] In one implementation, in step S2, the converter tool is a conversion PASS written according to the dialect of the front-end model. One front-end dialect corresponds to one PASS, which facilitates the subsequent access to various dialect models converted from different framework models.

[0015] In one implementation, in step S3, transition operators and hardware operators are defined in the low-level custom dialect. The transition operators are used to receive parameter information from higher-level operators and distinguish different operators that can be implemented by the same hardware module. In the higher-level PASS, each transition operator performs different compilation optimization operations.

[0016] The hardware operator is used to submit hardware configuration information, which corresponds to the hardware module. It is converted from information filled by the transition operator. In the lower-level PASS, the hardware operator performs the unified compilation optimization operation of the module.

[0017] In one implementation, in step S3, the compiler-optimized PASS written in the low-level custom dialect is hardware-dependent and can be flexibly added, deleted, or modified according to hardware requirements. The currently defined and implemented PASSes include the following categories in order:

[0018] Constant data processing includes preprocessing data that some hardware cannot support, generating constant data that the original model does not have but the hardware requires, and processing the constant data of the model into the corresponding precision format according to the hardware requirements.

[0019] Data layout format conversion: rearranging the constant data layout format that was originally carried by the model and does not match the hardware requirements.

[0020] Hardware configuration information generation: The software correctly generates and configures the information to enable the hardware's functionality.

[0021] Operator fusion and segmentation: hardware or software segmentation of operators that require excessive resources;

[0022] Memory allocation involves pre-allocating memory for the input and output tensors of each operator and their constant data, and adding address offset information.

[0023] Heterogeneous computation subgraph segmentation involves segmenting operators that cannot be supported by the NPU and placing them on the CPU for execution. This process divides the complete computation graph into multiple subgraphs aggregated from NPU operators and multiple subgraphs aggregated from CPU operators, while ensuring that the topological dependencies between the subgraphs are correct.

[0024] The transition operators are executed one by one from the high-level PASS, completing different compilation optimization operations to become hardware operators. Then, the low-level PASS are executed one by one to complete the unified compilation optimization operations of the corresponding hardware modules, and finally the IR computation graph of the low-level dialect is generated.

[0025] In one implementation, the optimization or conversion operation in step S3 includes: constant data processing, layout format conversion, generation and filling of hardware-related information, fusion and segmentation of hardware-related operators, memory resource allocation, and segmentation of heterogeneous computational subgraphs.

[0026] In one implementation, in step S4, the translate tool connects the low-level IR computation graph with the back-end code generation, submits all hardware operators to the back-end one by one according to the topology, completes the transmission of hardware configuration information and constant data, and finally relies on the back-end to generate hardware-executable binary machine code.

[0027] This invention provides a design method for a multi-level intermediate representation edge-side heterogeneous chip AI compiler. Utilizing the MLIR (Multi-Level Intermediate Representation) framework, an open-source compiler infrastructure, it can leverage its ever-expanding ecosystem to connect to various AI model frameworks and reuse some front-end compilation optimization operations. Simultaneously, custom compilation passes can be defined within the MLIR framework, enabling convenient and flexible migration or reproduction of industry-leading compilation optimization operations. The multi-level IR design of MLIR can separate hardware-independent and hardware-dependent passes, reducing the coupling between the AI ​​compiler and the chip, and facilitating the development of the compilation software stack.

[0028] The multi-layered intermediate representation compiler connects the front-end AI model framework and back-end code generation through two layers. Each layer is developed independently and completes different compilation operations: the high-level IR is decoupled from hardware design, interfaces with various front-end AI model frameworks, and can extend hardware-independent compilation optimization passes to form a unified custom dialect IR description file; the low-level IR is designed for heterogeneous AI chip designs, and customizes hardware-related compilation optimization passes, which can be flexibly adjusted according to hardware changes to prepare hardware module configuration information for back-end code generation. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the overall design architecture of the multi-level intermediate expression end-side heterogeneous chip AI compiler provided by the present invention.

[0030] Figure 2 This is a detailed flowchart illustrating the process of optimizing the front-end AI model to high-level dialect compilation, as provided by this invention.

[0031] Figure 3 This is a schematic diagram of the entire process of low-level dialect compilation optimization PASS provided by the present invention.

[0032] Figure 4 This is an illustration of the segmentation of heterogeneous computation subgraphs.

[0033] Figure 5 This is a flowchart of the translate submission and backend code generation process. Detailed Implementation

[0034] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, provides a more detailed explanation of the multi-level intermediate expression edge-side heterogeneous chip AI compiler design method proposed in this invention. The advantages and features of this invention will become clearer from the following description. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of this invention.

[0035] This invention provides a design method for a multi-level intermediate expression edge-side heterogeneous chip AI compiler, such as... Figure 1 The diagram shows the overall software stack architecture of a multi-layered intermediate expression edge-side heterogeneous chip AI compiler, with a detailed explanation using the popular ONNX front-end model framework as an example:

[0036] S11: The original ONNX model model.onnx file will first go through an optimization tool to complete operations such as constant folding, hardware-independent operator fusion, and splitting. Then, the model will be quantized by a quantization tool to generate the optimized quantized ONNX model model_quant.onnx.

[0037] S12: The community open-source project ONNX-MLIR can convert model_quant.onnx into the corresponding ONNX dialect MLIR description file model_onnx.mlir; where the dialect (Dialect) is a modular IR extension unit that can customize specific operations within the domain;

[0038] S13: The converter tool written in the high-level custom dialect can convert the model_onnx.mlir file into a high-level IR file model_high.mlir composed of unified high-level operators, while performing some compilation optimization operations;

[0039] S14: model_high.mlir is first converted into a low-level IR composed of transition operators through PASS in the low-level custom dialect. Then, the PASS is optimized by various hardware-related compilations to generate the final low-level IR file model_low.mlir. PASS is a modular optimization process that can perform different custom optimizations or downgrade operations on different operators through pattern matching tools.

[0040] S15: Finally, model_low.mlir will submit the hardware operators to the backend through the translate tool, and the code generation will be completed in the backend to obtain the hardware executable binary machine code model.bin; where translate is a tool for generating backend code after the low-level IR computation graph is completed, used to generate the final hardware executable task file or data.

[0041] like Figure 2 The following is a detailed explanation of front-end AI model optimization and high-level dialect compilation optimization:

[0042] Constant folding: In the original model, there may be two Add operators (adding a scalar) connected together. Such constants can be folded into a single Add operator.

[0043] Operator fusion: Multiple operators can be fused into a single equivalent operator according to matching rules. For example, the combination of the Sigmoid and Mul operators can be fused into a single Swish operator.

[0044] Operator segmentation: A certain operator may require too many resources, which cannot be supported by both hardware and software. It is necessary to segment it at the original model. For example, MatMul, which has a large amount of data in both left and right matrices, can be split using Slice and Add.

[0045] Low-bit quantization: The original input, output and constant data of the model may be in FP32, which the hardware may not support directly. Therefore, it is necessary to quantize them to FP16 / BF16 / INT8 precision, which can reduce the model size.

[0046] High-level dialect compilation optimization includes operations such as conversion from different dialects to a unified high-level dialect, elimination of redundant nodes, and graph standardization.

[0047] like Figure 3 The diagram shows the complete process of low-level dialect compilation optimization PASS, which includes the following steps:

[0048] S21: Converts high-level dialects to low-level dialects using a converter, consisting of transition operators;

[0049] S22: Constant data processing is divided into data preprocessing (such as filling grouped convolution weights), generation of LUT tables and other constant data, and conversion of constant data precision format, etc.

[0050] S23: Constant data needs to be converted in terms of layout format according to hardware requirements;

[0051] S24: Hardware operator conversion includes the generation of configuration information required by some special functional modules of hardware and the information filling of hardware operators. At this point, the transition operator will be converted into a hardware operator.

[0052] S25: Thanks to the special design of some hardware modules, some similar hardware operators can be combined into one operator to reduce the consumption of data reading and writing;

[0053] S26: Hardware-related operator segmentation is limited by hardware resources and requires hardware segmentation (filling in segmentation information) or software segmentation (actually segmenting into multiple operators) of some operators;

[0054] S27: Dynamic memory allocation pre-allocates memory for the input and output tensors of each operator and the constant data of the model, adds address offset information, and can use different algorithms to optimize the allocation strategy to improve hardware execution efficiency.

[0055] S28: To fully utilize the characteristics of heterogeneous AI chips, some operators that are difficult for the NPU (Neural Processing Unit) to support are implemented on the CPU (General Purpose Processing Unit). Therefore, the compiler needs to distinguish between NPU operators and CPU operators, then aggregate similar operators together, dividing the complete computation graph into multiple NPU subgraphs and multiple CPU subgraphs, while ensuring that the original topological dependencies remain unchanged. On heterogeneous AI chips, the NPU subgraphs and CPU subgraphs execute computational tasks on the NPU and CPU, respectively. At this point, the final low-level IR has been completely generated.

[0056] like Figure 4 The following is a representation of the heterogeneous computation subgraph segmentation:

[0057] Different nodes are categorized into NPU operators and CPU operators based on operator type, indicating that they should be executed on the computing architecture corresponding to the heterogeneous AI chip. Figure 4 Node0, Node2, Node4, and Node5 are NPU operators, while Node1 and Node3 are CPU operators.

[0058] After heterogeneous computation subgraph partitioning, operators of the same type are aggregated together based on topological relationships. Therefore, the computation graph is divided into three subgraphs: NPU subgraph 0, CPU subgraph 0, and NPU subgraph 1. Figure 1 It is important to note that the input to CPU subgraph 0 is the output of Node 0 in NPU subgraph 0, and the NPU subgraph... Figure 1 The input is the output of CPU subgraph 0 (i.e., Node 3) and the output of NPU subgraph 0 (i.e., Node 4), because the partitioning of heterogeneous computing tasks does not affect the original topological dependencies.

[0059] like Figure 5 The diagram shows the process of submitting code to the translate server and generating backend code.

[0060] The hardware operators in the low-level IR file are translated one by one and submitted to the backend according to the topology. Then, the backend performs operations such as register pre-allocation and instruction generation. Finally, the Bin Producer and JSON Dump generate hardware-executable binary machine code and JSON description files for developers to debug, respectively.

[0061] The present invention provides a design method for a multi-level intermediate expression end-side heterogeneous chip AI compiler, which achieves the following:

[0062] Front-end model converter extension: The converter processes MLIR description files. AI models under different frameworks only need to be converted through the corresponding community open-source MLIR project, and a unified high-level custom dialect IR file can be generated by the converter, which is convenient for subsequent processing.

[0063] High-level IR Pass extension: Industry-leading, hardware-independent compiler optimization passes can be integrated into high-level custom dialects;

[0064] Customized PASS writing for low-level IR: Based on the hardware design, custom compilation and optimization of PASS are performed within the low-level dialect, making full use of the characteristics of heterogeneous AI chips, which is controllable and can be flexibly iterated.

[0065] Compilation support for heterogeneous AI chips: Supports task partitioning on different computing architectures, which can make full use of the various computing processing units on heterogeneous chips;

[0066] Backend separation: Backend code generation is integrated as a sub-project and developed independently. Under the MLIR framework, only the final low-level IR description file needs to be submitted to the backend to complete the generation of binary machine code.

[0067] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the claims.

Claims

1. A design method for a multi-level intermediate expression edge-side heterogeneous chip AI compiler, characterized in that, Includes the following steps: Step S1: The front-end AI model framework performs operator fusion, segmentation, folding, and low-bit quantization operations that are independent of hardware and do not affect accuracy, and generates MLIR computation graph description files for the corresponding dialect using open-source MLIR conversion tools. Step S2: Using the MLIR framework, write a high-level custom dialect and a converter tool to convert the MLIR computation graph description files of different dialects in the front end into a unified high-level IR computation graph composed of high-level operators, retain the parameter information of the original model, and remove redundant nodes and information. Step S3: Write a low-level custom dialect and optimize the compilation PASS through the MLIR framework. First, convert the high-level operators into low-level transition operators. All PASS perform different optimization or transformation operations on each transition operator to generate a low-level IR computation graph composed of low-level hardware operators. Step S4: Using the translate tool written based on the MLIR framework and backend, the low-level IR calculation graph is submitted to the backend to complete the final code generation and obtain hardware-executable binary machine code.

2. The multi-level intermediate expression end-side heterogeneous chip AI compiler design method as described in claim 1, characterized in that, In step S1, some hardware-independent compilation optimization operations are performed within the front-end AI model framework, including: Multiple operators are merged into a single equivalent operator according to matching rules; a single large operator is split into multiple sub-operators; constant data in the computation graph is folded into other operators; and low-bit quantization ensures that resource consumption is reduced within the allowable range of precision loss.

3. The multi-level intermediate expression end-side heterogeneous chip AI compiler design method as described in claim 2, characterized in that, In step S1, the open-source MLIR conversion tool includes ONNX-MLIR, which converts the processed ONNX model into an SSA format MLIR description file, with the corresponding dialect defined by the open-source tool.

4. The multi-level intermediate expression end-side heterogeneous chip AI compiler design method as described in claim 1, characterized in that, In step S2, high-level operators are defined in the high-level custom dialect. The high-level operators are used to accept operators from different dialects in the front end, and contain all their parameter information to ensure that operators from different model frameworks are converted into unified high-level operators.

5. The multi-level intermediate expression end-side heterogeneous chip AI compiler design method as described in claim 4, characterized in that, In step S2, the converter tool is a conversion PASS written according to the dialect of the front-end model. One front-end dialect corresponds to one PASS, which facilitates the subsequent access to various dialect models converted from different framework models.

6. The multi-level intermediate expression end-side heterogeneous chip AI compiler design method as described in claim 1, characterized in that, In step S3, transition operators and hardware operators are defined in the low-level custom dialect. The transition operators are used to receive parameter information from higher-level operators and distinguish different operators that can be implemented by the same hardware module. In the higher-level PASS, each transition operator performs different compilation optimization operations. The hardware operator is used to submit hardware configuration information, which corresponds to the hardware module. It is converted from information filled by the transition operator. In the lower-level PASS, the hardware operator performs the unified compilation optimization operation of the module.

7. The multi-level intermediate expression end-side heterogeneous chip AI compiler design method as described in claim 6, characterized in that, In step S3, the compilation-optimized PASSes written in the low-level custom dialect are hardware-dependent and can be flexibly added, deleted, or modified according to hardware requirements. The currently defined and implemented PASSes include the following categories in order: Constant data processing includes preprocessing data that some hardware cannot support, generating constant data that the original model does not have but the hardware requires, and processing the constant data of the model into the corresponding precision format according to the hardware requirements. Data layout format conversion: rearranging the constant data layout format that was originally carried by the model and does not match the hardware requirements. Hardware configuration information generation: The software correctly generates and configures the information to enable the hardware's functionality. Operator fusion and segmentation: hardware or software segmentation of operators that require excessive resources; Memory allocation involves pre-allocating memory for the input and output tensors of each operator and their constant data, and adding address offset information. Heterogeneous computation subgraph segmentation involves segmenting operators that cannot be supported by the NPU and placing them on the CPU for execution. This process divides the complete computation graph into multiple subgraphs aggregated from NPU operators and multiple subgraphs aggregated from CPU operators, while ensuring that the topological dependencies between the subgraphs are correct. The transition operators are executed one by one from the high-level PASS, completing different compilation optimization operations to become hardware operators. Then, the low-level PASS are executed one by one to complete the unified compilation optimization operations of the corresponding hardware modules, and finally the IR computation graph of the low-level dialect is generated.

8. The multi-level intermediate expression end-side heterogeneous chip AI compiler design method as described in claim 1, characterized in that, The optimization or conversion operations in step S3 include: constant data processing, layout format conversion, generation and filling of hardware-related information, fusion and segmentation of hardware-related operators, memory resource allocation, and segmentation of heterogeneous computation subgraphs.

9. The multi-level intermediate expression end-side heterogeneous chip AI compiler design method as described in claim 1, characterized in that, In step S4, the translate tool connects the low-level IR calculation graph with the back-end code generation, submits all hardware operators to the back-end one by one according to the topology relationship, completes the transmission of hardware configuration information and constant data, and finally relies on the back-end to generate hardware-executable binary machine code.

Images