A dual semantic neural network compiling system and method

Abstract

The application discloses a bilingual semantic neural network compiling system and method, relates to the technical field of deep learning, and can receive an ONNX model derived from multiple deep learning frameworks, automatically generate an MLIRScript representation of a Python static subset, and provide executable debugging and MLIR conversion functions in an execution semantic mode and a compiling semantic mode respectively; in the execution semantic mode, intermediate tensor values, shapes and numerical distributions can be observed in real time, and model debugging and verification can be realized; in the compiling semantic mode, a standard MLIR high-level dialect representation is generated through static analysis, the operator topology, tensor types, shapes and parameters are kept consistent with the original ONNX model, and a basis is provided for subsequent multi-hardware platform optimization. The application realizes a neural network compiling path with unified input of multiple front ends and efficient deployment of multiple back ends, solves problems, such as difficult debugging and verification, framework binding limitation and model semantic loss, in the prior art, and improves the debuggability and portability of deep learning models across frameworks and hardware.

Description

Technical Field

[0001] This invention relates to the field of deep learning model compilation technology, and in particular to a dual semantic neural network compilation system and method. Background Technology

[0002] The current mainstream deep learning compilation system generally adopts a three-layer structure: the high-level model definition layer is provided by frameworks such as PyTorch, TensorFlow, and JAX, which is used to build the network structure; the intermediate representation layer is used to express the computation graph structure and operator dependencies, and typical solutions include ONNX and various high-level dialects based on MLIR; the low-level optimization and code generation layer is responsible for operator fusion, memory optimization and hardware mapping, and is usually based on toolchains such as MLIR low-level dialect, LLVM or TVM.

[0003] Although existing compilation toolchains can achieve end-to-end conversion from model to hardware, there are still significant shortcomings in multi-framework model input, multi-backend deployment, and debugging and verification.

[0004] First, existing compilation tools lack interpretable execution capabilities during the debugging and verification phases. Most tools primarily rely on batch-processing graph transformations, which cannot provide real-time monitoring of intermediate states such as tensor shape and numerical changes for complex operator fusion, graph-level rewriting, and dialect conversion processes. This results in a "black box" debugging process, making it difficult to meet the requirements for efficient model iteration.

[0005] Secondly, many deep learning compilation tools are strongly bound to specific frameworks. For example, PyTorch uses TorchScript to capture and compile model graphs through tracing and scripting mechanisms, allowing models to be executed directly in the Python environment or converted to Torch IR for optimization. However, its design philosophy is heavily dependent on the PyTorch framework and cannot support model inputs from ecosystems such as TensorFlow and JAX, limiting the flexibility of cross-framework deployment.

[0006] Therefore, the field of deep learning compilation urgently needs a new intermediate representation and compilation system that should meet the following requirements:

[0007] 1) Process ONNX models from multiple frameworks in a unified manner without relying on a specific framework runtime;

[0008] 2) It possesses both execution semantics and compilation semantics, supporting interpretable execution and debugging of intermediate states during the compilation process;

[0009] 3) It can be deeply integrated with the MLIR multi-level dialect system to achieve multi-level optimization from operator level, graph level to hardware level;

[0010] 4) It can automatically generate corresponding MLIR dialect codes for heterogeneous hardware such as CPU, GPU, and NPU.

[0011] In view of the above requirements, this invention proposes MLIRScript—a unified intermediate representation and compilation system for multiple front-ends implemented with Python as the host language. MLIRScript can use models exported from multiple deep learning frameworks (including TensorFlow, PyTorch, JAX, etc.) via ONNX as a unified input front-end, achieving standardization of model sources and framework decoupling at the front-end level. Simultaneously, MLIRScript integrates execution semantics and compilation semantics in its design: its execution semantics allow developers to run the intermediate representation in an executable and observable manner in a Python environment, enabling real-time debugging and verification during model transformation, operator rewriting, and graph optimization; its compilation semantics are responsible for mapping the unified model semantics to the standard MLIR representation and utilizing MLIR's multi-level dialect system to complete multi-level optimizations such as operator fusion, memory layout optimization, and hardware feature adaptation. Finally, MLIRScript can automatically generate backend MLIR dialect code for various heterogeneous hardware such as CPUs, GPUs, and NPUs, achieving a unified and efficient compilation path from cross-framework, multi-front-end input to multi-hardware backend deployment. Summary of the Invention

[0012] The purpose of this invention is to provide a dual-semantic neural network compilation method and system. It addresses the problems of debugging and verification difficulties, strong framework binding, semantic loss, and lack of a unified compilation path across hardware in existing deep learning compilation tools. By constructing an intermediate representation of Python static subset MLIRScript with execution semantics and compilation semantics, it achieves unified input across multiple frameworks, observable debugging of model semantics, more complete graph-level transformation with semantic preservation, and efficient MLIR compilation for multiple backends.

[0013] To achieve the above objectives, the present invention provides the following technical solution:

[0014] The first aspect of this invention provides a dual-semantic neural network compilation system, comprising the following core modules:

[0015] (1) Model input and parsing module;

[0016] (2) MLIRScript building blocks;

[0017] (3) Dual semantic execution and MLIR dialect generation module.

[0018] The system is designed for static computation graph models and does not include dynamic control flow branches (if / else, while, etc.) triggered by runtime data. This constraint ensures that the model structure is fully deterministic at compile time, enabling a static subset of MLIRScript to generate MLIR representations consistent with the model's semantics.

[0019] The model input and parsing module receives ONNX model files from multiple frameworks (including PyTorch, TensorFlow, JAX, etc.), and constructs a standardized operator description tree by parsing operator types, node connections, tensor shapes, data types, and parameter weights. This module performs the following functions: ① ONNX interface parsing; ② Node topology and dependency construction; ③ Tensor shape and data type inference; ④ Model validity and static compilability checks.

[0020] Through the steps described above, this module ensures that the subsequent MLIRScript build process has a unified and complete model input foundation.

[0021] The MLIRScript building block uses Python as its host language and automatically generates Python code conforming to the MLIRScript static subset based on the operator description tree. The MLIRScript static subset has the following constraints and capabilities: ① Dynamic control flow eliminating data dependencies: It prohibits control flow in the Python layer such as if / else / while statements that depend on runtime tensor values, ensuring that all graph structures are determined at compile time. ② Weakly dynamic shape support: It allows dimensions such as input batch size and sequence length to change at runtime, but the operator topology remains unchanged. ③ Executability and observability: MLIRScript code can run in the Python environment, and developers can directly view intermediate tensors, shape inference results, numerical ranges, and other information. ④ Structured representation of model parameters: Weights are expressed as Python arrays (a subset of NumPy), which can participate in the execution of semantic operations and can also be exported as MLIR constants during the compilation semantic stage.

[0022] This module transforms the ONNX graph into structured, executable Python code with semantic preservation capabilities, laying the foundation for subsequent dual-semantic compilation.

[0023] Dual Semantic Execution and MLIR Dialect Generation Module: This module is the core of the invention. It is responsible for interpreting MLIRScript into both executable and compileable semantics. The executable and compile semantics are based on the same MLIRScript abstract syntax structure.

[0024] (1) Execution semantics: MLIRScript code can be run directly in Python, perform operator calculations, and provide: printable and inspectable intermediate nodes and tensors; operator-level numerical checking and range analysis; shape inference and error localization; and visual debugging of the layer-by-layer execution process. Execution semantics makes the compilation of deep learning models no longer a transition process lacking the observability of intermediate states, greatly improving debugging efficiency.

[0025] (2) Compilation Semantics: By statically analyzing the MLIRScript code, ignoring its Python runtime behavior, only information such as graph structure, computational semantics, tensor types, and operator attributes is extracted and mapped to generate the MLIR dialect. In the compilation semantic mode: tensor computations in MLIRScript are converted into MLIR operators; control structures are converted into MLIR graphs or region structures; operator input / output types are converted into MLIR TensorTypes; and weights are extracted as MLIR constants. Since this invention is only oriented towards static graphs, the execution semantics are consistent with the behavior under compilation semantics, and there is no control flow semantic deviation.

[0026] The second aspect of this invention constructs a complete multi-stage conversion process method, including: ① ONNX → MLIRScript: preserving structure, shape and parameter semantics; ② MLIRScript → MLIR: converting the same Python expression into a high-level MLIR dialect; ③ MLIR → backend dialect.

[0027] This phased path retains more semantic information (runtime shape, local intermediate results, type constraints, etc.) than the traditional ONNX→MLIR direct conversion, making it easier to debug, optimize and adapt to the backend.

[0028] A third aspect of the present invention provides a computing device, including a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method described above.

[0029] A fourth aspect of the present invention provides a non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] (1) A front-end system for unified input of multiple frameworks: By using ONNX as a unified model description format, it enables seamless access to frameworks such as PyTorch, TensorFlow, and JAX, effectively eliminating the binding of existing tools to the runtime of the framework. For example, in the ResNet-50 experiment, this invention can process ONNX models exported by PyTorch / TensorFlow at the same time, while TorchScript cannot support TensorFlow models at all.

[0032] (2) Introduce Python-level dual semantic intermediate representation in the general compilation chain: MLIRScript can be executed and observed in the Python environment, providing debuggability for model conversion and solving the problem of debugging difficulties of existing compilers.

[0033] (3) More complete model semantic preservation: MLIRScript intermediate representation preserves auxiliary information such as shape inference, weight structure, and intermediate node value range, enabling MLIR optimization to utilize more semantics for accurate graph transformation.

[0034] (4) Reduce the difficulty of model migration and hardware adaptation: The unified MLIR output can be mapped to the dedicated dialect of hardware such as CPU, GPU, NPU, etc., reducing the rewriting cost of cross-hardware migration.

[0035] (5) Improve model optimization and debugging efficiency: Developers can use Python to debug the execution process of operators and locate errors more intuitively and faster than traditional compilation tools.

[0036] (6) Static subset design ensures semantic consistency: Since the model does not contain dynamic control flow, the execution semantics of MLIRScript are strictly consistent with the compilation semantics, ensuring the reliability of debugging results. Attached Figure Description

[0037] Figure 1 This is a diagram of the overall system structure.

[0038] Figure 2 Flowchart for the model input and parsing module;

[0039] Figure 3 Flowchart of the MLIRScript module's internal structure;

[0040] Figure 4 This is a schematic diagram of the dual semantic execution and compilation semantic mechanism. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0042] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0043] It should be emphasized here that the step markers mentioned below are not a limitation on the order of the steps, but should be understood as meaning that the steps can be executed in the order mentioned in the embodiments, or in a different order than in the embodiments, or several steps can be executed simultaneously.

[0044] Example 1: System Overall Structure

[0045] Please see Figure 1 This diagram illustrates the overall structure of the MLIRScript dual-semantic neural network compilation system of the present invention. The system includes:

[0046] (1) Model input and parsing module;

[0047] (2) MLIRScript building blocks;

[0048] (3) Dual semantic execution and MLIR dialect generation module.

[0049] The three modules are connected in sequence to form a complete compilation path from ONNX input, MLIRScript conversion to MLIR output.

[0050] Example 2: Model Input and Parsing Module

[0051] Please see Figure 2 This module is used to read ONNX model files exported from frameworks such as PyTorch, TensorFlow, and JAX. The system executes the following steps in sequence:

[0052] (1) Read the ONNX graph structure: parse the graph structure in the ONNX model and extract the node list, tensor information and initial parameters.

[0053] (2) Constructing the operator description tree: Construct a directed acyclic graph based on the node topological relationships, and organize it into an operator description tree structure in topological order. The operator description tree contains the following information: node type (such as Conv, MatMul, Relu, etc.); input and output tensor shapes; parameter weights and attributes (such as strides, pads, kernel_shape); and data dependencies between nodes.

[0054] (3) Static compilability check: This invention only supports static computation graphs. At this stage, the system checks the following: whether the graph contains dynamic control flow; whether there are nodes whose shape cannot be inferred; and whether the data type is supported. For models that do not meet the conditions, the system provides compilation prompts.

[0055] Example 3: MLIRScript Building Blocks

[0056] Please see Figure 3 The task of this module is to generate structured MLIRScript code from the operator description tree. MLIRScript is a static subset hosted in Python, which prohibits dynamic control flow in the Python layer but allows weakly dynamic shape input.

[0057] (1) Establish Python module structure: The system automatically generates the following basic structure.

[0058]

[0059] (2) Parameter mapping: Convert the initial weights in ONNX into NumPy array form:

[0060]

[0061] These parameters can be used to perform semantic operations or extracted into MLIR constants by the compiled semantics.

[0062] (3) Operator mapping rules

[0063] This system establishes a one-to-one MLIRScript call for each type of operator. Please refer to Table 1:

[0064]

[0065] The MLIRScript code generated by the system maintains a consistent topological order, ensuring clear semantics.

[0066] Example 4: Execution Semantics and Compilation Semantics Mechanism

[0067] I. Execution Semantic Pattern

[0068] Please see Figure 1 and Figure 4 Under execution semantics, MLIRScript code is similar to Python code and can be run directly:

[0069]

[0070] In this mode, the system provides: intermediate tensor printing; numerical checking; shape inference; and operator-by-operator execution path display. Developers can verify the model structure and numerical correctness before converting to MLIR.

[0071] II. Compilation Semantic Patterns: Please refer to [link / reference] Figure 1 and Figure 4 Under compiled semantics, the system does not perform actual numerical calculations, but instead performs static analysis on the abstract syntax tree of MLIRScript, extracts semantic information, and generates high-level MLIR dialects.

[0072] For example, the following MLIRScript call:

[0073]

[0074] This will generate the corresponding MLIR dialect representation:

[0075]

[0076] Example 5: A complete conversion example from ONNX to MLIRScript to MLIR

[0077] The complete process is illustrated below using a simple linear network structure as an example.

[0078] I. ONNX Input Example

[0079] Assume the model contains a sequence of operators: MatMul → Add → ReLU

[0080] II. MLIRScript Automatically Generated Results

[0081]

[0082] III. MLIR Dialect Output Examples

[0083]

[0084] This output can be further processed by the MLIR compiler for use on the target platform.

[0085] Example 6: Semantic Preservation and Debuggability

[0086] The significant advantage of this invention lies in the introduction of an executable IR through MLIRScript, ensuring that important semantics are not lost during the transformation process. These include, but are not limited to: ① tensor shape changes, ② the range and distribution of intermediate values, ③ operator parameters and properties, ④ model structural integrity, and ⑤ local optimization opportunities (operator merging, constant folding, etc.).

[0087] Developers can use it directly in Python:

[0088]

[0089] For model debugging, it is far superior to the traditional static ONNX→MLIR conversion method.

[0090] The above embodiments fully illustrate the feasibility and technical advantages of the present invention. The present invention achieves a unified front-end across multiple frameworks through MLIRScript; a dual semantic mechanism of execution and compilation semantics; a phased semantic preservation conversion from ONNX to MLIRScript to MLIR; an executable and observable intermediate representation; and an efficient MLIR compilation path for heterogeneous hardware platforms.

[0091] Example 7: Experimental Verification of ResNet-50 Model Debugging Efficiency and Semantic Preservation

[0092] To verify the technical advantages of this invention in terms of model debugging efficiency and semantic integrity, this embodiment selects the typical industrial vision model ResNet-50 for comparative experiments.

[0093] I. Experimental Environment Configuration

[0094] Hardware environment: Intel Core i7-12700H (14 cores, 20 threads, 32GB DDR5 memory); NVIDIA RTX 3060 (6GB VRAM, driver 525.147.05); 512GB NVMe SSD.

[0095] Software environment: Ubuntu 20.04 LTS (kernel 5.4.0-150-generic); PyTorch 2.0.1, TensorFlow 2.13.0 (for exporting ONNX models); MLIR 17.0.1 (supports MLIRScript compilation semantics); ONNX 1.14.0, NumPy 1.24.3 (for data processing).

[0096] II. Experimental Subjects and Variable Control

[0097] Experimental model: ResNet-50 (no pre-trained weights, only network structure is retained), input size 224×224×3, batch size 8; ONNX models were exported from PyTorch and TensorFlow frameworks respectively (named resnet50_pytorch.onnx and resnet50_tf.onnx), and their validity was verified by onnx.checker.check_model() to ensure that the number of operators and tensor shapes were completely consistent.

[0098] Preset fault simulation: To simulate common compilation and conversion errors in the industry, this experiment injects the same logical fault into the two ONNX models mentioned above.

[0099] ① Operator parameter error: The strides operator of the third layer Conv2d was changed from (2, 2) to (3, 3), which caused subsequent tensor shape chaining abnormalities; ② Tensor shape mismatch: The kernel_size operator of the seventh layer MaxPool2d was changed from (3, 3) to (4, 4), which caused the output dimension to be incompatible with the input of the next layer.

[0100] III. Experimental Procedures and Results Recording

[0101] Debugging efficiency test (comparison of error location time): "Error location time" is defined as the total time from "starting the tool → finding the error → locating the specific operator and parameters". Both tests were performed by two independent testers and the average value was taken. Please refer to Table 2 for the test results.

[0102]

[0103] The debugging operation logic of this invention is as follows:

[0104] The ONNX model is loaded through the model input and parsing module, and an operator description tree is generated.

[0105] The MLIRScript build module automatically generates a static subset of Python code, calls model.debug_shape(input_sample) (input_sample is a random tensor of (8, 3, 224, 224), and outputs in real time "The shape of the Conv2d output of the 3rd layer is abnormal (expected (8, 64, 112, 112), actual (8, 64, 75, 75))";

[0106] Calling `model.debug_value("conv3")` directly prints the operator parameters, and the operator weights and attributes are obtained directly based on the execution semantics, locating the error `strides=(3, 3)`, without the need for manual log parsing throughout the process.

[0107] Example 8: Semantic Integrity Preservation Test (Shape and Value Consistency)

[0108] Using the "error-free original ResNet-50ONNX model" as a benchmark, 100 random samples (8, 3, 224, 224) were input to verify the "tensor shape matching rate" (number of matching layers / total number of layers × 100%) and the "output mean absolute error (MAE)".

[0109] The original (error-free) ResNet-50 ONNX model was used; it was tested on 100 random input samples (shape (8, 3, 224, 224)); the following comparisons were made: ① PyTorch forward pass results and MLIRScript execution semantic results; ② MLIRScript execution semantic results and MLIR compilation semantic results, using Mean Absolute Error (MAE) as the metric.

[0110]

[0111] Please refer to Table 3 for the results:

[0112]

[0113] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits, appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0114] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0115] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0116] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any non-inventive modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dual-semantic neural network compilation system, characterized in that, It includes a model input and parsing module, an MLIRScript building module, and a dual semantic execution and MLIR dialect generation module; The model input and parsing module is used to receive ONNX model files exported from multiple deep learning frameworks, and parse the operator graph information, node topology, tensor shape, data type and weight parameters in them to generate a standardized operator description tree. The MLIRScript building module is used to automatically convert the operator description tree into an MLIRScript representation that conforms to a static subset of Python. The MLIRScript representation preserves the original model semantic structure, where weights are represented in constant form, and operators are called through a restricted operator library. The conversion allows some dimensions of the input tensor to take different values ​​at runtime without changing the topology of the operators. The dual semantic execution and MLIR dialect generation module is used to provide both execution semantics and compilation semantics for the MLIRScript representation. The execution semantics allow for executable and observable operation in a Python environment to verify the model logic and intermediate tensor information. Compilation semantics generate the corresponding MLIR representation through static parsing, which is used for subsequent hardware adaptation and optimization.

2. The dual-semantic neural network compilation system according to claim 1, characterized in that, The dual semantic execution and MLIR dialect generation module is used in the execution semantic mode to provide intermediate tensor values, shape inference, numerical range, and operator-by-operator running and debugging capabilities.

3. The dual-semantic neural network compilation system according to claim 1, characterized in that, The MLIRScript building blocks prohibit dynamic control flow that relies on runtime input, including if / else and while branches, to ensure that the operator topology is statically determined at compile time.

4. A dual-semantic neural network compilation system according to any one of claims 1-3, characterized in that, Each operator in the MLIRScript representation is mapped from an ONNX node, and corresponding MLIR high-level dialect instructions can be generated in the compilation semantic mode, wherein the tensor type, shape and operator properties remain consistent with the original ONNX model.

5. A dual-semantic neural network compilation system according to claim 4, characterized in that, The system supports ONNX model input from multiple deep learning frameworks, enabling unified conversion across multiple front-ends.

6. A dual-semantic neural network compilation method, implemented using the system described in any one of claims 1-5, characterized in that, Includes the following steps: S1. Model Input and Analysis: S101. Read ONNX graph structure: parse the graph structure in the ONNX model and extract the node list, tensor information and initial parameters. S102. Constructing an operator description tree: Construct a directed acyclic graph based on the node topological relationships, and organize it into an operator description tree structure in topological order. The operator description tree contains the following information: node type, input and output tensor shape, parameter weights and attributes, and data dependencies between nodes. S103. Static compilability check: Checks whether the graph contains dynamic control flow; whether there are nodes whose shape cannot be inferred; and whether the data types are supported. S2 and MLIRScript build; S201. Establish the Python module structure and automatically generate the basic structure; S202, Parameter Mapping: Convert the initial weights in ONNX into NumPy arrays. These parameters can be used to perform semantic operations or extracted into MLIR constants by the compiled semantics. S203, Operator Mapping Rules: Establish a one-to-one MLIRScript call for each type of operator, and ensure that the MLIRScript code generated by the system maintains the topological order and that the semantics are clear; S3, Dual Semantic Execution and Dialect Generation: S301, Semantic Execution Mode: The generated MLIRScript is like Python code and can be run directly to perform operator calculations. In this mode, the system also provides: intermediate tensor printing, numerical checking, shape inference, and display of the execution path for each operator. S302, Compilation Semantic Mode: The system does not perform actual numerical calculations, but instead performs static analysis on the abstract syntax tree of MLIRScript, extracts semantic information from it, and generates MLIR high-level dialects.

7. The dual-semantic neural network compilation method according to claim 6, characterized in that, In the compiled semantic mode, the semantic information extracted by static analysis includes graph structure, computational semantics, tensor type, and operator attributes.

8. The dual-semantic neural network compilation method according to claim 6, characterized in that, In the compilation semantic mode: tensor computations in MLIRScript are converted into MLIR operators; control structures are converted into MLIR graphs or region structures; operator input and output types are converted into MLIR Tensor Types; and weights are extracted as MLIR constants.

9. A computing device, comprising: processor; And a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in claim 6.

10. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method of claim 6.

Images