A tvm-based intermediate representation and data generation method

Abstract

The application aims to provide a method for generating intermediate representation and data based on TVM, which firstly constructs a TVM computing graph, then obtains TVM debugger data, and finally generates custom IR and data. That is, TVM is used as a front-end compiler to convert different deep learning frameworks into a high-level graph representation in TVM, the high-level graph representation is parsed to generate an extensible custom IR, and the custom IR is passed in the whole software tool chain. The application decouples the front-end algorithm framework and the compilation stage to obtain data as much as possible, and accelerates the research and development, debugging and verification of the deep learning compiler.

Description

Technical Field

[0001] This invention belongs to the field of deep learning compilation technology, specifically relating to an intermediate representation and data generation method based on TVM. Background Technology

[0002] The difficulty of deploying various deep learning models on different DL hardware has driven the research and development of deep learning compilers in the community. Several deep learning compilers have been proposed by industry and academia, such as Tensorflow XLA and TVM. Similarly, deep learning compilers take DL models described in different DL frameworks as input and then generate optimized code as output for different DL hardware. Currently, the general design architecture of deep learning compilers mainly includes a compilation front-end and a compilation back-end. The compilation front-end is mainly responsible for hardware-independent processing and optimization before sending it to the back-end. The compilation back-end's main task is to perform specific optimizations for the output hardware, compile it into the corresponding hardware instructions, and finally call the various hardware devices to perform computation. IR stands for intermediate representation, which expresses various parameters and inter-layer relationships of the neural network. Common IRs for CPUs and GPUs include ONNX IR, TVM IR, and MLIR, but manufacturers developing their own NPUs generally convert these IRs into custom IRs for easy expansion and adaptation to their own compilation back-ends. The reason for the need for Inference Reference (IR) is that neural network frameworks vary widely, and the saved model file formats differ greatly. Even within the same framework, the saved format changes with updates. Therefore, the compilation backend needs a deterministic, general, and scalable IR representation method to allow it to focus on the compilation itself. The compilation frontend needs to obtain model parameters from different frameworks as input to the compilation backend. Additionally, during the verification and testing phase, it needs to provide intermediate results of model inference to verify the correctness of the entire compilation toolchain. For example, based on wrappers or hooks, the forward functions of various operators in the PyTorch frontend can be rewritten to obtain activation data for each operator in the network. However, tools developed for specific deep learning frameworks do not support other mainstream frameworks. Therefore, a solution is urgently needed to decouple the frontend algorithm framework from the data acquisition during the compilation phase as much as possible, accelerating the development, debugging, and verification of deep learning compilers. Summary of the Invention

[0003] The purpose of this invention is to provide an intermediate representation and data generation method based on TVM, which decouples the front-end algorithm framework and data acquisition during the compilation stage as much as possible, thereby accelerating the research, development, debugging, and verification of deep learning compilers.

[0004] The technical solution adopted in this invention is an intermediate representation and data generation method based on TVM, which is implemented according to the following steps:

[0005] Step 1: Construct the TVM calculation graph;

[0006] Step 2: Obtain TVMdebugger data;

[0007] Step 3: Generate custom IR and data.

[0008] The invention is further characterized in that,

[0009] Step 1 is as follows:

[0010] The input to the deep learning compiler is a quantized neural network model, in one of the mainstream deep learning frameworks. When the mainstream deep learning framework is PyTorch, the input is a network model file in pth format and model input data in numpy format. The frontend module of the TVM framework provides conversion interfaces from various mainstream deep learning frameworks to a unified computation graph. According to the type of neural network model, the corresponding interface function is called, and the output is mod and params. Here, mod is the unified computation graph converted from the neural network model, saved in txt format, and params is the parameter data of the neural network model, saved in dictionary format. TVM generates a computation graph mod, where the first line contains the dimension and data type of the model input and parameters, and the remaining lines correspond to a single operator or operation, including the name and attributes of the operator.

[0011] Step 2 is as follows:

[0012] The TVM debugger serves as an interface for debugging the execution of the TVM computation graph. During TVM runtime, it provides access to the graph structure and tensor values. The optimization level is set via `opt_level`, ranging from 0 to 4. `Pass` represents a series of optimization functions based on Relay IR in TVM, used to simplify the computation graph and remove redundant operators. Each pass has a corresponding optimization level attribute; if the optimization level of a pass is higher than the value of `opt_level`, that pass is not executed, meaning no optimization is performed on the IR. The input consists of the model parameters `params` and the computation graph `mod` generated by TVM in step 1. First, the network model is compiled and code is generated based on TVM. Then, based on TVM... The debugger tool retrieves the optimized computation graph (_tvmdbg_graph_dump.json) in JSON format, along with the related parameters and activation data (output_tensors.params) generated after model execution. This data is saved as binary in serialized format and loaded as a dictionary using load_param_dict. Each tensor is of type tvm.ndarray. The optimized computation graph, constructed via relay, is maintained in _tvmdbg_graph_dump.json, containing all information about the graph, including nodes, arg_nodes, heads, node_row_ptr, and attrs. This is saved as a JSON serialized format, where nodes are placeholders or computation nodes in the JSON, stored as a list. Each node contains information about the operator type, node name, operator inputs, attributes, number of inputs, and number of outputs.

[0013] Step 3 is as follows:

[0014] 1) Operator types include "dense", "add", "nn_conv2d", "nn_relu", "nn_bias_add", "nn_max_pool2d", "nn_avg_pool2d", "concatenate", and "transpose". Operators other than param in the Graph JSON are categorized by operator name and then stored in the dictionary data_list.

[0015] 2) Read the first line of the TVM computation graph, obtain the model parameter names and parameter sizes, and store them in shape_dtype_dict;

[0016] 3) Analyze the TVM computation graph line by line, perform operator fusion, cleaning and optimization, obtain operator-related attributes and dependencies and save them to a custom IR. The custom IR represents the attributes, file list, quantization parameters and data type information of operators in each layer of the network. It adopts the lightweight JSON file format and the content is a Python dictionary structure.

[0017] When the network model is quantized using a quantization tool, different quantization methods are adopted according to different accuracy requirements and different hardware backends. When merging the quantization-related lines in the txt file, different fusion schemes are given for different quantization methods.

[0018] In step 3, during operator parsing, the op_name(p*) corresponding to the model parameters in TVMGRAPHJSON is determined according to the order in which the model parameters appear. The model parameters are then obtained from the TVM outputs data based on the op_name and saved in the parameters directory in npy format.

[0019] The mapping relationship between the operators in the computation graph generated in step 1 and the operators in the Graph JSON is established as follows:

[0020] Based on the operator type and inputs, determine the corresponding operator node name in the data_list dictionary, and retrieve the operator input / output activation data from the TVM outputs data based on the node name, saving it in the activations directory in npy format.

[0021] The beneficial effects of this invention are that it provides an intermediate representation and data generation method based on TVM (Transmission Theory Modeling). This method utilizes TVM as a front-end compiler to convert different deep learning frameworks into high-level graph representations within TVM, parses these high-level graph representations to generate scalable custom IRs, and then passes them throughout the software toolchain. This decouples model training, quantization frameworks, and the deep learning compilation toolchain, avoiding the problems of duplicate implementations or computational limitations of certain custom layers during conversions between frameworks. Simultaneously, it obtains model parameters and activation values ​​of each layer based on TVM, which are used as input to the compilation backend and to verify the correctness of the compilation toolchain. This avoids developing independent methods for obtaining model parameters and data for different deep learning frameworks, thus accelerating the deployment of algorithm models on AI chips. Attached Figure Description

[0022] Figure 1 This is the deep learning model deployment process.

[0023] Figure 2 This is a flowchart of the present invention. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0025] This invention provides an intermediate representation and data generation method based on TVM, the flowchart of which is shown below. Figure 2 As shown, please follow these steps:

[0026] Step 1: Construct the TVM calculation graph;

[0027] Step 1 is as follows:

[0028] Combination Figure 1 The deep learning compiler takes a quantized neural network model as input, in one of the mainstream deep learning frameworks (e.g., TensorFlow, PyTorch, ONNX, Caffe). When PyTorch is the mainstream framework, the input is a network model file in PTH format and model input data in NumPy format. The TVM framework's frontend module provides conversion interfaces from various mainstream deep learning frameworks to a unified computation graph, such as from_tensorflow, from_pytorch, from_ONNX, and from_caffe. Depending on the neural network model type, the corresponding interface function is called, and the output is `mod` and `params`. `mod` is the unified computation graph converted from the neural network model, saved in TXT format, and `params` contains the neural network model data. The parameter data is stored in dictionary format. TVM generates a computational graph (mod), where the first line contains the dimensions and data types of the model input and parameters. The remaining lines correspond to individual operators or operations, including the operator's name and attributes. For example, the model input is %input0:Tensor[(1,3,224,224),float32], where input0 is the model input node name, the input data dimension is [1,3,224,224], and the data type is float32. The weight parameters of the first convolutional layer are %layers.0.conv.weight:Tensor[(64,3,3,3),float32], where layers.0.conv.weight is the weight name of the first convolutional layer, the weight data dimension is [64,3,3,3], and the data type is float32. In the second line, nn.conv2d is the operator name, and padding, channels, and kernel_size are operator attributes.

[0029] Step 2: Obtain TVMdebugger data;

[0030] Step 2 is as follows:

[0031] The TVM debugger serves as an interface for debugging the execution of the TVM computation graph. During TVM runtime, it provides access to the graph structure and tensor values. The optimization level can be set by configuring opt_level, which ranges from 0 to 4 from low to high. 0 represents the lowest optimization level, indicating that no operator fusion is performed and only simple inference is performed. opt_level=0 indicates that no operator fusion is performed and only simple inference is performed. opt_level=1 indicates that operator fusion optimization is performed, and opt_level=2 indicates constant folding optimization, etc.

[0032] Pass, as a series of optimization functions based on Relay IR in TVM, is used to simplify the computation graph, remove some redundant operators, and improve the inference efficiency of the model. For example, the RemoveUnusedFunctions function removes redundant functions in relay IR, and the ToBasicBlockNormalForm function iterates through the functions in relay IR and converts each function into a basic block form. Each pass has a corresponding optimization level attribute. If the optimization level of a pass is higher than the value of opt_level, the pass will not be executed, i.e., no corresponding optimization will be performed on the IR. Input the model parameters params and computation graph mod generated by TVM in step 1. First, the network model is compiled and code is generated based on TVM. Then, the optimized computation graph _tvmdbg_graph_dump.json in JSON format and the relevant parameters and activation data output_tensors.params generated after the model is executed are obtained based on the TVM debugger tool. It is saved as binary in serialized format and loaded into dictionary format by load_param_dict. The single tensor type is tvm.ndarray. The optimized computation graph constructed by relay is kept in _tvmdbg_graph_dump.json, which contains all information about the graph, including nodes, arg_nodes, heads, node_row_ptr, and attr information. It is saved as JSON serialized format, where nodes are placeholders or computation nodes in JSON and are stored in the form of a list. Each node contains information about operator type, node name, operator input, attributes, number of inputs, and number of outputs.

[0033] Step 3: Generate custom IR and data.

[0034] Step 3 is as follows:

[0035] 1) Operator types include "dense", "add", "nn_conv2d", "nn_relu", "nn_bias_add", "nn_max_pool2d", "nn_avg_pool2d", "concatenate", and "transpose". Operators other than param in the Graph JSON are categorized by operator name and then stored in the dictionary data_list.

[0036] 2) Read the first line of the TVM computation graph, obtain the model parameter names and parameter sizes, and store them in shape_dtype_dict;

[0037] 3) Analyze the TVM computation graph line by line, perform operator fusion, cleaning and optimization according to the specific hardware architecture, obtain operator-related attributes and dependencies and save them to a custom IR. The custom IR represents the attributes, file list, quantization parameters and data type information of operators at each layer of the network. It adopts the lightweight JSON file format and the content is a Python dictionary structure.

[0038] When the network model is quantized using a quantization tool, different quantization methods are adopted according to different accuracy requirements and different hardware backends. When merging the quantization-related lines in the txt file, different fusion schemes are given for different quantization methods.

[0039] In step 3, during operator parsing, the op_name(p*) corresponding to the model parameters in TVMGRAPHJSON is determined according to the order in which the model parameters appear. Based on the op_name, the model parameters, such as the weights of the convolutional layer and quantization parameters, are obtained from the TVM outputs data and saved in the parameters directory in npy format.

[0040] The mapping relationship between the operators in the computation graph generated in step 1 and the operators in the Graph JSON is established as follows:

[0041] Based on the operator type and inputs, determine the corresponding operator node name in the data_list dictionary, and retrieve the operator input / output activation data from the TVM outputs data based on the node name, saving it in the activations directory in npy format.

[0042] In steps 1 to 3 above, certain parameters are fused according to hardware characteristics and architecture design. For example, in QIMING930, the four parameters related to BatchNorm are fused into bn_k and bn_b, the inverse quantization of weight is fused into the bn parameter, and the bias of conv and dense operators is fused into the bn parameter, etc.

[0043] This invention targets the Qiming930 architecture and uses TVM to convert different frameworks into a unified computation graph. It parses the computation graph line by line, fuses, cleans, and optimizes operators based on hardware characteristics and architecture, and generates a custom IR. A TVM debugger tool generates TVM outputs and Graph JSON, establishing a mapping relationship between model parameters, activation data of each operator in the custom IR, and TVM outputs. This generates model parameters and activation values ​​for each operator, used for verification and testing of the compilation toolchain. This decouples deep learning frameworks from the unified processing of the compilation frontend (including intermediate representation generation, model parameter and activation data generation), avoiding the problems of conversion between various mainstream frameworks and target device-supported frameworks, as well as the inability to compute certain layers due to the target framework's lack of support during conversion between different deep learning frameworks. This enables rapid deployment of various deep learning framework models on AI chips.

[0044] Definitions of abbreviations and key terms

[0045] TVM: Tensor Virtual Machine, which is a type of deep learning compiler;

[0046] IR: Intermediate Representation file;

[0047] XLA: Accelerated linear algebra;

[0048] DL: Deep learning.

Claims

1. A method for intermediate representation and data generation based on TVM, characterized in that, The specific steps are as follows: Step 1: Construct the TVM calculation graph; Step 1 is described in detail as follows: The input to the deep learning compiler is a quantized neural network model, in one of the mainstream deep learning frameworks. When the mainstream deep learning framework is PyTorch, the input is a network model file in pth format and model input data in numpy format. The frontend module of the TVM framework provides conversion interfaces from various mainstream deep learning frameworks to a unified computation graph. According to the type of neural network model, the corresponding interface function is called, and the output is mod and params. Here, mod is the unified computation graph converted from the neural network model, saved in txt format, and params is the parameter data of the neural network model, saved in dictionary format. TVM generates a computation graph mod, where the first line contains the dimension and data type of the model input and parameters, and the remaining lines correspond to a single operator or operation, including the name and attributes of the operator. Step 2: Obtain TVMdebugger data; Step 2 is described in detail below: The TVM debugger serves as an interface for debugging the execution of the TVM computation graph. During TVM runtime, it provides access to the graph structure and tensor values. The optimization level is set via `opt_level`, ranging from 0 to 4. `Pass` represents a series of optimization functions based on Relay IR in TVM, used to simplify the computation graph and remove redundant operators. Each pass has a corresponding optimization level attribute; if the optimization level of a pass is higher than the value of `opt_level`, that pass is not executed, meaning no optimization is performed on the IR. The input consists of the model parameters `params` and the computation graph `mod` generated by TVM in step 1. First, the network model is compiled and code is generated based on TVM. Then, based on TVM... The debugger tool retrieves the optimized computation graph _tvmdbg_graph_dump.json in JSON format, along with the relevant parameters and activation data output_tensors.params generated after model execution. This data is saved as binary in serialized format and loaded as a dictionary using load_param_dict. Each tensor is of type tvm.ndarray. The optimized computation graph, constructed via relay, is maintained in _tvmdbg_graph_dump.json, containing all information about the graph, including nodes, arg_nodes, heads, node_row_ptr, and attrs information. This is saved as a JSON serialized format, where nodes are placeholders or computation nodes in the JSON, stored as a list. Each node contains information about the operator type, node name, operator inputs, attributes, number of inputs, and number of outputs. Step 3: Generate custom IR and data; Step 3 is as follows: 1) Operator types include "dense", "add", "nn_conv2d", "nn_relu", "nn_bias_add", "nn_max_pool2d", "nn_avg_pool2d", "concatenate", and "transpose". Operators other than param in the Graph JSON are categorized by operator name and then stored in the dictionary data_list. 2) Read the first line of the TVM computation graph, obtain the model parameter names and parameter sizes, and store them in shape_dtype_dict; 3) Analyze the TVM computation graph line by line, perform operator fusion, cleaning and optimization, obtain operator-related attributes and dependencies and save them to a custom IR. The custom IR represents the attributes, file list, quantization parameters and data type information of operators in each layer of the network. It adopts the lightweight JSON file format and the content is a Python dictionary structure. When the network model is quantized using a quantization tool, different quantization methods are adopted according to different accuracy requirements and different hardware backends. When merging the quantization-related lines in the txt file, different fusion schemes are given for different quantization methods. In step 3, during the operator parsing process, the op_name(p*) corresponding to the model parameter in TVMGRAPHJSON is determined according to the order in which the model parameter appears. The model parameter is then obtained from the TVM outputs data based on the op_name and saved in the parameters directory in npy format.

2. The TVM-based intermediate representation and data generation method according to claim 1, characterized in that, The mapping relationship between the operators in the computation graph generated in step 1 and the operators in the Graph JSON is established as follows: Based on the operator type and inputs, determine the corresponding operator node name in the data_list dictionary, and retrieve the operator input / output activation data from the TVM outputs data based on the node name, saving it in the activations directory in npy format.

Images