An operator optimization method for deep learning compilers

Abstract

The present invention provides an operator optimization method for a deep learning compiler. The data structure of the intermediate representation is defined based on the deep learning framework ONNX, and the data structure of the intermediate representation is parsed to obtain an intermediate representation IR for representing different operation operators; according to the intermediate representation IR, hardware-oriented modifications are performed to parse out hardware fusion nodes for deep learning chips as hardware fusion operators; according to the intermediate representation IR and the parsed operators, operator fusion, operator scheduling and operator grouping are performed in sequence to obtain optimized operators, thereby realizing efficient operator optimization for deep learning compilers. Through the definition and parsing of the intermediate representation in the present invention, the operators represented by the same data structure in the model conversion in different deep learning frameworks can be efficiently executed in the chip by the optimization methods of operator fusion, operator scheduling and operator grouping.

Description

Technical Field

[0001] The present invention relates to an operator optimization method for a deep learning compiler, and belongs to the technical field of artificial intelligence applications. Background Art

[0002] With the rise of deep learning, the research on Convolutional Neural Network (CNN) has been further developed. As one of the representative technologies of artificial intelligence, it has been widely used in many fields such as computer vision, natural language processing and autonomous driving, and has achieved unprecedented breakthroughs and achievements, showing the dominant position of convolutional neural network in pattern recognition algorithms.

[0003] With the rise of artificial intelligence and the Internet of Things, the demand for practical applications for mobile terminals and embedded devices has skyrocketed. The difficulty of deploying various models on deep learning hardware has promoted the research and development of deep learning compilers. Deep learning compilers generate optimized code for a certain hardware platform based on deep learning models described by different frameworks, but how to achieve efficient parsing of neural network models and efficiently deploy the models to the corresponding deep learning hardware has become a difficulty. Summary of the invention

[0004] The technical problem solved by the present invention is: to overcome the deficiencies of the above-mentioned prior art, to provide an efficient operator optimization method for a compiler for a deep learning chip, and to achieve efficient operator optimization of a deep learning compiler.

[0005] The technical solution provided by the present invention is: an efficient operator optimization method for a deep learning compiler, the steps are as follows:

[0006] (1) Based on the deep learning framework ONNX, an intermediate representation data structure is defined, and the intermediate representation data structure is parsed to obtain an intermediate representation IR for representing different operation operators;

[0007] (2) According to the intermediate representation IR, the hardware fusion node for the deep learning compiler is parsed as the hardware fusion operator; the hardware fusion node combines multiple operators into one operator according to the Graph and Node data structures in the intermediate representation;

[0008] (3) Operator fusion, operator scheduling, and operator grouping are performed on the intermediate representation IR of step (1) and the operators parsed in step (2) in sequence to obtain optimized operators, thereby achieving efficient operator optimization for the compiler for deep learning chips.

[0009] Preferably, the data structure of the intermediate representation is specifically: a data structure of Graph and Node, wherein the data structure of graph is used to represent the computational graph of the neural network model, and is used to directly parse the input nodes, output nodes and operation nodes of the ONNX model. All input and output nodes and operation nodes are implemented by defining the Node data structure, so that the input and output layers and convolution, pooling, batch processing, nonlinear and other operation layers of the neural network model can be represented by the data structure of Graph and Node. Different operation layers also include hardware bit width and decimal point position information. The ONNX model (input and output layers and convolution, pooling, batch processing, nonlinear and other operation layers), hardware bit width and decimal point position information in the deep learning framework ONNX are parsed into the data structure of the intermediate representation; the ONNX model is represented by the ONNX model file;

[0010] The ONNX model file must conform to the following format: the hardware bit width and the decimal point position information of dynamic fixed-point quantization must be stored in the json file. The hardware bit width is the bit width of each operation layer in the ONNX model. The decimal point position information is the result of dynamic fixed-point quantization of the ONNX model, including the input and output data of each operation layer and the quantization bit data of the weight parameters.

[0011] Preferably, the hardware fusion operator for the deep learning compiler is defined as follows:

[0012] The adjacent operators in the intermediate representation Graph and Node data structures are fused into a hardware operator as a hardware fused operator. The adjacent operators do not contain any branch structures. Except for the first operator in the Graph, all other operators included in the hardware operators have only one parent node, and except for the last operator, all other operators included in the hardware operators have only one child node.

[0013] Preferably, the generation rules of the hardware fusion operator for the deep learning compiler are as follows:

[0014] There are two situations that constitute hardware fusion operators: one is that the first operator is any one of the three operators ConvNode (convolution node), UpsampleNode (upsample node) or AddNode (addition operation node), and the operator behind it must be a non-empty subset of the three operators PoolNode (pooling node), BatchNormalizationNode (batch processing node), and ReluNode (non-linear node). The second is that the operators that constitute it are a subset of the length of at least 2 in the three operator sets of PoolNode, BatchNormalizationNode, and ReluNode. The PoolNode here includes MaxPoolNode (maximum pooling node), AveragePoolNode (average pooling node), and GlobalAveragePoolNode (global average pooling node).

[0015] Preferably, the implementation method of operator fusion is as follows:

[0016] A finite state machine is used to realize the generation of hardware fusion operator. Its initial state is defined as fused1. When the next point can be fused, it jumps to state fused2. When the next point can be fused, it jumps to state fused3. When the next point can be fused, it jumps to state fused4. If it is found that any step in the middle cannot be fused or in the fused4 state, it will jump back to fused1. When jumping from other states to fused1, a hardware fusion node is generated as a hardware fusion operator, thereby realizing operator fusion.

[0017] Preferably, the implementation method of operator scheduling is as follows:

[0018] The intermediate representation Graph and Node after operator fusion are taken as input, and the output is a list containing all nodes (including input nodes). The order in the list represents the order in which each node is executed. The implementation process adopts a depth-first topological sorting algorithm.

[0019] Preferably, the operator grouping is implemented as follows:

[0020] The list of nodes scheduled by the operator (including the order) is taken as input, and the output of the operator grouping is a list of node groups. Each node group is composed of several CPU type nodes or several NPU type nodes (the input node will not be placed in the node group).

[0021] There are two types of node groups, CPU type and NPU type, each of which contains only nodes of the corresponding type. For NPU type node groups, it means the sequence of nodes that can be continuously calculated inside the NPU. For CPU type node groups, it means the sequence of nodes that are continuously calculated on the CPU.

[0022] Each node group is a single chain in the node graph. The situation marking the end of a node group is as follows:

[0023] 1) The output fork of the current node.

[0024] 2) The current node is an output node (no child nodes).

[0025] 3) The child nodes of the current node are of different types from the current node.

[0026] 4) After the next node is included, the convolution operation causes the row overlap of the feature map to be greater than 8 rows.

[0027] Preferably, the hardware fusion node combines multiple operators into one operator according to the Graph and Node data structures in the intermediate representation, thereby reducing the computation time of hardware calculations and achieving efficient operation.

[0028] The advantages of the present invention compared with the prior art are:

[0029] (1) The present invention realizes the efficient execution of neural network models on deep learning chips by designing an operator optimization method for deep learning compilers.

[0030] (2) The present invention implements the analysis of the deep learning framework ONNX by defining and analyzing the intermediate representation data structure, because the neural network model based on the ONNX framework can be used as an open neural network exchange format, so that it is only necessary to convert the models of other different frameworks into the network model of the ONNX framework to compile and optimize a variety of different models, so as to deploy them on the deep learning chip. At the same time, the intermediate representation data structure defined in the present invention can efficiently realize operator fusion, operator scheduling and operator grouping for deep learning chips;

[0031] (3) The optimization methods of operator fusion, operator scheduling and operator grouping in the present invention can realize the efficient execution of neural network algorithm operations on deep learning chips, reduce the operation time of operators, and improve the execution efficiency of operators; BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The overall scheme diagram of the operator optimization of the present invention;

[0033] Figure 2A design diagram of a finite state machine for hardware operator fusion of the present invention;

[0034] Figure 3 A flow chart for grouping operators of the present invention; DETAILED DESCRIPTION

[0035] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] The present invention realizes an operator optimization method for a deep learning compiler, and the method is used in the design of a deep learning compiler. The deep learning compiler has a deep learning chip. In the field of artificial intelligence, there are different deep learning frameworks. In order to successfully deploy multiple neural network models of different deep learning frameworks on deep learning chips, it has brought difficulties to the development of deep learning compilers. Only by adopting the solution of the present invention can the operator optimization problem in the deep learning compiler be solved. Through the definition and analysis of the intermediate representation in the present invention, the operators represented by the same data structure in the model conversion of different deep learning frameworks can be realized. Through the optimization methods of operator fusion, operator scheduling and operator grouping, the operators of the neural network model can be efficiently executed in the chip.

[0037] The deep learning framework ONNX is specifically an open neural network exchange format. It is a standard for representing deep learning models that enables models to be converted between different neural network frameworks.

[0038] Deep learning chips are specialized chips used to calculate deep learning operations. By designing efficient computing units for calculating deep learning, they provide powerful computing power for artificial intelligence algorithms and realize efficient computing of deep learning algorithm models.

[0039] A deep learning compiler is a tool that compiles deep learning models described by different deep learning frameworks into optimized codes on deep learning hardware platforms, thereby achieving efficient deployment of deep learning models on deep learning hardware.

[0040] The present invention provides an operator optimization method for a deep learning compiler, and the preferred solution includes the following steps:

[0041] (1) Based on the deep learning framework ONNX, the data structure of the intermediate representation is defined, and the data structure of the intermediate representation is parsed to obtain the intermediate representation IR, which is used to represent different operators; the preferred solution is as follows:

[0042] Based on the deep learning framework ONNX, the data structure of the intermediate representation is defined, specifically: Graph and Node data structures, where the graph data structure is used to represent the computational graph of the neural network model, and is used to directly parse the input nodes, output nodes, and operation nodes of the ONNX model. All input and output nodes and operation nodes are implemented by defining the Node data structure, so that the input and output layers of the neural network model and the operation layers such as convolution, pooling, batch processing, and nonlinearity can be represented by the Graph and Node data structures. Different operation layers also include hardware bit width and decimal point position information.

[0043] (2) According to the intermediate representation IR, the hardware fusion node for the deep learning chip is parsed as the hardware fusion operator; the preferred solution is as follows:

[0044] According to the intermediate representation IR, the hardware fusion node for deep learning chips is parsed, specifically: the adjacent operators in the intermediate representation Graph and Node data structures are fused into a hardware operator as a hardware fusion operator. The adjacent operators do not contain any branch structure; except for the first operator in the Graph, all other operators included in the hardware operators have only one parent node, and except for the last operator, all other operators included in the hardware operators have only one child node.

[0045] The analyzed hardware fusion nodes for deep learning chips can realize the calculation of multiple operators in one data path on the deep learning chip, thereby reducing the time of neural network reasoning.

[0046] (3) For the operators analyzed in step (1) and step (2), perform operator fusion, operator scheduling, and operator grouping in sequence to obtain optimized operators;

[0047] Operator fusion is performed on the operators parsed in step (1) and step (2) in sequence. Specifically, a finite state machine is used to realize the generation of hardware fusion operators, and all nodes that can generate hardware fusion operators are sequentially fused into a hardware fusion node, thereby reducing the number of operators in the neural network model, which can reduce the time of neural network reasoning.

[0048] For operator scheduling, the preferred solution is: use a depth-first topological sorting algorithm to sort the operator list after operator fusion to obtain a list containing all operators, where the order in the list represents the order in which each node is executed.

[0049] The optimal solution for operator grouping is as follows: the node list (including the order) that has been scheduled by the operator is output as a list of node groups. Each node group is composed of several CPU nodes or several NPU nodes. We require that the input node is not placed in the node group.

[0050] The optimized operators obtained reduce the number of operators in the neural network model, provide the correct execution order of operators and the groups executed in different processor structures, thus solving the operator optimization problem in the deep learning compiler. Through the optimization methods of operator fusion, operator scheduling and operator grouping, the operators of the neural network model can be efficiently executed in the chip.

[0051] The present invention provides an operator optimization method for a deep learning compiler, and the preferred steps are as follows:

[0052] (1) Based on the deep learning framework ONNX, an intermediate representation data structure is defined, and the intermediate representation data structure is parsed to obtain an intermediate representation IR for representing different operation operators;

[0053] (2) According to the intermediate representation IR, the hardware fusion node for deep learning chips is parsed as the hardware fusion operator; the hardware fusion node combines multiple operators into one operator according to the Graph and Node data structures in the intermediate representation, reducing the computing time of hardware calculations and thus achieving efficient operation. ;

[0054] (3) Operator fusion, operator scheduling and operator grouping are performed on the operators analyzed in step (1) and step (2) to obtain optimized operators, thereby realizing efficient operator selection for deep learning compilers. Figure 1 shown.

[0055] The data structure of the intermediate representation in the present invention is specifically: the data structure of Graph and Node, wherein the data structure of graph is used to represent the computational graph of the neural network model, and is used to directly parse the input nodes, output nodes and operation nodes of the ONNX model. All input and output nodes and operation nodes are implemented by defining the Node data structure, so that the input and output layers and the operation layers such as convolution, pooling, batch processing, and nonlinearity of the neural network model can be represented by the data structure of Graph and Node. Different operation layers also include hardware bit width and decimal point position information.

[0056] The hardware fusion operator in the present invention is defined as follows:

[0057] The adjacent operators in the intermediate representation Graph and Node data structures are fused into a hardware operator as a hardware fused operator. The adjacent operators do not contain any branch structures. Except for the first operator in the Graph, all other operators included in the hardware operators have only one parent node, and except for the last operator, all other operators included in the hardware operators have only one child node.

[0058] The preferred solution is: the generation rule of the hardware fusion operator in the present invention is preferably as follows:

[0059] There are two situations that constitute hardware fusion operators: one is that the first operator is any one of the three operators ConvNode (convolution node), UpsampleNode (upsample node) or AddNode (addition operation node), and the operator behind it must be a non-empty subset of the three operators PoolNode (pooling node), BatchNormalizationNode (batch processing node), and ReluNode (non-linear node). The second is that the operators that constitute it are a subset of the length of at least 2 in the three operator sets of PoolNode, BatchNormalizationNode, and ReluNode. The PoolNode here includes MaxPoolNode (maximum pooling node), AveragePoolNode (average pooling node), and GlobalAveragePoolNode (global average pooling node).

[0060] The preferred solution is: the implementation method of operator fusion in the present invention is preferably as follows:

[0061] As attached Figure 2 As shown in the figure, a finite state machine is used to realize the generation of hardware fusion operator. Its initial state is defined as fused1. When the next point can be fused, it jumps to state fused2. When the next point can be fused, it jumps to state fused3. When the next point can be fused, it jumps to state fused4. If it is found that fusion is not possible at any step in the middle or in the fused4 state, it will jump back to fused1. When jumping from other states to fused1, a hardware fusion node is generated as a hardware fusion operator, thereby realizing operator fusion.

[0062] The preferred solution is: the implementation method of operator scheduling in the present invention is as follows:

[0063] The intermediate representation Graph and Node after operator fusion are taken as input, and the output is a list containing all nodes (including input nodes). The order in the list represents the order in which each node is executed. The implementation process adopts a depth-first topological sorting algorithm.

[0064] The preferred solution is: the implementation method of operator grouping in the present invention is as follows:

[0065] The preferred solution is: Figure 3 As shown in the figure, the list of nodes scheduled by the operator (including the order) is taken as input, and the output of the operator grouping is a list of node groups. Each node group is composed of several CPU type nodes or several NPU type nodes (the input node will not be placed in the node group).

[0066] The preferred solution is: there are two types of node groups, CPU type and NPU type, each of which only contains nodes of the corresponding type. For the NPU type node group, it means the sequence of nodes that can be continuously calculated inside the NPU. For the CPU type node group, it means the sequence of nodes that are continuously calculated on the CPU.

[0067] The preferred solution is: each node group is a single chain in the node graph, and the situation marking the end of a node group is as follows:

[0068] 1) The output fork of the current node.

[0069] 2) The current node is an output node (no child nodes).

[0070] 3) The child nodes of the current node are of different types from the current node.

[0071] 4) After the next node is included, the convolution operation causes the row overlap of the feature map to be greater than 8 rows.

[0072] The present invention overcomes the difficulties of low storage and execution efficiency in the current deployment process of convolutional neural network algorithms in deep learning chips, provides an efficient compiler design method for deep learning chips, and improves the execution efficiency of neural network models on deep learning chips.

[0073] The present invention provides a method for designing an efficient compiler for deep learning, which greatly improves the efficiency of the efficient operation of the convolutional neural network model on the deep learning chip, and provides a more efficient compiler optimization method for implementing the reasoning process of the convolutional neural network algorithm in an embedded system.

[0074] The preferred solution is: Figure 1 This is the overall scheme diagram of operator optimization in the present invention. It takes the neural network model in ONNX format and the json file storing the hardware operation layer bit width and quantized decimal position information as input. After the definition and parsing of the intermediate representation, hardware-oriented modification and three operator optimization methods, the optimized operator in the deep learning compiler is obtained.

[0075] The preferred solution is: Figure 2 It is a design diagram of the finite state machine of the hardware operator fusion in the present invention, which shows the generation process of the hardware fusion operator in the present invention in detail and concisely through the finite state machine of four states.

[0076] The preferred solution is: Figure 3 This is a flowchart of operator grouping in the present invention. The list of nodes that have been scheduled by operators (including the order) is taken as input, the operators in the list are traversed, and whether the current operator is assigned to the operator group is determined by judging whether it meets the operator grouping conditions. The judgment conditions for operator grouping are described above.

[0077] The present invention realizes efficient execution of neural network models on deep learning chips by designing an operator optimization method for deep learning compilers, and realizes parsing of the deep learning framework ONNX by defining and parsing the intermediate representation data structure, because the neural network model based on the ONNX framework can be used as an open neural network exchange format, so that it is only necessary to convert the models of other different frameworks into the network model of the ONNX framework to realize the compilation and optimization of multiple different models, so as to deploy them on deep learning chips. At the same time, the intermediate representation data structure defined in the present invention can efficiently realize operator fusion, operator scheduling and operator grouping for deep learning chips;

[0078] The optimization methods of operator fusion, operator scheduling and operator grouping in the present invention can realize the efficient execution of neural network algorithm operations on deep learning chips, reduce the operation time of operators, and improve the execution efficiency of operators;

[0079] The operator optimization method for a deep learning compiler of the present invention has passed the test of the target detection network YOLOv3. Through the operator optimization scheme of the present invention, the optimization operator required by the deep learning chip is successfully compiled, and can be correctly and quickly deployed on the deep learning chip. Through the scheme of operator fusion, operator scheduling and operator grouping, the operator execution time is reduced by 4-10 times. At the same time, the definition and analysis of the intermediate representation provide feasibility for the deployment of models of various deep learning frameworks in deep learning chips.

Claims

1. An operator optimization method for a deep learning compiler, characterized in that Here are the steps: (1) Based on the deep learning framework ONNX, an intermediate representation data structure is defined, and the intermediate representation data structure is parsed to obtain an intermediate representation IR for representing different operation operators; (2) According to the intermediate representation IR, the hardware fusion node for the deep learning compiler is parsed as the hardware fusion operator; specifically, the adjacent operators in the intermediate representation Graph and Node data structures are fused into a hardware operator as the hardware fusion operator, and the adjacent operators do not contain any branch structure; except for the first operator in the Graph, all other operators included in the hardware operator have only one parent node, and except for the last operator, all other operators included in the hardware operator have only one child node; (3) According to the intermediate representation IR of step (1) and the operators parsed in step (2), operator fusion, operator scheduling and operator grouping are performed in sequence to obtain optimized operators, thereby realizing efficient operator optimization of the compiler for deep learning chips; Operator fusion is performed, specifically: a finite state machine is used to realize the generation of hardware fusion operator, whose initial state is defined as fused1. When the next point can be fused, it jumps to state fused2, when the next point can be fused, it jumps to state fused3, and when the next point can be fused, it jumps to state fused4; if it is found that any step in the middle cannot be fused or in the fused4 state, it will jump back to fused1; when jumping from other states to fused1, a hardware fusion node is generated as a hardware fusion operator, thereby realizing operator fusion; Operator scheduling is performed by using a depth-first topological sorting algorithm, taking the intermediate representation Graph and Node after operator fusion as input, and outputting a list of all nodes, including the input nodes. The order in the list represents the order in which each node is executed; Operator grouping is performed, specifically: the list of nodes scheduled by the operator is taken as input, and the output of the operator grouping is a list of node groups. Each node group is composed of several CPU-type nodes or several NPU-type nodes, and the input node is not placed in the node group.

2. The operator optimization method for a deep learning compiler according to claim 1, characterized in that: The data structure of the intermediate representation is specifically: the data structure of Graph and Node, where the graph data structure is used to represent the computational graph of the neural network model and to directly parse the input nodes, output nodes, and operation nodes of the ONNX model; all input and output nodes and operation nodes are implemented by defining the Node data structure, so that the input and output layers and convolution, pooling, batch processing, and nonlinear operation layers of the neural network model can be represented by the Graph and Node data structures. Different operation layers also include hardware bit width and decimal point position information; the ONNX model, hardware bit width, and decimal point position information in the deep learning framework ONNX are parsed into the data structure of the intermediate representation; ONNX models are represented by ONNX model files; The ONNX model file must conform to the following format: the hardware bit width and the decimal point position information of dynamic fixed-point quantization must be stored in the json file. The hardware bit width is the bit width of each operation layer in the ONNX model. The decimal point position information is the result of dynamic fixed-point quantization of the ONNX model, including the input and output data of each operation layer and the quantization bit data of the weight parameters.

3. The operator optimization method for a deep learning compiler according to claim 1, characterized in that: The generation rules of hardware fusion operators for deep learning compilers are as follows: There are two situations that constitute the hardware fusion operator: one is that the first operator is any one of the three operators ConvNode, UpsampleNode or AddNode. In this case, the operator behind it must be an arrangement of non-empty subsets of the three operators PoolNode, BatchNormalizationNode and ReluNode; the other is that the operators that constitute it are an arrangement of subsets of the three operator sets PoolNode, BatchNormalizationNode and ReluNode with a length of at least 2; the PoolNode here includes MaxPoolNode, AveragePoolNode and GlobalAveragePoolNode.

Images