Convolution operator fusion method

Abstract

The invention provides a convolution operator fusion method, which comprises the following steps of: S0, firstly determining the type of operators needing to be fused: S1, traversing a whole graph to find a sub-graph structure conforming to sequential series connection of N operators; s2, after finding the sub-graph, respectively taking out corresponding weights h and w, step length strides, input and output weight bit widths and activation attributes in N operators, judging whether the operators meet conditions for fusion or not, and skipping the sub-graph if the operators do not meet the conditions for fusion; if yes, performing S3; and S3, if the condition of convolution fusion is met, creating a new node, naming a new operator name, taking out all attributes in N contours, renaming the attributes, putting the attributes into the new node, then connecting corresponding input and output to the new node, enabling the new node to have a corresponding connection relationship in the graph, and deleting the nodes needing to be fused in the graph. The method can reduce the number of nodes and reduce time consumption.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a convolution operator fusion method. Background Art

[0002] In the existing technology, deep neural networks have become the core driving force of many major applications on mobile devices. In order to achieve high accuracy, DNN models have become deeper and deeper, with hundreds or even thousands of operator layers, resulting in high memory and high computing requirements for inference. Operator fusion (or kernel / layer fusion) is a key optimization of many advanced DNN execution frameworks, such as TensorFlow, TVM, and MNN, which aims to improve the efficiency of DNN inference. Operator fusion belongs to graph optimization. Operator fusion can reduce the number of nodes in the entire graph to reduce the data transfer from one node to another, thereby reducing the time spent in the underlying inference process.

[0003] However, the main defects of the existing technology are: in the process of operator fusion, either common operator combinations are fused, such as convolution and BN; or some hot operator combination subgraphs are fused, but multiple convolution operators are not fused.

[0004] In addition, common terms in the prior art include:

[0005] Operator: Deep learning algorithms are composed of computing units, which we call operators (OP for short), including conv, pooling, activation, etc.

[0006] Node: The basic computing unit of a neural network is a neuron, also known as a node. It receives input from other nodes or external sources and calculates output. Summary of the invention

[0007] In order to solve the above problems, the purpose of this application is: in this method, according to the requirement of reducing the computing time at the bottom layer, the entire graph will be traversed to determine whether the traversed node meets the fusion requirements. If it meets the requirements, multiple convolutions that need to be fused will be fused into a new operator.

[0008] Specifically, the present invention provides a method for convolution operator fusion, the method comprising the following steps:

[0009] S0, first clarify the type of operator that needs to be fused, that is, first determine the type of convolution to be fused, so that the correct convolution can be selected for fusion in the subsequent fusion;

[0010] S1, first traverse the entire graph, the graph is understood as a directed acyclic graph, and find a subgraph structure that meets the sequential connection of N operators, where N is the number of operators that need to be fused; S2, after finding the subgraph, respectively take out the corresponding weights h and w, strides, input and output weight bit widths, and activation attributes of the N operators to determine whether the operator meets the conditions for fusion. The attributes taken out here are the values ​​of the above parameters of the convolution operator taken out from the graph. The relationship between the taken out attributes and the judgment conditions is to take out the value corresponding to the kernel to determine whether it meets the fusion conditions;

[0011] If it does not match, the subgraph will be skipped. After skipping the subgraph, the next subgraph that meets the structure will be continued, and it will be judged whether the subgraph meets the fusion conditions. If it meets the conditions, it will be fused. If it does not meet the conditions, the next subgraph will be found or the entire graph will be terminated.

[0012] If it is in compliance, proceed to the next step S3;

[0013] S3, if the conditions for convolution fusion are met, a new node node is created and a new operator name is given. All attributes in N convolutions are taken out, renamed and put into the new node, and then the corresponding input and output are connected to the new node. Each node in the graph has the name information of the input node. The association relationship between nodes is understood through the input name of each node. The connection is to set the input of the subgraph as the input of the new node, delete the original input of the output node of the subgraph, and set the new node as the input of the subgraph output node. In this way, the new node is inserted into the graph so that it has a corresponding connection relationship in the graph, and the nodes that need to be fused in the graph are deleted.

[0014] The step S0 first determines the type of convolution to be fused because multiple convolutions are currently being fused. The convolutions include standard convolution, depth convolution, which is to perform convolution operations on each input channel independently, point convolution, which is an ordinary convolution with a weight width and height of 1x1, and dilated convolution, which is to directly insert spaces into the elements in the convolution kernel to expand the kernel to form a dilated convolution. Therefore, it is necessary to first determine the type of convolution to be fused.

[0015] In the step S0, convolution fusion is performed. According to different classifications of convolution, there are common convolution and deep convolution, including the need to fuse point convolution, deep convolution and point convolution.

[0016] In step S1, the directed acyclic graph is a directed graph without loops, that is, in graph theory, if a directed graph cannot start from a vertex and return to the vertex through several edges, then the graph is a directed acyclic graph.

[0017] In the step S1, N is 3 in the fusion of point_convolution, depthwise_convolution and point_convolution. The three operators of point_convolution, depthwise_convolution and point_convolution are fused. Here, point_convolution refers to the ordinary convolution with weights h and w of 1, and depthwise_convolution is the deep convolution. If other sub-graphs are fused, N is set accordingly. Assuming that depthwise_convolution and point_convolution are fused, N is 2.

[0018] No matter how many operators are fused, the big steps of fusion remain unchanged, only the subgraph structure in the small steps or the conditions for judging whether to perform fusion will change.

[0019] In step S2, in the convolution calculation process, it is necessary to know the input size, the size of the weight kernel, and the stride, and the convolution will be quantized according to different chips, so it is also necessary to obtain chip type information, quantization bitwidth, and activation type action_type information, so it is necessary to obtain the above parameters to determine whether these convolutions can be fused.

[0020] In step S2, the judgment conditions are: the weight width and height of the depth convolution are 3, 3; the step width and height of the three convolutions are all 1; the input bit width, output bit width, and weight bit width of the three convolutions are all 8; the activations of the three convolutions are all None or Relu or Relu6.

[0021] In step S2, it is understood that the graph contains multiple convolutions, and a convolution contains multiple parameter key-value pairs, including kernel:1,1; stride:1,1. The relationship between the extracted attribute and the judgment condition is to extract the value 1,1 corresponding to the kernel.

[0022] In step S3, the fused node deletion or node reduction method is to fuse the three convolutions into one convolution, thereby achieving the purpose of reducing two nodes, and fuse the three nodes of point conv-depthwise conv-point conv into a new conv node, which reduces two nodes for the entire graph.

[0023] Therefore, the advantage of this application is that multiple convolution operators are fused into a new operator, thereby reducing the number of nodes in the graph, reducing data transfer between nodes, and reducing the time consumption of underlying reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0025] Figure 1 It is a flow chart of the present application method.

[0026] Figure 2 It is a schematic diagram of the specific connection relationship before the three convolutions are fused in the embodiment of the present application.

[0027] Figure 3 It is a schematic diagram of three nodes being fused into a new conv node in an embodiment of the present application.

[0028] Figure 4 This is a schematic diagram of a pseudocode in which the annotations of the three nodes are removed from the diagram after the three nodes are merged and the corresponding input relationships are modified in the embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0030] In a neural network, since the network is basically composed of convolution operators, the underlying reasoning operation process is time-consuming. Based on the test of the underlying operators, the operators that can be fused are fused to reduce the time of reasoning operations. At the graph level, the operators that meet the requirements are fused into a new operator and then passed to the underlying layer for calculation.

[0031] The present application embodiment relates to a method for convolution operator fusion, such as Figure 1 As shown in the figure, multiple convolution operators are fused to obtain a new operator, which reduces the time consumption of underlying reasoning. The following steps are included:

[0032] S0, first clarify the type of operator that needs to be fused, that is, first determine the type of convolution to be fused, so that the correct convolution can be selected for fusion in the subsequent fusion;

[0033] The reason why we need to determine the fusion operator type first is that we are currently fusing multiple convolutions. As for convolution, there are standard convolution, depth convolution (convolution operation is performed independently on each input channel), point convolution (ordinary convolution with a weight width and height of 1x1), dilated convolution (elements in the convolution kernel are directly inserted into spaces to expand the kernel to form dilated convolution), etc. Therefore, we need to determine the type of convolution to be fused first, so that we can choose the correct convolution for fusion in the subsequent fusion (in this embodiment, point convolution-depth convolution-point convolution need to be fused. For the specific connection relationship, see Figure 2 shown).

[0034] S1, first traverse the entire graph, the graph is understood as a directed acyclic graph, find the subgraph structure that meets the sequence of N operators in series, where N is the number of operators that need to be fused; the graph here means a graph, and the graph here can be understood as a directed acyclic graph (a directed graph without loops, that is, in graph theory, if a directed graph cannot start from a vertex and return to the point through several edges, then the graph is a directed acyclic graph); in the current example of point_convolution, depthwise_convolution and point_convolution, N is 3 in the three convolution fusions. If other subgraphs are fused, assuming that depthwise_convolution and point_convolution are fused, N is 2, where N is the number of operators that need to be fused. No matter how many operators are fused, the big steps of fusion remain unchanged, only the subgraph structure in the small steps or the conditions for judging whether fusion can be performed will change slightly;

[0035] S2, after finding the subgraph, respectively extract the corresponding weights h and w, strides, input and output weight widths, and activation attributes of the N operators to determine whether the operator meets the conditions to be fused. The extracted attributes here refer to the values ​​of the above parameters of the convolution operator extracted from the graph. The relationship between the extracted attributes and the judgment conditions is to extract the values ​​corresponding to the kernel to determine whether the fusion conditions are met;

[0036] If it does not match, the subgraph will be skipped. After skipping the subgraph, the next subgraph that meets the structure will be continued, and it will be judged whether the subgraph meets the fusion conditions. If it meets the conditions, it will be fused. If it does not meet the conditions, the next subgraph will be found or the entire graph will be terminated.

[0037] If it is in compliance, proceed to the next step S3;

[0038] In the convolution calculation process, we need to know the input size, the size of the weight (kernel), the stride, etc., and the convolution will be quantized according to different chips, so we also need to know the chip type (for example, T40, T41 model chips of Beijing Ingenic Integrated Circuit Co., Ltd.), quantization bit width (bitwidth), activation type (action_type), etc., so to determine whether these convolutions can be fused, we need to start with some of the above parameters. The values ​​of the above parameters can be understood as the graph contains multiple convolutions, and a convolution contains multiple parameter key-value pairs, such as kernel:1,1; stride:1,1, etc. The relationship between extracting the attribute and the judgment condition is to extract the value 1,1 corresponding to the kernel to determine whether it meets the fusion condition. The specific judgment conditions in the current example are: the weight width and height of the depth convolution is 3,3; for example, the chip used is the Ingenic T40 model chip; the stride width and height of the three convolutions are all 1; the input bit width, output bit width, and weight bit width of the three convolutions are all 8; the activation of the three convolutions is None or Relu or Relu6;

[0039] S3, if the conditions for convolution fusion are met, a new node node is created and a new operator name is given. All attributes in N convolutions are taken out, renamed and put into the new node (the reason for renaming is that when they are all convolutions, their attribute names are the same and need to be renamed to distinguish them). Then the corresponding input and output are connected to the new node. Each node in the graph has the name information of the input node. The relationship between nodes can be understood through the input name of each node. The connection here is to set the input of the subgraph as the input of the new node, delete the original input of the output node of the subgraph, and set the new node as the input of the output node of the subgraph. The input name changes can be seen Figure 3 The name of each node after input changes, so that the new node is inserted into the graph, such as Figure 3 As shown, the corresponding connection relationship is made in the graph, and the nodes that need to be fused are deleted. The way to reduce nodes here is to fuse three convolutions into one convolution, thereby achieving the purpose of reducing two nodes. Figure 2 to Figure 3 The change merges the three nodes of point conv-depthwiseconv-point conv into a new conv node, which reduces two nodes for the entire graph. How to delete the reference Figure 4 As shown, the three nodes have been merged and the corresponding input relationships have been modified:

[0040] Create a new empty graph, denoted as Graph new_graph;

[0041] Put all nodes with connection relationships in graph into new_graph, so that there are no three nodes with cleared connection relationships in new_graph, thereby achieving the purpose of deleting the three convolution nodes that have completed fusion;

[0042] Save all nodes with connection relationships, expressed as:

[0043] Vector <node>nodes_vec;

[0044] Loop through all nodes in the graph and further

[0045] If the current node has input node information, further

[0046] nodes_vec.push_back(node);

[0047] Loop through all nodes in the nodes_vec container, expressed as:

[0048] for(auto node:nodes_vec)

[0049] Add the node to new_graph, expressed as:

[0050] new_graph.add_node(node).

[0051] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.< / node>

Claims

1. A method for convolution operator fusion, characterized in that: The method comprises the following steps: S0, first clarify the type of operator that needs to be fused, that is, first determine the type of convolution to be fused, so that the correct convolution can be selected for fusion in the subsequent fusion; S1, first traverse the entire graph, which is a directed acyclic graph, and find a subgraph structure that satisfies the sequential connection of N operators, where N is the number of operators that need to be fused; S2, after finding the subgraph, respectively extract the corresponding weights h and w, strides, input and output weight widths, and activation attributes of the N operators to determine whether the operator meets the conditions to be fused. The extracted attributes here refer to the values ​​of the above parameters of the convolution operator extracted from the graph. The relationship between the extracted attributes and the judgment conditions is to extract the values ​​corresponding to the kernel to determine whether the fusion conditions are met; If it does not match, the subgraph will be skipped. After skipping the subgraph, the next subgraph that meets the structure will be continued, and it will be judged whether the subgraph meets the fusion conditions. If it meets the conditions, it will be fused. If it does not meet the conditions, the next subgraph will be found or the entire graph will be terminated. If it is in compliance, proceed to the next step S3; S3, if the conditions for convolution fusion are met, a new node node is created and a new operator name is given. All attributes in N convolutions are taken out, renamed and put into the new node, and then the corresponding input and output are connected to the new node. Each node in the graph has the name information of the input node. The association relationship between nodes is understood through the input name of each node. The connection is to set the input of the subgraph as the input of the new node, delete the original input of the output node of the subgraph, and set the new node as the input of the subgraph output node. In this way, the new node is inserted into the graph so that it has a corresponding connection relationship in the graph, and the nodes that need to be fused in the graph are deleted.

2. The method for convolution operator fusion according to claim 1, characterized in that: The step S0 first determines the type of convolution to be fused because multiple convolutions are currently being fused. The convolutions include standard convolution, depth convolution, which is to perform convolution operations on each input channel independently, point convolution, which is an ordinary convolution with a weight width and height of 1x1, and dilated convolution, which is to directly insert spaces into the elements in the convolution kernel to expand the kernel to form a dilated convolution. Therefore, it is necessary to first determine the type of convolution to be fused.

3. The method for convolution operator fusion according to claim 2, characterized in that: In the step S0, convolution fusion is performed. According to different classifications of convolution, there are common convolution and deep convolution, including the need to fuse point convolution, deep convolution and point convolution.

4. The method for convolution operator fusion according to claim 1, characterized in that: In step S1, the directed acyclic graph is a directed graph without loops, that is, in graph theory, if a directed graph cannot start from a vertex and return to the vertex through several edges, then the graph is a directed acyclic graph.

5. The method for convolution operator fusion according to claim 1, characterized in that: In the step S1, N is 3 in the fusion of point_convolution, depthwise_convolution and point_convolution. The three operators of point_convolution, depthwise_convolution and point_convolution are fused. Here, point_convolution refers to the ordinary convolution with weights h and w of 1, and depthwise_convolution is the deep convolution. If other sub-graphs are fused, N is set accordingly. Assuming that depthwise_convolution and point_convolution are fused, N is 2.

6. The method for convolution operator fusion according to claim 4, characterized in that: No matter how many operators are fused, the big steps of fusion remain unchanged, only the subgraph structure in the small steps or the conditions for judging whether to perform fusion will change.

7. The method for convolution operator fusion according to claim 6, characterized in that: In step S2, in the convolution calculation process, it is necessary to know the input size, the size of the weight kernel, and the stride, and the convolution will be quantized according to different chips, so it is also necessary to obtain chip type information, quantization bitwidth, and activation type action_type information, so it is necessary to obtain the above parameters to determine whether these convolutions can be fused.

8. The method for convolution operator fusion according to claim 7, characterized in that: In step S2, the judgment conditions are: the weight width and height of the depth convolution are 3, 3; the step width and height of the three convolutions are all 1; the input bit width, output bit width, and weight bit width of the three convolutions are all 8; the activations of the three convolutions are all None or Relu or Relu6.

9. The method for convolution operator fusion according to claim 8, characterized in that: In step S2, it is understood that the graph contains multiple convolutions, and a convolution contains multiple parameter key-value pairs, including kernel:1,1; stride:1,1. The relationship between the extracted attribute and the judgment condition is to extract the value 1,1 corresponding to the kernel.

10. The method for convolution operator fusion according to claim 9, characterized in that: In the step S3, the fused node deletion or node reduction method is to fuse the three convolutions into one convolution, thereby achieving the purpose of reducing two nodes, and the three nodes of point conv-depthwiseconv-point conv are fused into a new conv node, which reduces two nodes for the entire graph.

Images