A method for convolution operator fusion
By identifying and fusing multiple convolution operators in graphic images and creating new operator nodes, the problems of high computing time and memory requirements in existing technologies are solved, and the inference efficiency of deep neural networks is improved.
Patent Information
- Application Number
- CN202310814211.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-03
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2043-07-03
AI Technical Summary
The operator fusion method of deep neural networks in the existing technology fails to effectively fuse multiple convolution operators, resulting in excessive computing time and memory requirements, affecting inference efficiency.
By traversing the directed acyclic graph of the graphic image, multiple convolution operators that meet the conditions are identified, new operator nodes are created, and multiple convolution operators are fused into a new operator to reduce the number of nodes in the graph and optimize the calculation path.
The number of nodes in the graph image is reduced, the computing time and data handling time of the underlying reasoning are reduced, and the reasoning efficiency of the deep neural network is improved.
Smart Images
Figure CN119250126B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to a convolution operator fusion method. Background Art
[0002] Deep neural networks (DNNs) have become the core driving force behind many major mobile applications. To achieve high accuracy, DNN models have become increasingly deep, with hundreds or even thousands of operator layers, resulting in high memory and computational requirements for inference. Operator fusion (or kernel / layer fusion) is a key optimization in many advanced DNN execution frameworks, such as TensorFlow, TVM, and MNN, designed to improve the efficiency of DNN inference. Operator fusion is a graph optimization that reduces the number of nodes in the entire graph, thereby reducing the amount of data moved between nodes and ultimately reducing the time spent on underlying inference.
[0003] However, the main drawback of the existing technology is that in the process of operator fusion, either common operator combinations are fused, such as convolution and BN; or some hot operator combination subgraphs are fused, but multiple convolution operators are not fused.
[0004] In addition, common terms in the prior art include:
[0005] Operator: Deep learning algorithms are composed of computing units, which we call operators (OPs), including conv, pooling, activation, etc.
[0006] Node: The basic computing unit of a neural network is a neuron, also known as a node. It receives input from other nodes or external sources and calculates output. Summary of the Invention
[0007] In order to solve the above problems, the purpose of this application is: in this method, according to the demand of reducing the computing time at the bottom layer, the entire graph will be traversed to determine whether the traversed node meets the fusion requirements. If it meets the requirements, multiple convolutions that need to be fused will be fused into a new operator.
[0008] Specifically, the present invention provides a method for convolution operator fusion, the method comprising the following steps:
[0009] S0, first clarify the operator type that needs to be fused, that is, first determine the type of convolution to be fused, so that the correct convolution can be selected for fusion in the subsequent fusion;
[0010] S1, first traverse the entire graph, the graph is understood as a directed acyclic graph, and find a subgraph structure that meets the sequence of N operators in series, where N is the number of operators that need to be fused; S2, after finding the subgraph, respectively extract the corresponding weights h and w, strides, input and output weight bit widths, and activation attributes of the N operators to determine whether the operator meets the conditions for fusion. The extracted attributes here are the values of the above parameters of the convolution operator extracted from the graph. The relationship between the extracted attributes and the judgment conditions is to extract the value corresponding to the kernel to determine whether it meets the fusion conditions;
[0011] If it does not meet the requirements, the subgraph will be skipped. After skipping the subgraph, the next subgraph that meets the structure will be continued, and the subgraph will be judged to see if it meets the fusion conditions. If it meets the requirements, it will be fused. If it does not meet the requirements, the next subgraph will be found or the entire graph will be terminated.
[0012] If it is in compliance, proceed to the next step S3;
[0013] S3, if the conditions for convolution fusion are met, a new node node is created and named a new operator name. All attributes in the N convolutions are taken out, renamed and placed in the new node, and then the corresponding input and output are connected to the new node. Each node in the graph has the name information of the input node. The association relationship between the nodes is understood by the input name of each node. The connection is to set the input of the subgraph to the input of the new node, delete the original input of the output node of the subgraph, and set the new node to the input of the subgraph output node. In this way, the new node is inserted into the graph so that it has a corresponding connection relationship in the graph, and the nodes that need to be fused in the graph are deleted.
[0014] The reason why the type of convolution to be fused is determined in step S0 is that multiple convolutions are currently being fused. There are standard convolutions, depthwise convolutions, which are independent convolution operations on each input channel, point convolutions, which are ordinary convolutions with a weight width and height of 1x1, and dilated convolutions, which are elements in the convolution kernel directly inserted into spaces to expand the kernel to form dilated convolutions. Therefore, it is necessary to first determine the type of convolution to be fused.
[0015] In the step S0, convolution fusion is performed. According to different classifications of convolution, there are common convolution and depth convolution, including the need to fuse point convolution, depth convolution, and point convolution.
[0016] In step S1, the directed acyclic graph is a directed graph without loops. In graph theory, if a directed graph cannot start from a vertex and return to the vertex through several edges, then the graph is a directed acyclic graph.
[0017] In the step S1, N is 3 in the three convolution fusions of point_convolution, depthwise_convolution and point_convolution. The three operators of point_convolution, depthwise_convolution and point_convolution are fused. Here, point_convolution refers to the ordinary convolution with weight h and w of 1, and depthwise_convolution is the depth convolution. If other sub-graphs are fused, N is set accordingly. Assuming that depthwise_convolution and point_convolution are fused, N is 2.
[0018] No matter how many operators are fused, the big steps of fusion remain unchanged, and only the subgraph structure in the small steps or the conditions for judging whether to perform fusion will change.
[0019] In step S2, during the convolution calculation process, it is necessary to know the input size, the size of the weight kernel, and the stride, and the convolution will be quantized according to different chips, so it is also necessary to obtain chip type information, quantization bitwidth, and activation type action_type information. Therefore, it is necessary to obtain the above parameters to determine whether these convolutions can be fused.
[0020] In step S2, the judgment conditions are: the weight width and height of the depthwise convolution are 3, 3; the step width and height of the three convolutions are all 1; the input bit width, output bit width, and weight bit width of the three convolutions are all 8; the activation of the three convolutions are all None, Relu, or Relu6.
[0021] In step S2, it is understood that the graph contains multiple convolutions, and a convolution contains multiple parameter key-value pairs, including kernel:1,1; stride:1,1. The relationship between the extracted attribute and the judgment condition is to extract the value 1,1 corresponding to the kernel.
[0022] In step S3, the fused node deletion or node reduction method is to fuse the three convolutions into one convolution, thereby achieving the purpose of reducing two nodes, and fusing the point conv-depthwise conv-point conv three nodes into a new conv node, which reduces two nodes for the entire graph.
[0023] Therefore, the advantage of this application is that multiple convolution operators are fused into a new operator, thereby reducing the number of nodes in the graph, reducing data transfer between nodes, and reducing the time consumption of underlying reasoning. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.
[0025] Figure 1 It is a flow chart of the present application method.
[0026] Figure 2 This is a schematic diagram of the specific connection relationship before the three convolutions are fused in the embodiment of the present application.
[0027] Figure 3 It is a schematic diagram of three nodes fused into a new conv node in an embodiment of the present application.
[0028] Figure 4 This is a schematic diagram of removing the annotations and pseudocode of the three nodes from the diagram after the three nodes are merged and the corresponding input relationships are modified in the embodiment of the present application. DETAILED DESCRIPTION
[0029] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail with reference to the accompanying drawings.
[0030] In neural networks, since the network is basically composed of convolution operators, the underlying reasoning operation process is relatively time-consuming. Based on the testing of the underlying operators, the operators that can be fused are fused to reduce the reasoning operation time. At the graph level, the operators that meet the requirements are fused into a new operator and then passed to the underlying layer for calculation.
[0031] The present application embodiment relates to a method for convolution operator fusion, such as Figure 1 As shown in the figure, multiple convolution operators are fused to obtain a new operator, which reduces the time consumption of underlying reasoning. The following steps are included:
[0032] S0, first clarify the operator type that needs to be fused, that is, first determine the type of convolution to be fused, so that the correct convolution can be selected for fusion in the subsequent fusion;
[0033] The fusion operator type is determined first because multiple convolutions are currently being fused. As for convolution, there are standard convolution, depth convolution (convolution operation is performed independently on each input channel), point convolution (normal convolution with a weight width and height of 1x1), dilated convolution (elements in the convolution kernel are directly inserted into the space to expand the kernel to form dilated convolution), etc., so it is necessary to determine the type of convolution to be fused first, so that the correct convolution can be selected for fusion in the subsequent fusion (in this embodiment, point convolution-depth convolution-point convolution need to be fused. For specific connection relationships, see Figure 2 shown).
[0034] S1, first traverse the entire graph, the graph is understood as a directed acyclic graph, find the subgraph structure that meets the sequence of N operators in series, where N is the number of operators that need to be fused; the graph here means a graph, and the graph here can be understood as a directed acyclic graph (a directed graph without loops, that is, in graph theory, if a directed graph cannot start from a vertex and return to the point through several edges, then the graph is a directed acyclic graph); in the current example of point_convolution, depthwise_convolution and point_convolution, N is 3 in the three convolution fusions. If other subgraphs are fused, assuming that depthwise_convolution and point_convolution are fused, N is 2, where N is the number of operators that need to be fused. No matter how many operators are fused, the large steps of fusion remain unchanged, only the subgraph structure in the small steps or the conditions for judging whether fusion can be performed will change slightly;
[0035] S2, after finding the subgraph, extract the corresponding weights h and w, strides, input and output weight widths, and activation attributes of the N operators respectively to determine whether the operator meets the conditions to be fused. The extracted attributes here refer to the values of the above parameters of the convolution operator extracted from the graph. The relationship between the extracted attributes and the judgment conditions is to extract the value corresponding to the kernel to determine whether it meets the fusion conditions;
[0036] If it does not meet the requirements, the subgraph will be skipped. After skipping the subgraph, the next subgraph that meets the structure will be continued, and the subgraph will be judged to see if it meets the fusion conditions. If it meets the requirements, it will be fused. If it does not meet the requirements, the next subgraph will be found or the entire graph will be terminated.
[0037] If it is in compliance, proceed to the next step S3;
[0038] During the convolution calculation process, we need to know the input size, the size of the weight (kernel), the stride, etc., and the convolution will be quantized according to different chips, so we also need to know the chip type (for example, the T40 and T41 chips of Beijing Junzheng Integrated Circuit Co., Ltd.), the quantization bit width (bitwidth), the activation type (action_type), etc., so to determine whether these convolutions can be fused, we need to start with some of the above parameters. The values of the above parameters can be understood as the graph contains multiple convolutions, and a convolution contains multiple parameter key-value pairs, such as kernel:1,1; stride:1,1, etc. Extracting the relationship between the attribute and the judgment condition is to extract the value 1,1 corresponding to the kernel to determine whether it meets the fusion conditions. The specific judgment conditions in the current example are: the weight width and height of the depthwise convolution are 3,3; for example, the chip used is the Junzheng T40 chip; the stride width and height of the three convolutions are all 1; the input bit width, output bit width, and weight bit width of the three convolutions are all 8; the activation of the three convolutions are all None or Relu or Relu6;
[0039] S3, if it meets the conditions of convolution fusion, creates a new node node and names a new operator name, takes out all the attributes in N convolutions, renames them and puts them in the new node (renaming is because when they are all convolutions, their attribute names are the same and need to be renamed to distinguish them), then connects the corresponding input and output to the new node. Each node in the graph has the name information of the input node. The relationship between the nodes is understood by the input name of each node. The connection here is to set the input of the subgraph to the input of the new node, delete the original input of the output node of the subgraph, and set the new node as the input of the output node of the subgraph. The input name changes can be seen Figure 3 The name of each node after input changes, so that the new node is inserted into the graph, such as Figure 3 As shown, it has a corresponding connection relationship in the graph, and the nodes that need to be fused in the graph are deleted. The way to reduce nodes here is to fuse three convolutions into one convolution, thereby achieving the purpose of reducing two nodes. Figures 2 to 3 The change is to merge the three nodes of point conv-depthwiseconv-point conv into a new conv node, which reduces two nodes for the entire graph. How to delete the reference Figure 4 As shown, the three nodes have been merged and the corresponding input relationships have been modified:
[0040] Create a new empty graph, represented as Graph new_graph;
[0041] Put all nodes with connected relationships in graph into new_graph, so that the three nodes with cleared connections do not exist in new_graph, thereby achieving the purpose of deleting the three convolution nodes that have completed fusion;
[0042] Save all connected nodes, expressed as:
[0043] Vector <node>nodes_vec;
[0044] Loop through all nodes in the graph and further
[0045] If the current node has input node information, further
[0046] nodes_vec.push_back(node);
[0047] Looping through all nodes in the nodes_vec container is represented as:
[0048] for(auto node:nodes_vec)
[0049] Add the node to new_graph, expressed as:
[0050] new_graph.add_node(node).
[0051] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.< / node>
Claims
1. A method for convolution operator fusion, characterized in that: The operator fusion belongs to image optimization, and the method includes the following steps: S0, first clarify the operator type that needs to be fused, that is, first determine the type of convolution to be fused, so that the correct convolution can be selected for fusion in the subsequent fusion; S1, first traverse the entire graph, which is a directed acyclic graph, and find a subgraph structure that satisfies the sequential connection of N operators, where N is the number of operators that need to be fused; S2, after finding the subgraph, extract the corresponding weights h and w, strides, input and output weight widths, and activation attributes of the N operators respectively to determine whether the operator meets the conditions to be fused. The extracted attributes here refer to the values of the above parameters of the convolution operator extracted from the graph. The relationship between the extracted attributes and the judgment conditions is to extract the value corresponding to the kernel to determine whether it meets the fusion conditions; If it does not meet the requirements, the subgraph will be skipped. After skipping the subgraph, the next subgraph that meets the structure will be continued, and the subgraph will be judged to see if it meets the fusion conditions. If it meets the requirements, it will be fused. If it does not meet the requirements, the next subgraph will be found or the entire graph will be terminated. If it is in compliance, proceed to the next step S3; In step S2, during the convolution calculation process, it is necessary to know the input size, the size of the weight kernel, and the stride. In addition, the convolution will be quantized according to different chips, so it is also necessary to obtain chip type information, quantization bitwidth, and activation type action_type information. Therefore, it is necessary to obtain the above parameters to determine whether these convolutions can be fused. S3, if the conditions for convolution fusion are met, a new node node is created and named a new operator name. All attributes in the N convolutions are taken out, renamed and placed in the new node, and then the corresponding input and output are connected to the new node. Each node in the graph has the name information of the input node. The association relationship between the nodes is understood by the input name of each node. The connection is to set the input of the subgraph to the input of the new node, delete the original input of the output node of the subgraph, and set the new node to the input of the subgraph output node. In this way, the new node is inserted into the graph so that it has a corresponding connection relationship in the graph, and the nodes that need to be fused in the graph are deleted.
2. The method for convolution operator fusion according to claim 1, characterized in that: The reason why the type of convolution to be fused is determined in step S0 is that multiple convolutions are currently being fused. There are standard convolutions, depthwise convolutions, which are independent convolution operations on each input channel, point convolutions, which are ordinary convolutions with a weight width and height of 1x1, and dilated convolutions, which are ordinary convolutions with a weight width and height of 1x1. The elements in the convolution kernel are directly inserted into spaces to expand the kernel to form a dilated convolution. Therefore, it is necessary to first determine the type of convolution to be fused.
3. The method for convolution operator fusion according to claim 2, characterized in that: In the step S0, convolution fusion is performed. According to different classifications of convolution, there are common convolution and depth convolution, including the need to fuse point convolution, depth convolution, and point convolution.
4. The method for convolution operator fusion according to claim 1, characterized in that: In step S1, the directed acyclic graph is a directed graph without loops. In graph theory, if a directed graph cannot start from a vertex and return to the vertex through several edges, then the graph is a directed acyclic graph.
5. The method for convolution operator fusion according to claim 1, wherein: In the step S1, N is 3 in the three convolution fusions of point_convolution, depthwise_convolution and point_convolution. The three operators of point_convolution, depthwise_convolution and point_convolution are fused. Here, point_convolution refers to the ordinary convolution with weight h and w of 1, and depthwise_convolution is the depth convolution. If other sub-graphs are fused, N is set accordingly. Assuming that depthwise_convolution and point_convolution are fused, N is 2.
6. The method for convolution operator fusion according to claim 4, characterized in that: No matter how many operators are fused, the big steps of fusion remain unchanged, and only the subgraph structure in the small steps or the conditions for judging whether to perform fusion will change.
7. The method for convolution operator fusion according to claim 1, characterized in that: In step S2, the judgment conditions are: the weight width and height of the depthwise convolution are 3, 3; the step width and height of the three convolutions are all 1; the input bit width, output bit width, and weight bit width of the three convolutions are all 8; the activation of the three convolutions are all None, Relu, or Relu6.
8. The method for convolution operator fusion according to claim 7, characterized in that: In step S2, it is understood that the graph contains multiple convolutions, and a convolution contains multiple parameter key-value pairs, including kernel:1,1; stride:1,1. The relationship between the extracted attribute and the judgment condition is to extract the value 1,1 corresponding to the kernel.
9. The method for convolution operator fusion according to claim 8, characterized in that: In step S3, the fused node deletion or node reduction method is to fuse the three convolutions into one convolution, thereby achieving the purpose of reducing two nodes, and fusing the point conv-depthwise conv-point conv three nodes into a new conv node, which reduces two nodes for the entire graph.