A computational graph optimization method, system, and storage medium based on a multi-level cost model.
By optimizing the computation graph through a multi-level cost model and subgraph replacement rules, the problem of the imbalance between search time and optimization effect in existing technologies is solved, and efficient computation of neural network models is achieved in different scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2023-10-20
- Publication Date
- 2026-05-26
AI Technical Summary
Existing computational graph optimization methods lack a balance between search time and optimization effectiveness, and the accuracy and reliability of cost models are insufficient, leading to increased computational resource requirements and limited efficiency and performance of neural networks during training and inference.
A multi-level cost model is adopted, including a first level based on static data prediction, a second level based on the actual execution results of operators, and a third level based on evaluation based on real operating information. Combined with subgraph replacement rules, the computation graph structure is optimized to improve efficiency.
By comprehensively evaluating multi-level cost models, the computational efficiency and performance of neural network models are improved, making them suitable for various scenarios and hardware platforms, and providing broad applicability and optimization effects.
Smart Images
Figure CN117350338B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence, and in particular to a computational graph optimization method, system, and storage medium based on a multi-level cost model. Background Technology
[0002] With the rapid development of artificial intelligence technology, neural networks have played an important role in key fields such as computer vision and natural language processing. However, as the number and complexity of neural network parameters continue to increase, the computational resource requirements during training and inference processes have also increased significantly, leading to problems such as longer computation time and increased memory consumption, which limits the efficiency and performance of neural networks in practical applications.
[0003] Computation graph optimization is one of the key techniques for improving the execution efficiency of neural networks. The computation process of a neural network can be abstracted as a directed acyclic graph, or computation graph. In this graph, edges represent data flow within the neural network, and nodes represent computational operations. Computation graphs contain a large amount of redundant computation and data transmission. By transforming the structure of the computation graph, redundant computation and data transmission can be reduced while maintaining functional equivalence, thereby improving the execution efficiency of the neural network.
[0004] Chinese patent document CN110321999B discloses a method for optimizing a neural network computation graph. The method includes: selecting layer objects capable of layer fusion based at least on optimization rules that reduce the number of data interactions between the computing platform and the external memory; fusing at least two adjacent layers in the computation graph according to the selected layer objects; wherein the at least two adjacent layers are at least one of the following: horizontally adjacent layers with the same feature map input; and vertically adjacent layers where the feature map calculation result of the previous layer is at least a part of the input of the next layer. This invention's computation graph optimization scheme can be performed automatically based on rules or through isomorphic subgraph matching. Thus, the optimal reconstruction method for the execution computation graph is found, improving the execution efficiency of the neural network computing platform.
[0005] Current computational graph optimization methods focus on subgraph replacement sequences based on cost models and backtracking search, aiming to minimize execution costs and find the globally optimal computational graph execution scheme within a huge search space. By establishing a cost model, the execution cost of different computational graph execution schemes can be evaluated, thus guiding the optimization process; backtracking search can find the optimal computational graph execution scheme in a large-scale search space. These methods improve the execution efficiency of neural networks to some extent.
[0006] However, current methods lack attention to balancing search time and optimization effectiveness, as well as the cost model itself. The cost model is used to evaluate the execution costs of different execution schemes, including computational overhead, memory usage, and data transfer. Improving the design and modeling methods of the cost model to enhance its accuracy and reliability is one of the keys to improving the performance of computational graph optimization.
[0007] Therefore, a neural network model optimization method based on a multi-level cost model is still needed, which comprehensively considers search time and optimization effect to adapt to different demand scenarios. Summary of the Invention
[0008] This invention provides a computational graph optimization method, system, and storage medium based on a multi-level cost model. It comprehensively considers the trade-off between search time and optimization effect, can meet different needs in various scenarios, guide the optimization direction of the computational graph, and thus improve the computational efficiency and performance of the neural network model.
[0009] A computational graph optimization method based on a multi-level cost model includes the following steps:
[0010] (1) Define a multi-level cost model to evaluate the execution cost of the computation graph execution scheme;
[0011] (2) Set the set of subgraph replacement rules;
[0012] (3) Based on the given neural network model file, convert it into unified computation graph information;
[0013] (4) Analyze the candidate computation graph structure and apply the subgraph replacement rule to obtain a new computation graph;
[0014] (5) Select a certain level of cost model to evaluate the performance of the computation graph;
[0015] (6) Based on the cost assessment results and scaling factors, select candidate calculation graphs;
[0016] (7) Repeat steps (4) to (6) above until the expected computation graph optimization effect or convergence condition is achieved, and the optimal computation graph is obtained.
[0017] In step (1), the cost model is divided into a first-level cost model, a second-level cost model, and a third-level cost model based on the cost of the cost model and the accuracy of the cost model.
[0018] The first-level cost model makes predictions based on static data such as the number of floating-point operations, the number of parameters, and the size of the input data of the neural network, as well as static hardware information such as the size of the video memory, the clock cycle, and the number of computing cores.
[0019] The second-level cost model predicts based on the actual execution results of a single operator on the target hardware. Since the execution cost of operators with the same parameters on the same hardware is fixed, the execution information of operators is stored in a hash table, and operators with the same configuration only need to be tested once. After obtaining the execution information of each operator, the total execution cost of the operators is used as the total execution cost of the computation graph.
[0020] The third-level cost model is divided into trial operation and formal operation. In the trial operation, CUDA Graph is used to record GPU task execution information, while in the formal operation, the task record information in CUDA Graph is executed directly.
[0021] In step (2), the subgraph replacement rule consists of an atomic graph and a target subgraph. If a subgraph that meets the subgraph replacement rule is found in the computation graph, then that part of the subgraph is replaced with the target subgraph.
[0022] In step (3), computation graph information is extracted based on the model object, including the network hierarchy, the dependencies between nodes, and the flow of data.
[0023] The specific process of step (4) is as follows:
[0024] By traversing the computation graph and checking the type and computational dependencies of each node, examining the inputs and outputs of the nodes, determining the computational order and dependencies between nodes, and obtaining the topological structure information of the computation graph;
[0025] Based on the preset set of subgraph replacement rules, check if there is a subgraph in the computation graph that matches the replacement rules. If it does, perform the replacement operation according to the subgraph replacement rules to obtain a new computation graph.
[0026] After replacing the subgraph, the dependencies between nodes and the connections between nodes need to be updated.
[0027] In step (6), when selecting candidate computation graphs, the scaling factor is set to 1.1 to increase the search space.
[0028] The present invention also provides a computational graph optimization system based on a multi-level cost model, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-described computational graph optimization method.
[0029] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the above-described computational graph optimization method.
[0030] Compared with the prior art, the present invention has the following beneficial effects:
[0031] 1. This invention comprehensively considers the cost model of computational graph optimization at multiple levels, from static data to the actual execution results of operators, and then to the real-world runtime performance evaluation of the entire network. This comprehensive evaluation balances search time and optimization effect, meets different needs in various scenarios, guides the optimization direction of the computational graph, and thus improves the computational efficiency and performance of the neural network model.
[0032] 2. Based on the optimization approach of a multi-level cost model, this invention is applicable to various neural network models and application scenarios. It can be flexibly applied to different computational graph structures and hardware platforms, providing broad applicability. Attached Figure Description
[0033] Figure 1 This is a schematic diagram of the computational graph optimization method based on a multi-level cost model according to the present invention.
[0034] Figure 2 This is an example of the optimization process for the subgraph replacement method used in this invention. Detailed Implementation
[0035] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.
[0036] like Figure 1 As shown, a computational graph optimization method based on a multi-level cost model includes the following steps:
[0037] Step 1: Define a multi-level cost model.
[0038] Based on specific application scenarios and requirements, a multi-level cost model including Level 1, Level 2, and Level 3 is defined to evaluate the execution cost of computation graph execution schemes. The pseudocode of the multi-level cost model used in this invention is as follows:
[0039]
[0040] Level 1: This method predicts based on static data such as the number of floating-point operations (FLOPs), parameter count, and input data size of the neural network, as well as static hardware information such as memory size, clock cycles, and number of computing cores. The prediction model used is a pre-trained model trained on typical convolutional neural network data, employing a multilinear regression algorithm using least squares. This method is simple to implement, easy to train, and has low computational cost, effectively reducing search overhead. Furthermore, it does not require execution on the target hardware, offering high flexibility. However, when the input features and the prediction result are not linearly related, the prediction results are not entirely accurate.
[0041] Level 2: Prediction based on the actual execution results of individual operators on the target hardware. Since the execution cost of operators with the same parameters on the same hardware is fixed, execution information of operators is stored in a hash table, and operators with the same configuration only need to be tested once. After obtaining the execution information of each operator, the sum of the execution costs of the operators is used as the total execution cost of the computation graph. This method has high computational cost but high accuracy.
[0042] Level 3: Based on the actual runtime execution information of the complete computation graph, the optimization direction is guided by the real execution performance of the entire network. Given a computation graph, its computation can be represented by a fixed set of GPU tasks. If this set of GPU task sequences is obtained in advance and passed to the execution engine for direct execution, there is no need to schedule the computation at runtime. Furthermore, given the same input shape, each execution of the convolutional neural network model corresponds to the same GPU task execution sequence. Therefore, the Level 3 cost model is divided into trial run and production run. In the trial run, the CUDA Graph is used to record GPU task execution information, while in the production run, the task record information in the CUDA Graph is directly executed, avoiding runtime scheduling overhead. This method has the highest computational cost but also the highest accuracy, best reflecting the computation graph execution cost in real-world scenarios.
[0043] Step two involves defining the subgraph replacement rule set. The subgraph replacement rule consists of an atomic graph and a target subgraph, determining which subgraphs can be replaced and how. If a subgraph matching the subgraph replacement rule is found in the computational graph, that subgraph can be replaced with the target subgraph, improving computational efficiency without affecting the correctness of the neural network model. Subgraph replacement rules can be manually defined based on rules of thumb and heuristics, such as merging consecutive convolutional layers into a larger convolutional layer. Alternatively, an automatic search method can be used, given an optimization objective, to automatically search and learn a suitable set of subgraph replacement rules.
[0044] Figure 2 This is an example of a subgraph replacement method. It first enlarges a 1x1 convolutional kernel to a 3x3 kernel, and then merges two convolutions of the same size and shape. On certain hardware, this optimization can result in performance improvements.
[0045] Step 3: Based on the given neural network model file, extract the computational graph information. The given model file may originate from various deep learning frameworks (such as TensorFlow, PyTorch, etc.) and have different model file formats. It needs to be converted into a unified abstract computational graph representation. Therefore, it needs to support loading neural network model files of various formats and create model objects containing the structure and parameter information of the neural network. Based on the model object, computational graph information can be extracted, such as the network's hierarchical structure, dependencies between nodes, and data flow. Subsequent steps 4 through 6 will be implemented based on this computational graph information.
[0046] Step four involves analyzing the computation graph of the neural network model. This is done by traversing the computation graph and examining the type and computational dependencies of each node, checking the node's input and output, determining the computational order and dependencies between nodes, and obtaining the topological structure information of the computation graph. Based on a pre-defined set of subgraph replacement rules, the computation graph is checked for subgraphs that conform to the replacement rules. If a subgraph exists, it is replaced according to the rules to obtain a new computation graph. After subgraph replacement, the dependencies between nodes and the connections between nodes need to be updated.
[0047] Step 5: Based on the multi-level cost model constructed in Step 1, evaluate the execution performance of the computation graph. The cost model level needs to be determined based on different use cases and requirements. According to the cost model, evaluate the time cost, data transfer cost, or memory usage cost of computational operations in the computation graph to quantify the computation graph performance and guide the optimization direction of the neural network model. If Level 1 is selected, the evaluation is directly based on static model and hardware information as the computational cost of the computation graph. If Level 2 is selected, the actual running performance of individual operators on the target hardware is summarized as the computational cost of the computation graph. If Level 3 is selected, the evaluation is divided into trial and formal runs of the computation graph, with the formal run evaluation, which removes runtime scheduling overhead, as the computational cost of the computation graph.
[0048] Step six: Based on the evaluation results of the multi-level cost model, select computational graphs that meet performance requirements in terms of memory usage, computation time, and data transfer overhead as candidate computational graphs for the next round of search. A scaling factor α is used to control the search space size. If α is set to 1, it's a simple greedy search; a new computational graph is only added to the candidate computational graph set if its execution performance is significantly better than the current one. This approach may lead to missing the global optimum. Figure 2In this method, expanding a 1x1 convolution to a 3x3 convolution leads to a performance degradation, and the greedy search with α = 1 cannot find this optimized path. When α is greater than 1, a temporary performance degradation of the computation graph is allowed during the search process to increase the search space and obtain a globally better computation graph execution scheme. It should be noted that when α is set too large, it will result in excessively long search time. In this method, this parameter is set to 1.1.
[0049] Step 7: Repeat the iteration. Repeat steps 4 through 6 until the expected optimization effect or convergence condition of the computational graph is achieved, and the optimal computational graph is obtained.
[0050] The pseudocode for the subgraph replacement algorithm is as follows:
[0051]
[0052] Based on the same inventive principle, this embodiment also provides a computational graph optimization device system based on a multi-level cost model, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the computational graph optimization method of the above embodiment.
[0053] Based on the same inventive principle, this embodiment also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the computational graph optimization method of the above embodiment.
[0054] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A computational graph optimization method based on a multi-level cost model, characterized in that, Includes the following steps: (1) Define a multi-level cost model to evaluate the execution cost of the computation graph execution scheme; specifically: Based on the cost of the cost model and its accuracy, cost models are divided into Level 1 cost models, Level 2 cost models, and Level 3 cost models. The first-level cost model makes predictions based on static data such as the number of floating-point operations, the number of parameters, and the size of the input data of the neural network, as well as static hardware information such as the size of the video memory, the clock cycle, and the number of computing cores. The second-level cost model predicts based on the actual execution results of a single operator on the target hardware. Since the execution cost of operators with the same parameters on the same hardware is fixed, the execution information of operators is stored in a hash table, and operators with the same configuration only need to be tested once. After obtaining the execution information of each operator, the total execution cost of the operators is used as the total execution cost of the computation graph. The third-level cost model is divided into trial operation and formal operation. In the trial operation, CUDA Graph is used to record GPU task execution information, while in the formal operation, the task record information in CUDA Graph is executed directly. (2) Set the set of subgraph replacement rules; (3) Based on the given neural network model file, convert it into unified computation graph information; (4) Analyze the candidate computation graph structure and apply the subgraph replacement rule to obtain a new computation graph; (5) Select a certain level of cost model to evaluate the execution performance of the computation graph; (6) Based on the cost assessment results and scaling factors, select candidate calculation graphs; (7) Repeat steps (4) to (6) above until the expected computation graph optimization effect or convergence condition is achieved, and the best computation graph is obtained.
2. The computational graph optimization method based on a multi-level cost model according to claim 1, characterized in that, In step (2), the subgraph replacement rule consists of an atomic graph and a target subgraph. If a subgraph that meets the subgraph replacement rule is found in the computation graph, then that part of the subgraph is replaced with the target subgraph.
3. The computational graph optimization method based on a multi-level cost model according to claim 1, characterized in that, In step (3), computation graph information is extracted based on the model object, including the network hierarchy, the dependencies between nodes, and the flow of data.
4. The computational graph optimization method based on a multi-level cost model according to claim 1, characterized in that, The specific process of step (4) is as follows: By traversing the computation graph and checking the type and computational dependencies of each node, examining the inputs and outputs of the nodes, determining the computational order and dependencies between nodes, and obtaining the topological structure information of the computation graph; Based on the preset set of subgraph replacement rules, check if there is a subgraph in the computation graph that matches the replacement rules. If it does, perform the replacement operation according to the subgraph replacement rules to obtain a new computation graph. After replacing the subgraph, the dependencies between nodes and the connections between nodes need to be updated.
5. The computational graph optimization method based on a multi-level cost model according to claim 1, characterized in that, In step (6), when selecting candidate computation graphs, the scaling factor is set to 1.1 to increase the search space.
6. A computational graph optimization system based on a multi-level cost model, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the computational graph optimization method according to any one of claims 1-5.
7. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the computational graph optimization method as described in any one of claims 1-5.