Multi-layer convolution operator fusion optimization method, device, equipment, medium and product

Abstract

The invention discloses a multilayer convolution operator fusion optimization method and device, equipment, a medium and a product, and relates to the technical field of compilation optimization, and the method comprises the steps: determining an original calculation graph of a to-be-deployed model, and determining a plurality of fusible operator sub-graphs in the original calculation graph based on a hardware performance model; optimizing the original calculation graph based on the fusible operator sub-graph to obtain an optimized calculation graph; obtaining a parallel optimization code of the to-be-deployed model based on the optimization calculation graph and the fragment size; the target machine runs the parallel optimization code to obtain the optimization performance of the to-be-deployed model; according to the method, the original calculation graph is optimized based on the fusible operator sub-graph, so that the performance of the target machine during execution of the neural network model can be improved.

Description

Technical Field

[0001] The present application relates to the field of compilation optimization technology, and in particular to a multi-layer convolution operator fusion optimization method, device, equipment, medium and product. Background Art

[0002] As an important branch of deep learning, convolutional neural networks (CNNs) have achieved remarkable results in many fields such as image recognition, object detection, and speech recognition. In convolutional neural networks, convolutional layers and pooling layers are the basic units that constitute the network. They extract features from images through convolution and pooling operations. However, with the complexity of the network structure, the amount of computation and memory consumption have increased significantly, becoming one of the key factors restricting the performance improvement of CNN. In the deep convolutional neural network (DCNN) framework, the computational graph is a very important concept that graphically represents the layer structure, data flow, and dependencies between operations of the neural network. The computational graph enables the framework to optimize the execution process, including but not limited to memory usage and computational efficiency. In the inference stage, operator fusion is a common optimization technique that merges multiple operations into a single operation, thereby reducing the number of memory accesses, reducing data copying and cache invalidation, and improving computational efficiency. For example, if the output of a neural network layer is directly used as the input of the next layer, the forward propagation operations of the two layers can be merged during inference to avoid unnecessary intermediate data storage. Different deep learning frameworks, such as TensorFlow, PyTorch, and MXNet, have their own mechanisms and strategies for implementing operator fusion. These frameworks usually provide a series of tools and application programming interfaces that allow developers to manually optimize computational graphs or automatically apply optimization strategies. The key technologies for operator fusion include: graph optimization and parallel execution optimization. Graph optimization refers to analyzing and transforming computational graphs before execution to identify operators that can be fused, and to reduce the overhead of memory allocation and release by reusing the memory of intermediate data, thereby improving the execution efficiency of deep learning models. Existing framework technologies such as TensorFlow and TensorAlgebraSuperOptimizer (TASO) provide advanced programming interfaces to implement operator fusion and other optimizations by automatically analyzing and transforming computational graphs. XLA (Accelerated LinearAlgebra) implements operator fusion and memory optimization by converting the entire computational graph into a single, highly optimized machine code. By reusing the memory of intermediate data, the overhead of memory allocation and release is reduced. TASO pays special attention to horizontal operator fusion, improving program parallelism, and thus increasing the running speed of the model. Parallel execution automatic optimization refers to the compiler identifying opportunities for parallel execution through code generation and allocating operators that can be processed in parallel to multiple processing units. Existing work mainly includes Halide and Tensor Virtual Machine (TVM).Halide automatically fuses and parallelizes operators by analyzing algorithm dependencies and data access patterns to optimize execution efficiency. TVM's AutoTuner can search for the optimal parallel execution strategy, including parameters such as the number of threads and block size, to adapt to different hardware and models. Operator implementation is optimized for specific hardware (such as GPU, TPU) to take advantage of the parallel processing capabilities of the hardware.

[0003] The execution based on the traditional computation graph may have the problem of low hardware efficiency, especially the large overhead in data access. The background technology may include the existing neural network computation graph construction and execution method, and the operator fusion technology in the deep learning compiler. The invention patent with the publication number CN110321999A, entitled Neural Network Computation Graph Optimization Method (hereinafter referred to as invention [1]) improves efficiency by reducing the number of data interactions between the computing platform and the external memory. The specific method is to select horizontally adjacent layers with the same feature map input based on the optimization rule, or vertically adjacent layers whose feature map calculation results of the previous layer are the inputs of the next layer, and fuse at least two adjacent layers according to these layer objects. The invention patent with the publication number CN112711422B, entitled A Fusion Operator Design Method and Heterogeneous Computing System for Heterogeneous Computing (hereinafter referred to as invention [2]) provides an optimized compilation method, which reduces the storage and access requirements of intermediate results by fusing the preprocessing layers in the neural network, determines the calculation order of the operators in the fusion layer, obtains the input and output call times and splitting strategy of the operators, thereby improving the execution efficiency and realizing an automated compilation optimization process. The invention patent with publication number CN115374912A, entitled A method for designing fusion operators and heterogeneous computing systems for heterogeneous computing (referred to as invention [3]), redesigns and splits the normalization operator through specific steps to optimize the interaction of data in the heterogeneous computing system. The invention patent with publication number CN115659281A, entitled A method and device for adaptively accelerating operator fusion (referred to as invention [4]), adds key attributes to the nodes of the neural network calculation graph, and performs graph weighting, and uses deep reinforcement learning technology to adaptively determine the optimal fusion scheme of the operator. The invention patent with publication number CN110490309A, entitled A method for operator fusion for neural networks and related products thereof (referred to as invention [5]), realizes efficient fusion of operators by splitting and mapping the output data of the operators, and making fusion judgments based on the size of these data blocks. Although the above optimization methods have alleviated the computing and memory pressure of CNN to a certain extent, the following problems still exist: 1. Insufficient optimization of the calculation graph: Most existing optimization methods optimize a single operator or a local calculation graph, and lack a global optimization strategy for the entire calculation graph. This results in a large amount of redundant calculations and memory accesses during the computational graph compilation process. Existing fusion methods often target specific operators and lack the ability to generalize and optimize convolutional fusion computational graphs. They have limitations in terms of operator fusion, etc.: Invention [1] can only select horizontal or vertical fusion layer objects, cannot fuse multiple layers, and cannot combine hardware performance models for targeted performance optimization. Invention [2] has limited fusion capabilities and is limited to the optimization of preprocessing layers. 2. Poor flexibility: Different CNN models have different structures and computing requirements, and existing optimization methods often lack sufficient flexibility and are difficult to adapt to various complex network structures and hardware structures.The fusion method proposed in invention [3] focuses on the fusion of convolution, normalization and activation layers, but lacks consideration of the fusion scheme of multiple general operators. Inventions [4] and [5] focus on the optimization of the computational graph itself, but do not perform targeted optimization based on the characteristics of the hardware. Summary of the invention

[0004] The purpose of this application is to provide a multi-layer convolution operator fusion optimization method, device, equipment, medium and product, which can optimize the neural network model deployed on the target machine and improve the performance of the target machine when executing the neural network model.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a multi-layer convolution operator fusion optimization method, comprising:

[0007] The host computer is used to analyze the computation graph of the model to be deployed to determine the original computation graph; the model to be deployed is a convolutional neural network model; the original computation graph includes multiple nodes and multiple edges; one edge is used to connect two nodes; the node is used to describe the operation or data in the model to be deployed; the edge is used to describe the data flow and dependency relationship between nodes; the dependency relationship is a first dependency relationship {1×1} or a second dependency relationship {α×α}, where α is the size of the convolution filter;

[0008] Build a hardware performance model of the target machine;

[0009] Based on the hardware performance model, determining a plurality of fusible operator subgraphs in the original computation graph;

[0010] Optimizing the original computation graph based on the fusible operator subgraph to obtain an optimized computation graph;

[0011] Based on the optimized computation graph and shard size, the parallel optimized code of the model to be deployed is obtained;

[0012] The target machine runs the parallel optimization code to obtain the optimization performance of the model to be deployed; the performance indicators describing the optimization performance include execution time and resource consumption.

[0013] Optionally, before using the host computer to analyze the calculation graph of the model to be deployed and determining the original calculation graph, the following steps are also included:

[0014] The model to be deployed is deployed on the target machine and runs to obtain the original calculation results and original performance of the model to be deployed; the performance indicators describing the original performance include execution time and resource consumption.

[0015] Optionally, after the target machine runs the optimized code and obtains the optimized performance of the model to be deployed, the following steps are further included:

[0016] The multi-layer convolution operator fusion optimization strategy is adjusted according to the optimization performance and the original performance; the multi-layer convolution operator fusion optimization strategy includes: slice size.

[0017] Optionally, the fusion category of the fusible operator subgraphs is vertical fusion, horizontal-vertical fusion or vertical-horizontal fusion;

[0018] The vertical fusion is used to express a single-input single-output structure;

[0019] The horizontal-vertical fusion is used to express a multi-input single-output structure;

[0020] The vertical-horizontal fusion is used to express a single-input multiple-output structure.

[0021] Optionally, optimizing the original computation graph based on the fusible operator subgraph to obtain an optimized computation graph includes:

[0022] Determine that the original computation graph is the optimized computation graph at the 0th iteration;

[0023] Determine any fusionable operator subgraph as a subgraph to be optimized;

[0024] Let the number of iterations n = 1;

[0025] Determine whether a non-redundant calculation condition is met to obtain a first determination result; the non-redundant calculation condition is whether the dependency relationships between all output nodes and subsequent operators in the subgraph to be optimized are all first dependency relationships {1×1}; the output node is a node connected to the output quantity;

[0026] If the first judgment result is yes, directly merge all operators in the subgraph to be optimized into a fused operator node;

[0027] If the first judgment result is no, redundant calculation is performed on all operators in the subgraph to be optimized to obtain a fused operator node;

[0028] In the optimization calculation graph at the n-1th iteration, all output nodes in the subgraph to be optimized are replaced with fusion operator nodes to obtain a pending calculation graph;

[0029] Deploy the model to be deployed on the target machine based on the pending computation graph and run it to obtain the pending computation results and pending performance;

[0030] Determine whether a fusion condition is met to obtain a second determination result; the fusion condition is that the pending calculation result is the same as the original calculation result, and the pending performance is better than the original performance;

[0031] If the second judgment result is yes, then the undetermined calculation graph is determined to be the optimized calculation graph at the nth iteration;

[0032] If the second judgment result is no, determining the optimization calculation graph at the n-1th iteration as the optimization calculation graph at the nth iteration;

[0033] Update the subgraph to be optimized, increase the value of the iteration number n by 1, and return to the step "determine whether the non-redundant calculation condition is met and obtain the first judgment result" until all fusible operator subgraphs are traversed, and the optimized calculation graph at the nth iteration is obtained as the optimized calculation graph.

[0034] Optionally, before optimizing the original computation graph based on the fusible operator subgraph to obtain an optimized computation graph, the method further includes:

[0035] Determine the intermediate data of each fusible operator subgraph;

[0036] Remove fusible operator subgraphs whose intermediate data size is larger than the target machine's shared memory capacity.

[0037] In a second aspect, the present application provides a multi-layer convolution operator fusion optimization device. Optionally, the multi-layer convolution operator fusion optimization device applies the multi-layer convolution operator fusion optimization method according to any one of claims 1 to 6. The multi-layer convolution operator fusion optimization device includes:

[0038] The original calculation graph determination module uses the host computer to analyze the calculation graph of the model to be deployed and determine the original calculation graph;

[0039] A hardware performance model building module is used to build a hardware performance model of the target machine;

[0040] A fusible operator subgraph determination module, used to determine multiple fusible operator subgraphs in the original computation graph based on the hardware performance model;

[0041] An optimized computation graph determination module, used to optimize the original computation graph based on the fusible operator subgraph to obtain an optimized computation graph;

[0042] The optimized code compilation module is used to obtain the parallel optimized code of the model to be deployed based on the optimized calculation graph and shard size;

[0043] The optimization code running module is used to control the target machine to run the parallel optimization code to obtain the optimization performance of the model to be deployed.

[0044] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned multi-layer convolution operator fusion optimization method.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multi-layer convolution operator fusion optimization method.

[0046] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the above-mentioned multi-layer convolution operator fusion optimization method when executed by a processor.

[0047] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0048] The present application provides a multi-layer convolution operator fusion optimization method, device, equipment, medium and product, which determines the original calculation graph of the model to be deployed, determines multiple fusionable operator subgraphs in the original calculation graph based on the hardware performance model; optimizes the original calculation graph based on the fusionable operator subgraph to obtain an optimized calculation graph; obtains the parallel optimization code of the model to be deployed based on the optimized calculation graph and the slice size; the target machine runs the parallel optimization code to obtain the optimized performance of the model to be deployed; from the traditional single-input single-output mode, it is expanded to the general mode of single-input multiple output, multiple-input single output and multiple-input multiple output, so as to achieve a wider and more flexible fusion optimization. In view of the use of a general dependency relationship in the calculation graph, a solution based on redundant calculation is proposed to further expand the fusion capability and ensure that the optimization method can adapt to various complex network structures and computing requirements; it is proposed to use the roof model of performance-memory access bandwidth as a cost model, which can comprehensively consider the relationship between computing performance and memory access bandwidth, and provide precise guidance for parallel code generation. Based on the cost model, a fusion parallel code that is more suitable for a specific hardware architecture will be generated, which will reduce unnecessary memory access operations, improve computing density, and give full play to hardware performance. By optimizing the code generation strategy, this application aims to reduce program startup overhead while improving the code's adaptability to hardware to ensure good performance on different hardware platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 This is a flow chart of a multi-layer convolution operator fusion optimization method in one embodiment of the present application.

[0051] Figure 2 A fusionable operator subgraph of the type of vertical fusion in an embodiment of the present application

[0052] Figure 3 A fusionable operator subgraph of the type horizontal-vertical fusion in one embodiment of the present application

[0053] Figure 4 A fusionable operator subgraph of the type of vertical-horizontal fusion in an embodiment of the present application

[0054] Figure 5 This is a first schematic diagram of a non-redundant computing structure in an embodiment of the present application;

[0055] Figure 6 This is a second schematic diagram of a non-redundant computing structure in an embodiment of the present application;

[0056] Figure 7 This is a third schematic diagram of a non-redundant computing structure in an embodiment of the present application;

[0057] Figure 8 This is a fourth schematic diagram of a non-redundant computing structure in an embodiment of the present application;

[0058] Fig. 9 This is a first schematic diagram of a redundant computing structure in an embodiment of the present application;

[0059] Fig.10 This is a second schematic diagram of a redundant computing structure in an embodiment of the present application;

[0060] Fig.11 This is a third schematic diagram of a redundant computing structure in an embodiment of the present application;

[0061] Fig.12 This is a fourth schematic diagram of a redundant computing structure in an embodiment of the present application;

[0062] Fig.13 This is a schematic diagram of a fusible operator subgraph before fusion in one embodiment of the present application;

[0063] Fig.14 This is a schematic diagram of fusion of fusionable operator subgraphs in one embodiment of the present application;

[0064] Fig.15 This is a diagram of the overall architecture of multi-layer convolution operator fusion optimization in one embodiment of the present application.

[0065] Fig.16 It is a structural schematic diagram of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0067] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0068] In existing computational graph optimization, fusion optimization methods are often limited to specific and limited subgraph types, such as single-input single-output mode, which limits the comprehensiveness and depth of the optimization effect. With the complexity of network structure, traditional fusion strategies can only optimize operators with many-to-one dependencies, which can hardly meet the needs of efficient computing. In addition, there is insufficient support for the fusion of complex subgraphs such as multi-input and multi-output, multi-input and single output, resulting in a large amount of potential parallelism and computing resource utilization not being fully explored. In addition, in the existing deep learning framework, there are two major problems with the code generation method: one is to select the code through the multi-version method in the computing library. Although this method can adapt to different hardware, it increases the startup overhead of the program and affects the overall performance; the second is to use a hardware-independent code generation method. Although it simplifies the development process, the generated code has poor adaptability to specific hardware and it is difficult to give full play to the hardware performance. Therefore, in an exemplary embodiment, if Figure 1 As shown, a multi-layer convolution operator fusion optimization method is provided, including:

[0069] Step 101: Use the host computer to analyze the computation graph of the model to be deployed and determine the original computation graph. The model to be deployed is a convolutional neural network model. The original computation graph includes multiple nodes and multiple edges. An edge is used to connect two nodes. The node is used to describe the operation or data in the model to be deployed. The edge is used to describe the data flow and dependency between nodes. The dependency is the first dependency {1×1} or the second dependency {α×α}, where α is the size of the convolution filter.

[0070] Step 102: Build a hardware performance model of the target machine.

[0071] Specifically, optimization decision-making computer calculation graph analysis is used.

[0072] The execution subject is the optimization decision-making computer (host computer), and the calculation graph analysis steps include:

[0073] Step 1.1: Load and parse the model computation graph. Read the structure of the model computation graph from the input file, including nodes (representing operations or data) and edges (representing data flow and dependencies).

[0074] Step 1.2: Analyze the computational memory access characteristics of the nodes. Identify nodes that can be optimized in the computational graph, especially nodes such as convolutional layers and addition layers with relatively low computational memory access.

[0075] Step 1.3: Build a hardware performance model for the target machine. Based on the hardware's parallel computing capabilities and memory access bandwidth, evaluate the types of performance bottlenecks that may exist for the application, providing a basis for subsequent operator fusion.

[0076] Step 1.4: Output analysis results. Give the calculation bottleneck analysis results of the computing node for the specified hardware, summarize the structural characteristics of the calculation graph and the node dependency, and provide reference information for the subsequent operator fusion steps.

[0077] Step 103: Based on the hardware performance model, multiple fusionable operator subgraphs in the original computation graph are determined. The fusion category of the fusionable operator subgraphs is vertical fusion, horizontal-vertical fusion, or vertical-horizontal fusion.

[0078] like Figure 2 , vertical fusion is used to express a single-input single-output structure.

[0079] like Figure 3 ,Horizontal-vertical fusion is used to express a multi-input single-output structure.

[0080] like Figure 4 ,Vertical-horizontal fusion is used to express a single-input multiple-output structure.

[0081] Step 104: Optimize the original computation graph based on the fusible operator subgraph to obtain an optimized computation graph.

[0082] Specifically, given a candidate set of operator fusion subgraphs, the dependency bounds are calculated to ensure:

[0083] Execution entity: Optimization decision computer (host computer), the fusion steps include:

[0084] Step 2.1: Identify fusionable operator subgraphs. Based on the analysis results of step 1.3, find operator pairs that can be fused and divide the operator subgraphs into three types for fusion mode candidates. The three types are: Figures 2 to 4 As shown, they are vertical fusion, horizontal-vertical fusion, and vertical-horizontal fusion.

[0085] Step 2.2: Implement fusion operation. There are two cases when fusion operation is performed. Case 1 is the case of non-redundant calculation, and the subsequent operator dependency is element-level dependency, that is, the dependency shape is {1×1}, such as Figures 5 to 8 As shown, fusion can be performed directly; Case 2 is the case of redundant calculation, and the shape of the subsequent operator dependency is {α×α}, such as Figures 9 to 12 , it is necessary to perform redundant calculations on the single stream processor used according to the specific shape of the operator and the dependencies of the output layer operator shards and the dependencies of the intermediate layers in the parallel computing, without performing very expensive synchronization operations, to ensure the correctness of the calculation results.

[0086] Step 2.3: Evaluate the benefits of fusion. Calculate the benefits that fusion may bring, such as reduced computational workload and optimized memory access, to ensure that fusion is profitable in the performance model. In particular, for operators with redundant computation, the cost of redundant computation needs to be reflected in the performance model.

[0087] Step 2.4: Verify the correctness of subgraph fusion and replace it. Ensure that the fused computational graph result is consistent with the original computational graph. Figure 1 The selected operator pairs are fused, the new fusion operator nodes are used to replace the original subgraphs, and the computation graph is updated. The fusion operator subgraphs before and after fusion can be as follows: Fig.13 and Fig.14 shown.

[0088] Step 105: Based on the optimized computation graph and the shard size, obtain the parallel optimized code of the model to be deployed.

[0089] Specifically, hardware customized compilation optimization is implemented, and template operators are automatically generated.

[0090] Execution entity: The optimization decision computer (host computer) works in collaboration with the target machine (slave computer).

[0091] The parallel optimization code generation steps include:

[0092] Step 3.1: Screen candidate subgraphs for optimization through performance models to reduce performance overhead. Based on the characteristics of the target machine, including hardware information such as processor architecture, memory bandwidth, cache size, shared memory size, and optimized instruction sets. Based on the limitations of the shared memory capacity of the target machine, remove subgraphs that cannot be fused due to the large size of intermediate data; based on the fused computing performance model and hardware performance model information, screen operators with performance optimization potential for optimization.

[0093] Step 3.2: Generate parallel optimization code. According to the fusion strategy, formulate a targeted fusion code parallel optimization strategy, slice the input matrix in the length and width dimensions, perform inter-GPU block parallelism, and perform intra-GPU block thread parallelism in the slice length, slice width and channel dimensions of the input matrix.

[0094] Step 3.3: Customize compilation strategy. According to the characteristics of the target machine, formulate targeted compilation optimization strategies, such as instruction selection, loop unrolling, memory access optimization, etc. Convert the optimized computation graph into code optimized for the target machine, including code that utilizes specific assembly instructions or utilizes specific hardware accelerators (such as tensor matrix multiplication instructions).

[0095] Step 3.4: Optimization correctness verification. Deploy the optimized code to the target machine, verify whether the calculation results are consistent with the original calculation results, and prepare for performance verification of deployment to the target machine.

[0096] Step 106: The target machine runs the parallel optimization code to obtain the optimization performance of the model to be deployed. The performance indicators describing the optimization performance include execution time and resource consumption.

[0097] Before step 101, the method further includes:

[0098] Step 107: Deploy the model to be deployed on the target machine and run it to obtain the original calculation results and original performance of the model to be deployed. The performance indicators describing the original performance include execution time and resource consumption.

[0099] After step 106, the method further includes:

[0100] Step 108: Adjust the multi-layer convolution operator fusion optimization strategy according to the optimized performance and the original performance. The multi-layer convolution operator fusion optimization strategy includes: slice size.

[0101] Specific target machine performance verification and parallel parameter iterative optimization.

[0102] Execution subject: target machine (lower computer).

[0103] The target machine performance verification steps include:

[0104] Step 4.1: Configure the test environment. Make sure the target machine is in a state suitable for testing, including shutting down other programs, setting a stable power supply and fixing the hardware clock cycle frequency, etc.

[0105] Step 4.2: Execute the optimized program. Run the optimized program on the target machine and record performance indicators such as execution time and resource consumption. Comparative analysis. Compare the optimized performance indicators with the original performance to evaluate the optimization effect.

[0106] Step 4.3: For the proposed fusion method of the parallel optimization code generation step, multiple key performance factors, such as shard parallelism, intermediate data size, and the number of redundant calculations, all involve the shard size, so an iterative optimization of the shard size is adopted.

[0107] Step 4.4: Feedback and optimization iteration: Provide feedback and adjust the optimization strategy based on the verification results.

[0108] Before step 104, the method further includes:

[0109] Step 109: Determine the intermediate data of each fusible operator subgraph.

[0110] Step 1010: Delete the fusible operator subgraphs whose intermediate data size is larger than the target machine's shared memory capacity.

[0111] Step 104 includes:

[0112] Step 104-1: Determine that the original computation graph is the optimized computation graph at the 0th iteration.

[0113] Step 104 - 2 : Determine any fusible operator subgraph as a subgraph to be optimized.

[0114] Step 104 - 3 : Set the number of iterations n=1.

[0115] Step 104-4: Determine whether the non-redundant calculation condition is met, and obtain a first determination result. The non-redundant calculation condition is whether the dependency relationship between all output nodes and subsequent operators in the subgraph to be optimized is the first dependency relationship {1×1}. The output node is a node connected to the output quantity.

[0116] Step 104 - 5 : If the first judgment result is yes, all operators in the subgraph to be optimized are directly merged into a fused operator node.

[0117] Step 104 - 6 : If the first judgment result is no, redundant calculations are performed on all operators in the subgraph to be optimized to obtain a fused operator node.

[0118] Step 104-7: In the optimization calculation graph at the n-1th iteration, all output nodes in the subgraph to be optimized are replaced with fusion operator nodes to obtain a pending calculation graph.

[0119] Step 104-8: Deploy the model to be deployed on the target machine based on the pending computation graph and run it to obtain pending computation results and pending performance.

[0120] Step 104-9: Determine whether the fusion condition is met and obtain a second determination result. The fusion condition is that the pending calculation result is the same as the original calculation result, and the pending performance is better than the original performance.

[0121] Step 104 - 10: If the second judgment result is yes, determine that the pending calculation graph is the optimized calculation graph at the nth iteration.

[0122] Step 104 - 11: If the second judgment result is no, then determine the optimization calculation graph at the n-1th iteration as the optimization calculation graph at the nth iteration.

[0123] Step 104-12: Update the subgraph to be optimized, increase the value of the iteration number n by 1, and return to step 104-4 until all fusible operator subgraphs are traversed, and the optimized calculation graph at the nth iteration is obtained as the optimized calculation graph.

[0124] like Fig.15 By fusing operators such as convolution and addition, combined with instruction scheduling and pipelining technology, the computing efficiency is improved. The optimization scheme related to the target machine (lower machine) is implemented on the optimization decision computer (host computer), and the key steps such as calculation graph analysis, operator fusion, hardware customized compilation optimization are optimized, and finally the performance verification is performed on the target machine.

[0125] Load the computational graph, identify possible fusion subgraphs, give fusion subgraph candidates, and give the hardware performance model function:

[0126]

[0127] Where Op represents the operator, p(·) represents the theoretical maximum performance of executing the operator on the target machine, c(·) is the operator's computational memory access ratio, and p max is the peak performance of the target machine, c Peak is the computational access-to-memory ratio when the target machine reaches peak performance. The model is basically consistent with the classic rooftop performance model. Based on the above calculation results, the theoretical computational performance peak of the operator on the hardware can be obtained. When the theoretical computational access-to-memory ratio of the operator is higher than the computational access-to-memory ratio of the target machine, the bottleneck is memory access, and the peak performance of the machine hardware has not been reached. Its peak performance depends on the computational access-to-memory ratio of the operator itself; conversely, the bottleneck is computation, and the peak performance of the machine hardware can be reached. Its peak performance should theoretically be equal to the peak performance of the hardware.

[0128] according to Fig.15 The fusion rules shown are used to traverse and match the computational graph, thereby generating the correct high-performance computing code. Fig.11 In the case shown, it is necessary to perform redundant calculations on CONV1 at the block parallel level to ensure the correct calculations of CONV2 and CONV3, and give a parallel code with redundant calculations.

[0129] A fusion subgraph performance model function is given to prune operators that exceed the shared memory capacity and have no possibility of performance gain.

[0130] For a general operator, its performance model function can be evaluated using the number of computations and global memory accesses:

[0131]

[0132] Among them, c(·) is the computational memory access ratio of the operator, comp(·) represents the computation of the operator, and gm ld (·) is the access cost to global memory, which can be optimized by using read-only memory, gm st (·) is the optimization for writing to global memory. Using this performance model function, the computation-to-memory ratio of general operators can be calculated, which can be used as an indirect indicator to reflect the relative performance of executing the function on the hardware device.

[0133] for Fig.11 The case shown here is a model function with redundant calculations:

[0134]

[0135] Among them, reComp(·) represents the redundant calculation introduced for fusion. in is the input of the operator subgraph, out is the output of the operator subgraph, and flt represents the filter. According to the formula of the operator calculation and the access ratio of the global memory, the sharding parameters of the parallel operator are flexibly adjusted. Specific operation: If the calculation access ratio increases significantly, it means that the adopted sharding size is too small, and the sharding size should be appropriately increased, and the parallelism should be adjusted at the same time; vice versa.

[0136] Parallel tuning. Tune key parallel parameters. The width of the shard is defined as T x , the length of the slice is defined as T y , respectively optimizing the distribution of length and width can better explore the search space more comprehensively. Pruning the length and width of non-approximate factors and odd numbers can improve the exploration efficiency of the search space.

[0137] This embodiment extends from the traditional single-input single-output mode to the general modes of single-input multiple-output, multiple-input single-output and multiple-input multiple-output, to achieve a wider and more flexible fusion optimization. A solution based on redundant computing is proposed for the computational graph using a general dependency relationship, and the fusion capability is further expanded to ensure that the optimization method can adapt to various complex network structures and computing requirements; it is proposed to use the roof model of performance-memory access bandwidth as a cost model, which can comprehensively consider the relationship between computing performance and memory access bandwidth, and provide precise guidance for parallel code generation. Based on the cost model, a fusion parallel code that is more suitable for a specific hardware architecture will be generated, reducing unnecessary memory access operations, improving computing density, and giving full play to hardware performance. By optimizing the code generation strategy, this application aims to reduce program startup overhead, while improving the adaptability of the code to hardware, and ensuring good performance on different hardware platforms. It can integrate subgraph types that cannot be integrated in previous technical solutions, consider the performance optimization benefits and overhead brought by redundant computing, and perform hardware adaptive fusion according to the hardware performance model and the operator fusion performance model.

[0138] This embodiment can integrate sub-graph types that cannot be integrated in previous technical solutions, consider the performance optimization benefits and overheads brought by redundant calculations, and perform hardware adaptive fusion based on the hardware performance model and the operator fusion performance model. Through the fusion of general convolution operators, combined with code generation and compilation optimization technology, a new method of computer hardware adaptive software code optimization is customized to improve computing efficiency and reduce computing time.

[0139] In an exemplary embodiment, a multi-layer convolution operator fusion optimization device is provided, characterized in that the multi-layer convolution operator fusion optimization device applies the multi-layer convolution operator fusion optimization method, and the multi-layer convolution operator fusion optimization device includes:

[0140] The original calculation graph determination module uses the host computer to analyze the calculation graph of the model to be deployed and determine the original calculation graph.

[0141] The hardware performance model building module is used to build the hardware performance model of the target machine.

[0142] The fusible operator subgraph determination module is used to determine multiple fusible operator subgraphs in the original calculation graph based on the hardware performance model.

[0143] The optimized computation graph determination module is used to optimize the original computation graph based on the fusible operator subgraph to obtain an optimized computation graph.

[0144] The optimized code compilation module is used to obtain the parallel optimized code of the model to be deployed based on the optimized calculation graph and shard size.

[0145] The optimization code running module is used to control the target machine to run the parallel optimization code to obtain the optimization performance of the model to be deployed.

[0146] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.16 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-layer convolution operator fusion optimization method is implemented.

[0147] Those skilled in the art will understand that Fig.16 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0148] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0149] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0150] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0151] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0152] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0153] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A multi-layer convolution operator fusion optimization method, characterized in that: include: The host computer is used to analyze the computation graph of the model to be deployed to determine the original computation graph; the model to be deployed is a convolutional neural network model; the original computation graph includes multiple nodes and multiple edges; one edge is used to connect two nodes; the node is used to describe the operation or data in the model to be deployed; the edge is used to describe the data flow and dependency relationship between nodes; the dependency relationship is a first dependency relationship {1×1} or a second dependency relationship {α×α}, where α is the size of the convolution filter; Build a hardware performance model of the target machine; Based on the hardware performance model, determining a plurality of fusible operator subgraphs in the original computation graph; Optimizing the original computation graph based on the fusible operator subgraph to obtain an optimized computation graph; Based on the optimized computation graph and shard size, the parallel optimized code of the model to be deployed is obtained; The target machine runs the parallel optimization code to obtain the optimized performance of the model to be deployed; The performance indicators describing the optimization performance include execution time and resource consumption.

2. The multi-layer convolution operator fusion optimization method according to claim 1, characterized in that: Before using the host computer to analyze the calculation graph of the model to be deployed and determine the original calculation graph, it also includes: The model to be deployed is deployed on the target machine and runs to obtain the original calculation results and original performance of the model to be deployed; the performance indicators describing the original performance include execution time and resource consumption.

3. The multi-layer convolution operator fusion optimization method according to claim 2, characterized in that: After running the optimized code on the target machine and obtaining the optimized performance of the model to be deployed, it also includes: The multi-layer convolution operator fusion optimization strategy is adjusted according to the optimization performance and the original performance; the multi-layer convolution operator fusion optimization strategy includes: slice size.

4. The multi-layer convolution operator fusion optimization method according to claim 1, characterized in that: The fusion category of the fusible operator subgraphs is vertical fusion, horizontal-vertical fusion or vertical-horizontal fusion; The vertical fusion is used to express a single-input single-output structure; The horizontal-vertical fusion is used to express a multi-input single-output structure; The vertical-horizontal fusion is used to express a single-input multiple-output structure.

5. The multi-layer convolution operator fusion optimization method according to claim 1, characterized in that: Optimizing the original computation graph based on the fusible operator subgraph to obtain an optimized computation graph includes: Determine that the original computation graph is the optimized computation graph at the 0th iteration; Determine any fusionable operator subgraph as a subgraph to be optimized; Let the number of iterations n = 1; Determine whether a non-redundant calculation condition is met to obtain a first determination result; the non-redundant calculation condition is whether the dependency relationships between all output nodes and subsequent operators in the subgraph to be optimized are all first dependency relationships {1×1}; the output node is a node connected to the output quantity; If the first judgment result is yes, directly merge all operators in the subgraph to be optimized into a fused operator node; If the first judgment result is no, redundant calculation is performed on all operators in the subgraph to be optimized to obtain a fused operator node; In the optimization calculation graph at the n-1th iteration, all output nodes in the subgraph to be optimized are replaced with fusion operator nodes to obtain a pending calculation graph; Deploy the model to be deployed on the target machine based on the pending computation graph and run it to obtain the pending computation results and pending performance; Determine whether a fusion condition is met to obtain a second determination result; the fusion condition is that the pending calculation result is the same as the original calculation result, and the pending performance is better than the original performance; If the second judgment result is yes, then the undetermined calculation graph is determined to be the optimized calculation graph at the nth iteration; If the second judgment result is no, determining the optimization calculation graph at the n-1th iteration as the optimization calculation graph at the nth iteration; Update the subgraph to be optimized, increase the value of the iteration number n by 1, and return to step "determine whether the non-redundant calculation condition is met and obtain the first judgment result" until all fusible operator subgraphs are traversed, and the optimized calculation graph at the nth iteration is obtained as the optimized calculation graph.

6. The multi-layer convolution operator fusion optimization method according to claim 1, characterized in that: Before optimizing the original computation graph based on the fusible operator subgraph to obtain the optimized computation graph, the method further includes: Determine the intermediate data of each fusible operator subgraph; Remove fusible operator subgraphs whose intermediate data size is larger than the target machine's shared memory capacity.

7. A multi-layer convolution operator fusion optimization device, characterized in that: The multi-layer convolution operator fusion optimization device applies the multi-layer convolution operator fusion optimization method according to any one of claims 1 to 6, and the multi-layer convolution operator fusion optimization device comprises: The original calculation graph determination module uses the host computer to analyze the calculation graph of the model to be deployed and determine the original calculation graph; A hardware performance model building module is used to build a hardware performance model of the target machine; A fusible operator subgraph determination module, used to determine multiple fusible operator subgraphs in the original computation graph based on the hardware performance model; An optimized computation graph determination module, used to optimize the original computation graph based on the fusible operator subgraph to obtain an optimized computation graph; The optimized code compilation module is used to obtain the parallel optimized code of the model to be deployed based on the optimized calculation graph and shard size; The optimization code running module is used to control the target machine to run the parallel optimization code to obtain the optimization performance of the model to be deployed.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-layer convolution operator fusion optimization method described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-layer convolution operator fusion optimization method described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-layer convolution operator fusion optimization method described in any one of claims 1 to 6 is implemented.

Images