Multi-layer convolution operator fusion optimization methods, devices, equipment, media and products

Abstract

This application discloses a method, apparatus, device, medium, and product for multi-layer convolution operator fusion optimization, relating to the field of compiler optimization technology. The method includes: determining the original computation graph of the model to be deployed; determining multiple fusionable operator subgraphs in the original computation graph based on a hardware performance model; optimizing the original computation graph based on the fusionable operator subgraphs to obtain an optimized computation graph; obtaining parallel optimization code for the model to be deployed based on the optimized computation graph and the slice size; and running the parallel optimization code on the target machine to obtain the optimized performance of the model to be deployed. This application improves the performance of the target machine when executing a neural network model by optimizing the original computation graph based on fusionable operator subgraphs.

Description

Technical Field

[0001] This application relates to the field of compiler optimization technology, and in particular to a method, apparatus, device, medium and product for multi-layer convolution operator fusion optimization. Background Technology

[0002] Convolutional Neural Networks (CNNs), as an important branch of deep learning, have achieved remarkable results in various fields such as image recognition, object detection, and speech recognition. In CNNs, convolutional layers and pooling layers are the basic units that constitute the network, extracting features from images through convolution and pooling operations. However, with the increasing complexity of the network structure, computational cost and memory consumption increase significantly, becoming one of the key factors restricting the improvement of CNN performance. In the Deep Convolutional Neural Network (DCNN) framework, the computation graph is a very important concept, graphically representing the layer structure, data flow, and dependencies between operations of the neural network. The computation graph enables the framework to optimize the execution process, including but not limited to memory usage and computational efficiency. During the inference phase, operator fusion is a common optimization technique that combines multiple operations into a single operation, thereby reducing memory accesses, data copying, and cache invalidation, and improving computational efficiency. For example, if the output of one neural network layer is directly used as the input of the next layer, the forward propagation operations of these two layers can be merged during inference, avoiding unnecessary intermediate data storage. Different deep learning frameworks, such as TensorFlow, PyTorch, and MXNet, each have their own mechanisms and strategies for operator fusion. These frameworks typically provide a series of tools and application programming interfaces (APIs) that allow developers to manually optimize the computation graph or automatically apply optimization strategies. Key technologies for operator fusion include graph optimization and parallel execution optimization. Graph optimization refers to analyzing and transforming the computation graph before execution to identify operators that can be fused. By reusing the memory of intermediate data, the overhead of memory allocation and deallocation is reduced, improving the execution efficiency of deep learning models. Existing frameworks such as TensorFlow and TensorAlgebraSuperOptimizer (TASO) provide high-level programming interfaces to achieve operator fusion and other optimizations by automatically analyzing and transforming the computation graph. XLA (Accelerated Linear Algebra) achieves operator fusion and memory optimization by converting the entire computation graph into a single, highly optimized machine code. It reduces the overhead of memory allocation and deallocation by reusing the memory of intermediate data. TASO focuses particularly on lateral operator fusion to improve program parallelism and thus increase model execution speed. Automatic optimization for parallel execution refers to the compiler identifying opportunities for parallel execution through code generation and allocating operators that can be processed in parallel to multiple processing units. Existing work mainly includes Halide and Tensor Virtual Machine (TVM).Halide optimizes execution efficiency by automatically merging and parallelizing operators through analysis of algorithm dependencies and data access patterns. TVM's AutoTuner searches for optimal parallel execution strategies, including parameters such as the number of threads and block size, to adapt to different hardware and models. It also optimizes operator implementations for specific hardware (such as GPUs and TPUs) to leverage the hardware's parallel processing capabilities.

[0003] Traditional computation graph execution may have problems with low hardware efficiency, especially in terms of large overhead in data access. Background technologies may include existing neural network computation graph construction and execution methods, as well as operator fusion technology in deep learning compilers. The invention patent with publication number CN110321999A, entitled Neural Network Computation Graph Optimization Method (hereinafter referred to as the invention [1]), improves efficiency by reducing the number of data interactions between the computing platform and external memory. The specific method is to select horizontally adjacent layers with the same feature map input based on optimization rules, or vertically adjacent layers whose feature map calculation results are the input of the next layer, and fuse at least two adjacent layers according to these layer objects. The invention patent with publication number CN112711422B, entitled A Fusion Operator Design Method and Heterogeneous Computing System for Heterogeneous Computing (hereinafter referred to as the invention [2]), provides an optimized compilation method. By fusing the preprocessing layer in the neural network, the storage and access requirements of intermediate results are reduced, the calculation order of operators in the fusion layer is determined, and the input and output call counts and splitting strategies of operators are obtained, thereby improving execution efficiency and realizing an automated compilation optimization process. The invention patent with publication number CN115374912A, entitled "A Fusion Operator Design Method and Heterogeneous Computing System for Heterogeneous Computing" (hereinafter referred to as "Invention [3]") redesigns and splits the normalization operator through specific steps to optimize the interaction of data in the heterogeneous computing system. The invention patent with publication number CN115659281A, entitled "An Adaptive Accelerated Operator Fusion Method and Device" (hereinafter referred to as "Invention [4]") adds key attributes to the nodes of the neural network computation graph and performs graph weighting, and uses deep reinforcement learning technology to adaptively determine the optimal fusion scheme of the operator. The invention patent with publication number CN110490309A, entitled "An Operator Fusion Method for Neural Networks and Related Products" (hereinafter referred to as "Invention [5]") achieves efficient fusion of operators by splitting and mapping the output data of the operator and fusion judgment based on the size of these data blocks. Although the above optimization methods alleviate the computation and memory pressure of CNN to a certain extent, the following problems still exist: 1. Insufficient optimization of computation graph: Most of the existing optimization methods are optimized for a single operator or local computation graph, and lack a global optimization strategy for the entire computation graph. This results in a large amount of redundant computation and memory access during the computation graph compilation process. Existing fusion methods are often targeted at specific operators and lack the ability to generalize and optimize for convolutional fusion computation graphs. They have limitations in operator fusion, etc.: Invention [1] can only select horizontal or vertical fusion layer objects in a limited way, cannot fuse multiple layers, and cannot combine hardware performance models for targeted performance optimization. The fusion capability of Invention [2] is limited and is limited to the optimization of preprocessing layers. 2. Poor flexibility: Different CNN models have different structures and computational requirements, while existing optimization methods often lack sufficient flexibility and are difficult to adapt to various complex network structures and hardware structures.The fusion method proposed in Invention [3] focuses on the fusion of convolution, normalization and activation layers, but lacks consideration for the fusion scheme of multiple general operators. Inventions [4] and [5] focus on the optimization of the computation graph itself, but do not combine the characteristics of the hardware for targeted optimization. Summary of the Invention

[0004] The purpose of this application is to provide a multi-layer convolution operator fusion optimization method, apparatus, device, medium and product, which can optimize the neural network model deployed on the target machine and improve the performance of the target machine when executing the neural network model.

[0005] To achieve the above objectives, this application provides the following solution:

[0006] Firstly, this application provides a multi-layer convolution operator fusion optimization method, including:

[0007] The host computer is used to analyze the computational graph of the model to be deployed to determine the original computational graph. The model to be deployed is a convolutional neural network model. The original computational graph includes multiple nodes and multiple edges. One edge is used to connect two nodes. The nodes are used to describe the operations or data in the model to be deployed. The edges are used to describe the data flow and dependencies between nodes. The dependencies are either a first dependency {1×1} or a second dependency {α×α}, where α is the size of the convolutional filter.

[0008] Construct a hardware performance model of the target machine;

[0009] Based on the hardware performance model, multiple fusionable operator subgraphs are determined in the original computation graph;

[0010] The original computation graph is optimized based on the fusionable operator subgraph to obtain an optimized computation graph;

[0011] Based on the optimized computation graph and partition size, the parallel optimized code for the model to be deployed is obtained;

[0012] The target machine runs the parallel optimization code to obtain the optimized performance of the model to be deployed; the performance metrics describing the optimized performance include execution time and resource consumption.

[0013] Optionally, before using the host computer to perform computational graph analysis on the model to be deployed and determine the original computational graph, the following steps are also included:

[0014] The model to be deployed is deployed to the target machine and run to obtain the original calculation results and original performance of the model to be deployed; the performance indicators describing the original performance include execution time and resource consumption.

[0015] Optionally, after running the optimization code on the target machine and obtaining the optimized performance of the model to be deployed, the process also includes:

[0016] The multi-layer convolution operator fusion optimization strategy is adjusted based on the optimized performance and the original performance; the multi-layer convolution operator fusion optimization strategy includes: slice size.

[0017] Optionally, the fusion category of the fusionable operator subgraph is vertical fusion, horizontal-vertical fusion, or vertical-horizontal fusion;

[0018] The vertical fusion is used to describe a single-input, single-output structure;

[0019] The horizontal-vertical fusion is used to describe a multi-input single-output structure;

[0020] The vertical-horizontal fusion is used to describe a single-input, multiple-output structure.

[0021] Optionally, the original computation graph is optimized based on the fusionable operator subgraph to obtain an optimized computation graph, including:

[0022] The original computation graph is determined to be the optimized computation graph at the 0th iteration.

[0023] Identify any fusionable operator subgraph as the subgraph to be optimized;

[0024] Let the number of iterations n = 1;

[0025] Determine whether the no-redundancy computation condition is met to obtain the first determination result; the no-redundancy computation condition is whether the dependency relationship between all output nodes and subsequent operators in the subgraph to be optimized is the first dependency relationship {1×1}; the output node is the node connected to the output quantity.

[0026] If the first judgment result is yes, then all operators in the subgraph to be optimized are directly merged into a single merged operator node;

[0027] If the first judgment result is negative, redundant calculations are performed on all operators in the subgraph to be optimized to obtain the fusion operator node;

[0028] In the optimization computation graph during the (n-1)th iteration, all output nodes in the subgraph to be optimized are replaced with fusion operator nodes to obtain the computation graph to be determined.

[0029] Based on the undetermined computation graph, the model to be deployed is deployed on the target machine and run, resulting in undetermined computation results and undetermined performance.

[0030] Determine whether the fusion condition is met to obtain a second determination result; the fusion condition is that the undetermined calculation result is the same as the original calculation result, and the undetermined performance is better than the original performance;

[0031] If the second judgment result is yes, then the undetermined computation graph is determined to be the optimized computation graph at the nth iteration;

[0032] If the second judgment result is negative, then the optimized computation graph at the (n-1)th iteration is determined to be the optimized computation graph at the nth iteration.

[0033] Update the subgraph to be optimized, increment the iteration number n by 1, and return to the step "determine whether the condition of no redundant calculation is met and obtain the first judgment result" until all subgraphs of fusionable operators are traversed, and the optimized computation graph at the nth iteration is obtained as the optimized computation graph.

[0034] Optionally, before optimizing the original computation graph based on the fusionable operator subgraph to obtain the optimized computation graph, the method further includes:

[0035] Determine the intermediate data for each fusionable operator subgraph;

[0036] Remove fusionable operator subgraphs whose intermediate data size is larger than the target machine's shared memory capacity.

[0037] Secondly, this application provides a multi-layer convolution operator fusion optimization apparatus. Optionally, the multi-layer convolution operator fusion optimization apparatus applies the multi-layer convolution operator fusion optimization method described above. The multi-layer convolution operator fusion optimization apparatus includes:

[0038] The original computational graph determination module uses a host computer to analyze the computational graph of the model to be deployed and determine the original computational graph.

[0039] The hardware performance model building module is used to build the hardware performance model of the target machine.

[0040] A fusionable operator subgraph determination module is used to determine multiple fusionable operator subgraphs in the original computation graph based on the hardware performance model.

[0041] The optimized computation graph determination module is used to optimize the original computation graph based on the fusionable operator subgraph to obtain an optimized computation graph.

[0042] An optimized code compilation module is used to obtain parallel optimized code for the model to be deployed based on the optimized computation graph and shard size;

[0043] An optimized code execution module is used to control the target machine to run the parallel optimized code and obtain the optimized performance of the model to be deployed.

[0044] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described multi-layer convolution operator fusion optimization method.

[0045] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described multi-layer convolution operator fusion optimization method.

[0046] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described multi-layer convolution operator fusion optimization method.

[0047] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0048] This application provides a method, apparatus, device, medium, and product for multi-layer convolutional operator fusion optimization. It determines the original computation graph of the model to be deployed, identifies multiple fusionable operator subgraphs within the original computation graph based on a hardware performance model, optimizes the original computation graph based on these fusionable operator subgraphs to obtain an optimized computation graph, and generates parallel optimization code for the model to be deployed based on the optimized computation graph and slice size. The target machine runs the parallel optimization code to obtain the optimized performance of the model to be deployed. It extends from the traditional single-input single-output mode to a general mode encompassing single-input multiple-output, multiple-input single-output, and multiple-input multiple-output, achieving broader and more flexible fusion optimization. Addressing the common dependencies in the computation graph, a solution based on redundant computation is proposed to further expand the fusion capability, ensuring the optimization method can adapt to various complex network structures and computational requirements. A roof model utilizing performance-memory bandwidth is proposed as a cost model, which comprehensively considers the relationship between computational performance and memory bandwidth, providing precise guidance for parallel code generation. Based on the cost model, fusion parallel code more adapted to specific hardware architectures is generated, reducing unnecessary memory access operations, increasing computational density, and thus fully utilizing hardware performance. By optimizing the code generation strategy, this application aims to reduce program startup overhead while improving the code's adaptability to hardware, ensuring good performance on different hardware platforms. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart of a multi-layer convolution operator fusion optimization method in one embodiment of this application.

[0051] Figure 2 This is a fusionable operator subgraph of the category of vertical fusion in one embodiment of this application.

[0052] Figure 3 This is a fusionable operator subgraph of the category of horizontal-vertical fusion in one embodiment of this application.

[0053] Figure 4 This is a fusionable operator subgraph of type vertical-horizontal fusion in one embodiment of this application.

[0054] Figure 5 This is a first schematic diagram of a non-redundant computational structure in one embodiment of this application;

[0055] Figure 6 This is a second schematic diagram of a non-redundant computational structure in one embodiment of this application;

[0056] Figure 7 This is a third schematic diagram of a non-redundant computational structure in one embodiment of this application;

[0057] Figure 8 This is a fourth schematic diagram of a non-redundant computational structure in one embodiment of this application;

[0058] Figure 9 This is a first schematic diagram of a redundant computing structure in one embodiment of this application;

[0059] Figure 10 This is a second schematic diagram of a redundant computational structure in one embodiment of this application;

[0060] Figure 11 This is a third schematic diagram of a redundant computational structure in one embodiment of this application;

[0061] Figure 12 This is a fourth schematic diagram of a redundant computational structure in one embodiment of this application;

[0062] Figure 13 This is a schematic diagram of the fusionable operator subgraph before fusion in one embodiment of this application;

[0063] Figure 14 This is a schematic diagram of the fused subgraphs of the fusion operator in one embodiment of this application;

[0064] Figure 15 This is a diagram of the overall architecture for multi-layer convolution operator fusion optimization in one embodiment of this application.

[0065] Figure 16 This is a schematic diagram of the structure of a computer device according to one embodiment of this application. Detailed Implementation

[0066] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] In existing computational graph optimization, fusion optimization methods are often limited to specific and finite subgraph types, such as single-input single-output patterns, which restricts the comprehensiveness and depth of optimization effects. With the increasing complexity of network structures, traditional fusion strategies can only optimize operators with many-to-one dependencies, making it difficult to meet the demands of efficient computation. Furthermore, there is insufficient support for fusion of complex subgraphs such as multiple-input multiple-output and multiple-input single-output subgraphs, resulting in a large amount of potential parallelism and computational resource utilization not being fully explored. In addition, existing deep learning frameworks have two major problems with code generation methods: first, selecting code through multiple versions of methods in the computation library, while adaptable to different hardware, increases program startup overhead and affects overall performance; second, using hardware-independent code generation methods, while simplifying the development process, results in poor adaptability to specific hardware, making it difficult to fully utilize hardware performance. Therefore, in an exemplary embodiment, such as... Figure 1 As shown, a multi-layer convolution operator fusion optimization method is provided, including:

[0069] Step 101: Analyze the computational graph of the model to be deployed using the host computer to determine the original computational graph. The model to be deployed is a convolutional neural network model. The original computational graph includes multiple nodes and multiple edges. An edge connects two nodes. Nodes describe the operations or data in the model to be deployed. Edges describe the data flow and dependencies between nodes. Dependencies are either a first dependency {1×1} or a second dependency {α×α}, where α is the size of the convolutional filter.

[0070] Step 102: Construct a hardware performance model of the target machine.

[0071] Specifically, it utilizes computer-aided graph analysis for optimal decision-making.

[0072] The executing entity is the optimization decision computer (host computer), and the computational graph analysis steps include:

[0073] Step 1.1: Load and parse the model computation graph. Read the structure of the model computation graph from the input file, including nodes (representing operations or data) and edges (representing data flow and dependencies).

[0074] Step 1.2: Analyze the computation and memory access characteristics of nodes. Identify nodes in the computation graph that can be optimized, especially convolutional layers, additive layers, and other nodes with relatively low computation and memory access.

[0075] Step 1.3: Construct a hardware performance model of the target machine. Based on the hardware's parallel computing capabilities and memory access bandwidth, assess the types of performance bottlenecks that the hardware may present for the application, providing a foundation for subsequent operator fusion.

[0076] Step 1.4: Output Analysis Results. Provide the analysis results of the computational bottlenecks of the computing nodes for the specified hardware, summarize the structural characteristics of the computation graph and node dependencies, and provide reference information for subsequent operator fusion steps.

[0077] Step 103: Based on the hardware performance model, identify multiple fusionable operator subgraphs in the original computation graph. The fusion category of the fusionable operator subgraphs is vertical fusion, horizontal-vertical fusion, or vertical-horizontal fusion.

[0078] like Figure 2 Vertical fusion is used to describe a single-input, single-output structure.

[0079] like Figure 3 Horizontal-vertical fusion is used to describe a multi-input single-output structure.

[0080] like Figure 4 Vertical-horizontal fusion is used to describe a single-input, multiple-output structure.

[0081] Step 104: Optimize the original computation graph based on the fusionable operator subgraph to obtain the optimized computation graph.

[0082] Specifically, given a candidate set of computable subgraphs to be merged, the computation dependency boundary guarantees:

[0083] Execution entity: Optimization decision-making computer (host computer); fusion steps include:

[0084] Step 2.1: Identify fusionable operator subgraphs. Based on the analysis results of Step 1.3, find fusionable operator pairs and classify the operator subgraphs into three types for fusion mode candidates. The three types are as follows: Figures 2 to 4 As shown, these represent vertical blending, horizontal-vertical blending, and vertical-horizontal blending, respectively.

[0085] Step 2.2: Perform the fusion operation. There are two cases when performing the fusion operation. Case 1 is the case with no redundant computation, where subsequent operator dependencies are element-level dependencies, i.e., the dependency shape is {1×1}, such as... Figures 5 to 8 As shown, fusion can be performed directly; case two involves redundant computation, where the subsequent operator dependency relationship has the shape {α×α}, as shown... Figures 9 to 12 This requires performing redundant computations on a single stream processor based on the specific shape of the operator and the dependencies of the output layer operator fragments in parallel computing, as well as the dependencies of its intermediate layers. This eliminates the need for expensive synchronization operations, thus ensuring the correctness of the computation results.

[0086] Step 2.3: Evaluate the benefits of fusion. Calculate the potential benefits of fusion, such as reduced computational load and optimized memory access, to ensure that fusion is potentially beneficial in the performance model. Especially for operators with redundant computations, the cost of these redundancies needs to be reflected in the performance model.

[0087] Step 2.4: Verify the correctness of the subgraph fusion and replace it. Ensure that the result of the fused computation graph is consistent with the original computation. Figure 1 To merge the selected operator pairs, replace the original subgraphs with new fusion operator nodes, and update the computation graph. The fusion operator subgraphs before and after fusion are as follows: Figure 13 and Figure 14 As shown.

[0088] Step 105: Based on the optimized computation graph and shard size, obtain the parallel optimization code for the model to be deployed.

[0089] Specifically, hardware-customized compilation optimization is implemented, and template operators are automatically generated.

[0090] The execution entity is the collaborative work of the optimization decision-making computer (host computer) and the target machine (lower computer).

[0091] Parallel optimization code generation steps include:

[0092] Step 3.1: Filter candidate subgraphs for optimization using a performance model to reduce performance overhead. Based on the target machine characteristics, including processor architecture, memory bandwidth, cache size, shared memory size, and other hardware information, as well as the instruction set for optimization. Considering the limitations of the target machine's shared memory capacity, subgraphs that cannot be merged due to excessively large intermediate data sizes are eliminated. Based on the merged computational performance model and hardware performance model information, operators with performance optimization potential are selected for optimization.

[0093] Step 3.2: Generate parallel optimization code. Based on the fusion strategy, formulate a targeted parallel optimization strategy for the fusion code. The input matrix is ​​sharded in its length and width dimensions for inter-GPU block parallelism, and in its shard length, shard width, and channel dimensions for intra-GPU block thread parallelism.

[0094] Step 3.3: Customize the compilation strategy. Based on the characteristics of the target machine, develop targeted compilation optimization strategies, such as instruction selection, loop unrolling, and memory access optimization. Convert the optimized computation graph into code optimized for the target machine, including code utilizing specific assembly instructions or specific hardware accelerators (e.g., tensor matrix multiplication instructions).

[0095] Step 3.4: Optimization Correctness Verification. Deploy the optimized code to the target machine and verify whether the calculation results are consistent with the original calculation results, preparing for performance verification on the target machine.

[0096] Step 106: Run the parallel optimization code on the target machine to obtain the optimized performance of the model to be deployed. Performance metrics describing the optimized performance include execution time and resource consumption.

[0097] Before step 101, the following is also included:

[0098] Step 107: Deploy the model to be deployed on the target machine and run it to obtain the raw computation results and raw performance of the model. Performance metrics describing the raw performance include execution time and resource consumption.

[0099] Following step 106, the following is also included:

[0100] Step 108: Adjust the multi-layer convolution operator fusion optimization strategy based on the optimized performance and the original performance. The multi-layer convolution operator fusion optimization strategy includes: slice size.

[0101] Specific target machine performance verification and parallel parameter iterative optimization.

[0102] Execution subject: target machine (lower-level machine).

[0103] The target machine performance verification steps include:

[0104] Step 4.1: Configure the test environment. Ensure the target machine is in a suitable testing state, including closing other programs, setting a stable power supply, and fixing the hardware clock cycle frequency.

[0105] Step 4.2: Execute the optimized program. Run the optimized program on the target machine and record performance metrics such as execution time and resource consumption. Comparative analysis. Compare the optimized performance metrics with the original performance to evaluate the optimization effect.

[0106] Step 4.3: The proposed fusion method for parallel optimization code generation steps involves several key performance factors, such as shard parallelism, intermediate data size, and redundant computation quantity, all of which are related to shard size. Therefore, iterative optimization of shard size is adopted.

[0107] Step 4.4: Feedback and Optimization Iteration. Based on the verification results, provide feedback and adjust the optimization strategy.

[0108] Before step 104, the following is also included:

[0109] Step 109: Determine the intermediate data for each fusionable operator subgraph.

[0110] Step 1010: Delete fusionable operator subgraphs whose intermediate data size is larger than the target machine's shared memory capacity.

[0111] Step 104 includes:

[0112] Step 104-1: Determine that the original computation graph is the optimized computation graph at the 0th iteration.

[0113] Step 104-2: Determine any fusionable operator subgraph as the subgraph to be optimized.

[0114] Step 104-3: Let the number of iterations n = 1.

[0115] Step 104-4: Determine whether the redundancy-free computation condition is met, and obtain the first determination result. The redundancy-free computation condition is whether the dependencies between all output nodes in the subgraph to be optimized and subsequent operators are all first dependencies {1×1}. Output nodes are nodes connected to the output quantity.

[0116] Step 104-5: If the first judgment result is yes, then directly merge all operators in the subgraph to be optimized into a single merged operator node.

[0117] Step 104-6: If the first judgment result is negative, then perform redundant calculations on all operators in the subgraph to be optimized to obtain the fusion operator node.

[0118] Step 104-7: In the optimized computation graph during the (n-1)th iteration, replace all output nodes in the subgraph to be optimized with fusion operator nodes to obtain the computation graph to be determined.

[0119] Step 104-8: Deploy the model to be deployed on the target machine and run it based on the undetermined computation graph, and obtain the undetermined computation results and undetermined performance.

[0120] Step 104-9: Determine whether the fusion condition is met to obtain the second judgment result. The fusion condition is that the undetermined calculation result is the same as the original calculation result, and the undetermined performance is better than the original performance.

[0121] Step 104-10: If the second judgment result is yes, then the undetermined computation graph is determined to be the optimized computation graph for the nth iteration.

[0122] Step 104-11: If the second judgment result is negative, then determine the optimized computation graph at the (n-1)th iteration as the optimized computation graph at the nth iteration.

[0123] Step 104-12: Update the subgraph to be optimized, increment the iteration number n by 1, and return to step 104-4 until all fusionable operator subgraphs are traversed, and the optimized computation graph at the nth iteration is obtained as the optimized computation graph.

[0124] like Figure 15 By fusing operators such as convolution and addition, and combining instruction scheduling and pipelining techniques, computational efficiency is improved. The optimization scheme related to the target machine (lower-level machine) is implemented on the optimization decision computer (host computer), undergoing key steps such as computation graph analysis, operator fusion, and hardware-customized compilation optimization, and finally, performance verification is performed on the target machine.

[0125] Load the computation graph, identify possible fusion subgraphs, provide candidate fusion subgraphs, and provide the hardware performance model function:

[0126]

[0127] Where Op represents the operator, p(·) represents the theoretical maximum performance of executing the operator on the target machine, c(·) is the computation-to-memory ratio of the operator, and p max For the peak performance of the target machine, c Peak This represents the computation-to-memory ratio at which peak performance is achieved on the target machine. The model is largely consistent with the classic roof performance model. Based on the above calculations, the theoretical peak computational performance of the operator on the hardware can be obtained. When the operator's theoretical computation-to-memory ratio is higher than that of the target machine, the bottleneck is memory access, and the peak performance of the machine hardware has not been reached; its peak performance depends on the operator's own computation-to-memory ratio. Conversely, when the theoretical computation-to-memory ratio is lower, the bottleneck is computation, and the peak performance of the machine hardware can be reached; theoretically, its peak performance should be equal to the peak performance of the hardware.

[0128] according to Figure 15 The fusion rules shown traverse and match the computation graph to generate correct high-performance computing code. For Figure 11 As shown, redundant computation of CONV1 is required at the block parallel level to ensure the correct computation of CONV2 and CONV3. Parallel code with redundant computation is provided.

[0129] A fusion subgraph performance model function is given, which prunes operators that exceed the shared memory capacity and have no potential performance gain.

[0130] For a general operator, its performance model function can be evaluated using the amount of computation and global memory accesses:

[0131]

[0132] Where c(·) is the computation-to-memory ratio of the operator, comp(·) represents the computation of the operator, and gm ld (·) represents the cost of accessing global memory, which can be optimized using read-only memory, gm st (·) represents optimization for global memory writes. Using this performance model function, the computation-to-memory ratio of general operators can be calculated, which can indirectly reflect the relative performance of executing the function on hardware devices.

[0133] for Figure 11 The example shown has a model function with redundant calculations:

[0134]

[0135] Here, `reComp(·)` represents the redundant computation introduced for fusion. `in` is the input to the operator subgraph, `out` is the output of the operator subgraph, and `flt` represents the filter. Based on the operator's computation and the formula for the global memory access ratio, the sharding parameters of the parallel operator are flexibly adjusted. Specifically: if the calculated memory access ratio increases significantly, it indicates that the shard size is too small, and the shard size should be appropriately increased while adjusting the parallelism; conversely, the opposite is also true.

[0136] Parallel optimization. Optimize key parallel parameters. The width of the partition is defined as T. x The length of a fragment is defined as T. y By optimizing the distribution of length and width separately, a more comprehensive exploration of the search space can be achieved. Pruning lengths and widths with non-approximate factors and odd numbers improves the efficiency of the search space exploration.

[0137] This embodiment expands from the traditional single-input single-output mode to a general mode encompassing single-input multiple-output, multiple-input single-output, and multiple-input multiple-output, achieving broader and more flexible fusion optimization. For computation graphs employing general dependencies, a solution based on redundant computation is proposed to further extend fusion capabilities, ensuring the optimization method can adapt to various complex network structures and computational requirements. A roof model utilizing performance-memory bandwidth is proposed as a cost model, which comprehensively considers the relationship between computational performance and memory bandwidth, providing precise guidance for parallel code generation. Based on the cost model, fused parallel code more adapted to specific hardware architectures is generated, reducing unnecessary memory access operations, increasing computational density, and thus fully utilizing hardware performance. By optimizing the code generation strategy, this application aims to reduce program startup overhead while improving code adaptability to hardware, ensuring good performance across different hardware platforms. It can fuse subgraph types that were previously unfusionable, considering the performance optimization benefits and overhead of redundant computation, and performing hardware-adaptive fusion based on hardware performance models and operator fusion performance models.

[0138] This embodiment can integrate subgraph types that could not be integrated in previous technical solutions. It considers the performance optimization benefits and overhead brought by redundant calculations, and performs hardware-adaptive integration based on hardware performance models and operator fusion performance models. By integrating general convolution operators and combining code generation and compilation optimization techniques, it customizes a new method for hardware-adaptive code optimization, thereby improving computational efficiency and reducing computation time.

[0139] In one exemplary embodiment, a multi-layer convolution operator fusion optimization apparatus is provided, characterized in that the multi-layer convolution operator fusion optimization apparatus applies the multi-layer convolution operator fusion optimization method, and the multi-layer convolution operator fusion optimization apparatus includes:

[0140] The original computational graph determination module uses a host computer to analyze the computational graph of the model to be deployed and determine the original computational graph.

[0141] The hardware performance model building module is used to build the hardware performance model of the target machine.

[0142] The fusionable operator subgraph determination module is used to determine multiple fusionable operator subgraphs in the original computation graph based on the hardware performance model.

[0143] The optimized computation graph determination module is used to optimize the original computation graph based on the fusionable operator subgraph to obtain an optimized computation graph.

[0144] The code compilation module is optimized to obtain parallel optimized code for the model to be deployed based on the optimized computation graph and shard size.

[0145] An optimized code execution module is used to control the target machine to run the parallel optimized code and obtain the optimized performance of the model to be deployed.

[0146] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 16 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-layer convolution operator fusion optimization method.

[0147] Those skilled in the art will understand that Figure 16 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0148] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0149] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0150] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0151] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0152] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0153] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multi-layer convolution operator fusion optimization method, characterized in that, include: The host computer is used to analyze the computational graph of the model to be deployed to determine the original computational graph. The model to be deployed is a convolutional neural network model. The original computational graph includes multiple nodes and multiple edges. One edge is used to connect two nodes. The nodes are used to describe the operations or data in the model to be deployed. The edges are used to describe the data flow and dependencies between nodes. The dependencies are either a first dependency {1×1} or a second dependency {α×α}, where α is the size of the convolutional filter. Construct a hardware performance model of the target machine; Based on the hardware performance model, multiple fusionable operator subgraphs are determined in the original computation graph; The original computation graph is optimized based on the fusionable operator subgraph to obtain an optimized computation graph; Based on the optimized computation graph and partition size, the parallel optimized code for the model to be deployed is obtained; The target machine runs the parallel optimization code to obtain the optimized performance of the model to be deployed; The performance metrics describing the optimized performance include execution time and resource consumption; The original computation graph is optimized based on the fusionable operator subgraph to obtain an optimized computation graph, including: The original computation graph is determined to be the optimized computation graph at the 0th iteration. Identify any fusionable operator subgraph as the subgraph to be optimized; Let the number of iterations n = 1; Determine whether the no-redundancy computation condition is met to obtain the first determination result; the no-redundancy computation condition is whether the dependency relationship between all output nodes and subsequent operators in the subgraph to be optimized is the first dependency relationship {1×1}; the output node is the node connected to the output quantity. If the first judgment result is yes, then all operators in the subgraph to be optimized are directly merged into a single merged operator node; If the first judgment result is negative, redundant calculations are performed on all operators in the subgraph to be optimized to obtain the fusion operator node; In the optimization computation graph during the (n-1)th iteration, all output nodes in the subgraph to be optimized are replaced with fusion operator nodes to obtain the computation graph to be determined. Based on the undetermined computation graph, the model to be deployed is deployed on the target machine and run, resulting in undetermined computation results and undetermined performance. Determine whether the fusion condition is met to obtain a second determination result; the fusion condition is that the undetermined calculation result is the same as the original calculation result, and the undetermined performance is better than the original performance; If the second judgment result is yes, then the undetermined computation graph is determined to be the optimized computation graph at the nth iteration; If the second judgment result is negative, then the optimized computation graph at the (n-1)th iteration is determined to be the optimized computation graph at the nth iteration. Update the subgraph to be optimized, increment the iteration number n by 1, and return to the step "determine whether the condition of no redundant calculation is met and obtain the first judgment result" until all subgraphs of fusionable operators are traversed, and the optimized computation graph at the nth iteration is obtained as the optimized computation graph.

2. The multi-layer convolution operator fusion optimization method according to claim 1, characterized in that, Before using the host computer to perform computational graph analysis on the model to be deployed and determine the original computational graph, the following steps are also included: The model to be deployed is deployed to the target machine and run to obtain the original calculation results and original performance of the model to be deployed; the performance indicators describing the original performance include execution time and resource consumption.

3. The multi-layer convolution operator fusion optimization method according to claim 2, characterized in that, After running the optimization code on the target machine and obtaining the optimized performance of the model to be deployed, the process also includes: The multi-layer convolution operator fusion optimization strategy is adjusted based on the optimized performance and the original performance; the multi-layer convolution operator fusion optimization strategy includes: slice size.

4. The multi-layer convolution operator fusion optimization method according to claim 1, characterized in that, The fusion category of the fusionable operator subgraph is vertical fusion, horizontal-vertical fusion, or vertical-horizontal fusion; The vertical fusion is used to describe a single-input, single-output structure; The horizontal-vertical fusion is used to describe a multi-input single-output structure; The vertical-horizontal fusion is used to describe a single-input, multiple-output structure.

5. The multi-layer convolution operator fusion optimization method according to claim 1, characterized in that, Before optimizing the original computation graph based on the fusionable operator subgraph to obtain the optimized computation graph, the process further includes: Determine the intermediate data for each fusionable operator subgraph; Remove fusionable operator subgraphs whose intermediate data size is larger than the target machine's shared memory capacity.

6. A multi-layer convolution operator fusion optimization device, characterized in that, The multi-layer convolution operator fusion optimization device applies the multi-layer convolution operator fusion optimization method as described in any one of claims 1-5, and the multi-layer convolution operator fusion optimization device includes: The original computational graph determination module uses a host computer to analyze the computational graph of the model to be deployed and determine the original computational graph. The hardware performance model building module is used to build the hardware performance model of the target machine. A fusionable operator subgraph determination module is used to determine multiple fusionable operator subgraphs in the original computation graph based on the hardware performance model. The optimized computation graph determination module is used to optimize the original computation graph based on the fusionable operator subgraph to obtain an optimized computation graph. An optimized code compilation module is used to obtain parallel optimized code for the model to be deployed based on the optimized computation graph and shard size; An optimized code execution module is used to control the target machine to run the parallel optimized code and obtain the optimized performance of the model to be deployed.

7. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the multilayer convolution operator fusion optimization method according to any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the multi-layer convolution operator fusion optimization method as described in any one of claims 1-5.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the multi-layer convolution operator fusion optimization method as described in any one of claims 1-5.

Images