A hardware feature related deep model computation graph automatic optimization method

Abstract

The application discloses a hardware characteristic related deep model calculation graph automatic optimization method, comprising the following steps: converting a deep learning model into an intermediate representation according to a preset rule; performing first operator fusion on the intermediate representation according to a static operator fusion rule to form a composite subgraph; performing parallelization processing on the operators in the composite subgraph according to hardware resources, performing second operator fusion, and forming an aggregated subgraph; performing cross-boundary optimization processing on the aggregated subgraph; forming a list of original operators and fused operators according to the records of the first operator fusion and the second operator fusion; constructing a cost model on a target hardware platform based on the list of original operators and fused operators, and evaluating the fusion effect; and according to the evaluation result of the fusion effect, the fused operators are split, and the generated subgraph is used as the cut-in of the operator layer to generate code.

Description

Technical Field

[0001] This invention belongs to the field of computer science, specifically the field of deep learning model compilation acceleration technology, and in particular, an automatic optimization method for deep model computation graphs related to hardware characteristics. Background Technology

[0002] Deep learning has developed rapidly in recent years, leading to the emergence of numerous deep learning algorithm development frameworks. Due to its wide range of applications and enormous computational demands, deep learning algorithms typically need to run on various general-purpose and specialized hardware, such as different types of CPUs, GPUs, TPUs, and NPUs. This has resulted in an explosion of framework and hardware combinations. Simultaneously, many algorithm networks have emerged, such as YOLO, BERT, and GPT. These networks are composed of operators of different types, shapes, and connection relationships, ultimately running on different types and models of hardware. This makes the development and implementation of optimal operators for specific scenarios very costly.

[0003] A computation graph is a directed graph consisting of: a set of nodes, each representing an operation or computation; and a set of directed edges, each labeled with the relationships between nodes (data transfer and control dependencies). Optimizing the computation graph can improve the speed of neural network training and inference. Currently, mainstream frameworks such as TensorFlow, PyTorch, and TVM employ multiple computation graph optimization techniques simultaneously to accelerate computation. TensorFlow provides a graph optimizer API that users can directly call; TVM uses operator fusion and other methods for computational optimization. However, the mainstream graph optimization processes in current deep learning compilers typically require significant manual effort, and the optimization strategies are hardware-independent, failing to maximize the utilization of hardware resources.

[0004] Therefore, when optimizing mainstream graphs in deep learning compilers, reducing manual costs and improving hardware resource utilization have become key research issues. Summary of the Invention

[0005] In view of the above problems, the present invention provides an automatic optimization method for deep model computation graphs related to hardware characteristics, which at least solves some of the above technical problems. This method can reduce manual costs and improve hardware resource utilization when optimizing mainstream graphs in deep learning compilers.

[0006] This invention provides an automatic optimization method for hardware-feature-dependent deep model computation graphs, including:

[0007] S1, S1, Obtain the deep learning model; convert the deep learning model into an intermediate representation according to preset rules; the intermediate representation is used to describe the structure of the computation graph;

[0008] S2. Perform the first operator fusion on the intermediate representation according to the static operator fusion rules to form a composite subgraph;

[0009] S3. Parallelize the operators in the composite subgraph according to hardware resources, perform a second operator fusion, and form an aggregated subgraph.

[0010] S4. Perform cross-boundary optimization processing on the aggregated subgraph;

[0011] S5. Based on the records of the first operator fusion and the second operator fusion, form a list of original operators and a list of fused operators; construct a cost model on the target hardware platform based on the list of original operators and the list of fused operators, and evaluate the fusion effect;

[0012] S6. Based on the evaluation results of the fusion effect, the fusion operator is segmented, and the generated subgraph is used as the input generation code of the operator layer.

[0013] Furthermore, the intermediate representation is used to define the computation type, number of inputs and number of outputs of each operator in the computation graph, as well as the relevant parameters of the computation; the intermediate representation is also used to record the number of constituent operators of the fusion operator, as well as the relevant information of the constituent operators.

[0014] Further, S2 specifically includes: based on the intermediate representation after conversion, traversing the computation graph in reverse order starting from the output, and fusing operators in the computation graph that satisfy the static operator fusion rules; iterating this process multiple times until there are no operators in the computation graph that can be fused; and re-integrating the fused computation graph to form a composite subgraph.

[0015] Furthermore, the static operator fusion rules include:

[0016] Data dependencies exist between the merged operators;

[0017] Merging should not create directed loops between subgraphs;

[0018] After fusion, the on-chip cache is sufficient, and there is no need to move data from main memory during the computation process.

[0019] Furthermore, in S3, the parallelization process includes: abstracting the underlying hardware resources into virtual computing units, scheduling the computation to be performed onto the computing units, and enabling the operator to complete the parallelization process.

[0020] Furthermore, the parallelization process includes intra-operator parallelism and inter-operator parallelism;

[0021] The internal parallelism of the operator includes: sorting out the parallel logic between operators when merging operators, and parallelizing the computations that have no data dependency within the merged operator when hardware resources allow;

[0022] The parallelism between operators includes: continuing to perform parallel operations on the fused operators after operator fusion; and simultaneously readjusting the internal parallelism of the operators while the fused operators are in parallel.

[0023] Furthermore, if the extra resources can achieve inter-operator parallelism while reducing the internal parallelism of operators, then the inter-operator parallelism strategy should be given priority.

[0024] Furthermore, S4 specifically includes:

[0025] To traverse the computation graph using topological sorting, first select a node with an in-degree of 0 to execute, then delete that node and the edges connected to it, and then find the next node with an in-degree of 0 to execute.

[0026] Cross-boundary optimization is performed during the traversal of the computation graph; the cross-boundary optimization includes: constant folding, inlining optimization, and common sub-representation extraction.

[0027] Furthermore, in S5, the cost model is trained in the following manner:

[0028] Traverse the computation graph, extract the features of all operators in the computation graph at this time, and construct the number of operators and operator types of the fusion operator based on the features;

[0029] Select a preset number of operators and deploy them on the hardware, then measure the hardware's time and energy consumption as marker data;

[0030] Based on the semi-supervised algorithm, a model is constructed using labeled data and predictions are made using unlabeled data;

[0031] Add high-confidence data to the labeled dataset, use it as labeled data, and retrain the model;

[0032] Training ends when the number of data points in the labeled dataset exceeds a preset proportion, resulting in a well-trained cost model.

[0033] Furthermore, in step S6, if the fusion effect does not meet the preset conditions, the fusion operator is segmented.

[0034] Compared with existing technologies, the hardware-characteristic-related automatic optimization method for deep model computation graphs described in this invention has the following beneficial effects:

[0035] The intermediate representation used in this invention is for better compatibility with existing deep learning compilers, enabling this method to be more widely applied in the field of deep learning compilation acceleration. Expanding the composite operators in the original model into fine-grained operators has two advantages: first, it allows for finer-grained optimization of the internals of the original operators; second, it unifies the operator types, reduces the number of operators provided by the framework, and facilitates subsequent fusion and cost model construction.

[0036] This invention accelerates the inference process by reducing main memory reads and maximizing hardware resource utilization through operator fusion and parallel processing, respectively. Arranging operator fusion before parallel processing reduces the amount of data processed by parallelization and enables coordinated handling of parallelism between and within operators.

[0037] This invention uses a cost model to evaluate the energy and time consumption of operators and performs operator fusion partitioning, enabling the graph optimization scheme to dynamically adapt to different hardware backends. Through fusion and parallelization, the cost model's evaluation of operator performance more closely approximates its actual performance on the hardware, resulting in more accurate predictions.

[0038] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0039] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0040] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0041] Figure 1 This is a schematic diagram of the overall process of the automatic optimization method for deep model computation graph related to hardware characteristics provided in the embodiments of the present invention.

[0042] Figure 2 This is a semantic tree diagram provided for an embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of the static operator fusion rule provided in an embodiment of the present invention.

[0044] Figure 4 This is a schematic diagram of operator parallelism provided in an embodiment of the present invention.

[0045] Figure 5 A schematic diagram of the cost model construction process provided in an embodiment of the present invention.

[0046] Figure 6 This is a schematic diagram of the operator segmentation process provided in an embodiment of the present invention.

[0047] Figure 7 This is a schematic diagram of a CNN structure provided in an embodiment of the present invention.

[0048] Figure 8 This is a schematic diagram of a neural network structure provided in an embodiment of the present invention.

[0049] Figure 9 This is a schematic diagram illustrating operator fusion for complex models provided in an embodiment of the present invention.

[0050] Figure 10 This is a schematic diagram illustrating parallel processing of a computation graph according to an embodiment of the present invention.

[0051] Figure 11 This is a schematic diagram illustrating operator segmentation of a CNN structure according to an embodiment of the present invention. Detailed Implementation

[0052] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0053] See Figure 1 As shown, this embodiment of the invention provides an automatic optimization method for deep model computation graphs related to hardware characteristics, specifically including the following steps:

[0054] S1. Obtain a deep learning model; convert the deep learning model into an intermediate representation according to preset rules; the intermediate representation is used to describe the structure of the computation graph;

[0055] S2. Perform the first operator fusion on the intermediate representation according to the static operator fusion rules to form a composite subgraph;

[0056] S3. Parallelize the operators in the composite subgraph according to hardware resources, perform a second operator fusion, and form an aggregated subgraph.

[0057] S4. Perform cross-boundary optimization processing on the aggregated subgraph;

[0058] S5. Based on the records of the first operator fusion and the second operator fusion, form a list of original operators and fused operators, build a cost model on the target hardware platform, and evaluate the fusion effect;

[0059] S6. Based on the evaluation results of the fusion effect, the fusion operator is segmented, and the generated subgraph is used as the input generation code of the operator layer.

[0060] The following sections will provide a detailed explanation of each of the above steps.

[0061] (1) Intermediate Reference

[0062] It accepts mainstream models such as ONNX, PyTorch, and Tensorflow as input and transforms them into custom intermediate representations to describe the structure of the computation graph; for example, it transforms them into TVM Relay IR for further processing. Relay is a versatile programming language used for intermediate representations in machine learning systems; Relay IR is a purely expression-oriented language.

[0063] The intermediate representation defines the computation type, number of inputs, and number of outputs for each operator in the computation graph, as well as the necessary parameters associated with that computation. For fusion operators, the intermediate representation also records how many operators they consist of and related information about those operators. The resulting semantic tree is as follows: Figure 2 As shown.

[0064] (2) Static fusion of operators

[0065] Overall process: Based on the transformed IR, traverse the computation graph in reverse order starting from the output, and merge operators that satisfy the static operator fusion rules; iterate this process multiple times until there are no operators in the computation graph that can be merged; re-integrate the merged computation graph to form a composite subgraph.

[0066] Static operator fusion rule reference Figure 3 As shown, the purpose of this operator fusion is to reduce the number of times data is moved from main memory. To ensure that potential fusion operators are not missed during subsequent cost model evaluation, a greedy strategy is used to fuse as many operators with data dependencies as possible. However, it is essential to ensure that the computational load of the fused operators does not exceed the hardware's capacity; therefore, some fusions generally considered to have little effect on performance improvement are discarded. The basic fusion criteria should satisfy: 1) Data dependencies exist between fused operators; 2) Fusion should not create directed cycles between subgraphs; 3) Hardware resources after fusion allow computation to be completed without moving data from main memory; that is, the on-chip cache is sufficient after fusion, and data does not need to be moved from main memory during computation.

[0067] The commonly used operators in deep learning include the following five categories:

[0068] 1) Algebraic operators. Operations that compute corresponding elements between two tensors, including ReLU, ADD, and MUL. 2) Broadcast operators. Operators that process tensors of different shapes during computation through copying operations. If the tensor shape needs to be adjusted during the computation of an algebraic operator, it is also included in the broadcast operator category, including BN. 3) Reduction operators. Operations that reduce the number of elements in a tensor along a specified axis, including SUM and ARGMAX. 4) Complex operators. Common and computationally intensive operators in neural networks, including CONV5x5 and FC. 5) Non-fusionable operators. Operators that do not yield any benefit after fusion with other operators, including constants. 6) Two-phase operators. Operators that exhibit different behaviors after fusion on different hardware, and it is impossible to determine whether there is any benefit after fusion, including MAXPOOL and CONV1x1. Based on the fusion criteria and supported by a large number of historical experiments, the following static fusion rules are derived. (The same type of operator may be classified into different types of operators depending on the specific parameters. For example, CONV1*1 is a two-phase operator, while CONV5*5 is a complex operator. The specific classification depends on the actual situation.)

[0069] Static operator fusion rules refer to fusions that will improve performance on most hardware platforms. For example, the fusion of CONV+RELU+BN conforms to rule one of stable fusion. Prohibited fusion rules refer to fusions that will degrade performance or have no impact on overall performance in most cases. Except for fusion types prohibited by the prohibited fusion rules, operators can be fused provided they meet the basic fusion rules. If an operator type conforms to the stable fusion rules, it will not be split subsequently; otherwise, during the operator splitting stage, a cost model will be used to evaluate whether the fusion is effective. If performance is reduced after fusion, the fused operator will be split again. Operator fusion completed in this stage cannot guarantee a completely positive impact on inference, as different operators exhibit varying performance on different hardware. Therefore, hardware-aware operator splitting is performed later to improve backend adaptability.

[0070] (3) Parallelization processing

[0071] The underlying hardware resources are abstracted into virtual computing units, and the computations to be performed are scheduled to these computing units, enabling operators to perform parallel processing. By fusing the parallelism within an operator and the parallelism between the fused operators, the hardware resources are fully utilized. The parallel structure is shown in Figure 4.

[0072] Parallelization includes intra-operator parallelism and inter-operator parallelism; where:

[0073] Operator-internal parallelism includes: streamlining the parallel logic between operators during operator fusion, and parallelizing computations within the merged operator that have no data dependencies, provided that hardware resources allow; and using loop optimization techniques to ensure that parallelism is not limited to the merged operators, but also accelerates computations within each operator.

[0074] Inter-operator parallelism includes: continuing to perform parallel operations on the fused operators after operator fusion; simultaneously readjusting the internal parallelism of operators while fused operators are in parallel; if the extra resources can achieve inter-operator parallelism while reducing the internal parallelism of operators, then the inter-operator parallelism strategy should be given priority.

[0075] (4) Cross-boundary optimization

[0076] On the aggregated subgraph, cross-boundary optimization operations such as constant propagation and common subexpression extraction are performed. Specifically, a topological sort is used to traverse the computation graph. First, a node with an in-degree of 0 is selected for execution, then that node and its connected edges are deleted, and then the next node with an in-degree of 0 is found for execution. Cross-boundary optimization processing is performed during the traversal of the computation graph. This cross-boundary optimization processing includes constant folding, inlining optimization, and common subexpression extraction. Among them:

[0077] Constant folding: If all inputs to a computation are constants, the result is computed directly and replaced with a node in the graph.

[0078] Inline optimization: Expand function nodes with small code size and simple calculations, and delete useless nodes.

[0079] Common subexpression elimination: In a program, if several expressions have the same type, parameters, and input, they are called common subexpressions. For common subexpressions, only one expression needs to be evaluated; the values ​​of the others can be obtained through assignment. During the traversal, the program records the calculations that have already been processed; if the calculation to be processed is the same, it is directly replaced.

[0080] (5) Cost Model

[0081] The main function of the cost model in this embodiment of the invention is to evaluate the running time and power consumption of each operator on the target hardware platform without actual deployment, providing a benchmark for dynamic fusion and segmentation of the computation graph. To achieve automatic optimization aware of the target hardware platform, a new and most suitable cost model needs to be constructed for each different hardware. Cost models using deep learning methods require a large amount of data during construction, and acquiring data in actual use consumes considerable time. To accelerate the optimization process, the amount of data must be reduced. Therefore, a semi-supervised regression model is used as the benchmark model.

[0082] The process of establishing a cost model is as follows: Figure 5As shown. First, based on different hardware architectures, such as CPU, GPU, FPGA, NPU, etc., corresponding pre-trained models are extracted. Second, the computation graph is traversed, and features of all operators (fusion operators and their constituent operators) in the graph are extracted, including input data size, input data type, number of operators constituting the fusion operator, and operator type. Then, 10% of the operators are selected and deployed on hardware to actually measure time and energy consumption, which are used as labeled data. Next, according to a semi-supervised algorithm, a model is constructed using the labeled data and predictions are made on unlabeled data. Finally, data with high confidence are added to the labeled dataset, used as labeled data, and the model is retrained. When a certain proportion of data is in the labeled dataset, training ends, and the trained cost model is obtained.

[0083] The fusion effect is evaluated using the trained cost model. If the evaluation results show that the performance deteriorates after fusion, the segmentation operator is used. Finally, the generated subgraph is used as input to the operator layer to generate code.

[0084] (6) Dynamic recutting of subgraph

[0085] The computation graph is traversed in reverse order. Based on the records from the fusion operator, operators that do not conform to the stable fusion rules are re-segmented, and the operators before and after the segmentation are recorded. The recorded content mainly includes the operator features required by the cost model. After segmentation, the internal parallel logic of some fusion operators is disrupted and needs to be rearranged according to the parallel rules between operators. This also needs to be taken into account when training the cost model. The time-consuming and energy-consuming cost models are used to evaluate the operators before and after segmentation. If both time-consuming and energy-consuming decrease after segmentation, the operator is segmented; otherwise, it is adjusted according to the user's needs. If the user requires the lowest time consumption, segmentation is performed if time consumption decreases; if the user requires the lowest energy consumption, segmentation is performed if energy consumption decreases. To prevent excessive time or energy consumption, an energy consumption limit is set. If the limit is exceeded, the segmented operators are readjusted. The specific process is as follows: Figure 6 As shown.

[0086] The following describes several specific implementation methods of this solution. In actual implementation, the methods described in this disclosure are not limited to those described herein. To better illustrate how this method effectively optimizes the solution, some relevant technical concepts and principles are first described.

[0087] Example 1: This example briefly illustrates the principle behind this method's reduction of model inference time. A simple CNN structure is shown below. Figure 7 As shown on the left. According to the present invention, the result of optimization is as follows... Figure 7As shown on the right side. In the original structure, after the CONV calculation, the data needs to be stored in memory, and during the preparation for BN, the data has to be transferred from memory to the on-chip cache. After the BN calculation, the data also needs to be transferred twice before starting the RELU layer calculation. According to the principle of operator fusion, in the optimized computational graph structure, the data after the CONV calculation is stored in the on-chip cache, and the BN operation is directly performed. After the BN calculation, the data is also stored in the cache, and then the RELU operation is directly carried out. Compared with the original structure, the process of transferring data from memory four times between operators is omitted, saving a large amount of inference time.

[0088] Example 2: Through this example, briefly illustrate the principle strongly related to hardware of this method. As Figure 8 The left side shows the structure before the fusion of two operators in a neural network; Figure 8 The middle shows the structure after the fusion of these two operators on hardware A; Figure 8 The right side shows the structure after the fusion of these two operators on hardware B. For different hardware A and B, this method will construct different cost models and output different computational graphs according to the running time of the operators evaluated by the cost models. For hardware A, construct cost model a. Cost model a predicts that the running time of CONV1 is t1, the running time of CONV2 is t2, and the running time of the fused composite operator is t3. Since t3 < t1 + t2, it is decided to fuse this operator, and the computational graph shown in Figure 8 the middle is output. For hardware B, construct cost model b. Cost model b predicts that the running time of CONV1 is t1', the running time of CONV2 is t2', and the running time of the fused composite operator is t3'. Since t3' > t1' + t'2, it is decided to split this operator, and the computational graph shown in Figure 8 the right side is output. The method of using the same set of operator fusion strategies for all hardware platforms has a small optimization cost, but obviously these strategies are not suitable for every kind of hardware, and there will be a certain loss of performance on some hardware. This method will dynamically modify the output computational graph for different hardware to minimize the power consumption of the model running on all hardware.

[0089] Example 3: Through this example, briefly illustrate the process of operator fusion for complex models by this method. Figure 9 The left side shows a partial structure in Google Net; Figure 9The right side shows the structure after static operator fusion using this method. Traversing the computation graph in reverse order, the Concat operator's input is an unfusionable operator, so it is not fused. Conv1x1 and the Conv3x3 operator satisfy the basic fusion rule but not the stable fusion rule, so they are fused into a Conv3-Conv1 operator, and the fusion benefit of this operator will be re-evaluated later. The Conv1x1 operator and the Conv5x5 operator satisfy the basic fusion rule and are fused into a Conv5-Conv1 operator. Conv1x1 and MaxPool3x3 satisfy the basic fusion rule but not the stable fusion rule, so they are fused into a MaxPool3-Conv1 operator. Conv1x1 and MaxPool3x3 do not satisfy the basic fusion rule, so they are not fused. Conv3x3 and ReLU satisfy the stable fusion rule and are fused into a Conv3-ReLU operator, and this fusion operator will not be further split later. The fusion of other operators is similar.

[0090] Example 4: This example briefly illustrates the process of parallel processing of the computation graph using this method. Figure 10 The parts of the computation graph after static operator fusion that undergo parallel processing are marked; other parts only undergo parallel processing within each operator. In the box shown above, operators have data dependencies, so they are processed serially. In the box shown below, operators have no data dependencies and can be processed in parallel. The degree of parallelism is determined based on the abstracted virtual computation unit. If the hardware has four computation units capable of performing convolution operations, the four operators in the box shown below can be completely parallelized, and the final runtime depends only on the operator with the longest runtime. If the hardware has only one computation unit capable of performing convolution operations, then the operators in the red box can only be processed serially, and the final runtime is the sum of the runtimes of the four operators.

[0091] Example 5: This example briefly illustrates the process of operator segmentation in the CNN structure using this method. Operators that have been fused but do not conform to the stable fusion rule are re-evaluated using a cost model. The inference time or energy consumption before and after fusion is predicted separately (selected according to actual needs). Operators whose energy consumption increases after fusion are then re-segmented. For example... Figure 11 As shown, after evaluation by the cost model, CONV1-CONV3 showed a decrease in both inference time and energy consumption after fusion, so the fused state was maintained. After evaluation by the cost model, CONV1-CONV5 showed an increase in both inference time and energy consumption after fusion, so the two operators were separated and returned to the unfused state.

[0092] This invention provides an automatic optimization method for deep model computation graphs that is relevant to hardware characteristics. Compared to other computation graph optimization methods, this invention provides the optimal computation graph for different hardware, solving the problem of low coupling between layers and hardware. It eliminates a large amount of manual optimization, using a computer to automatically generate possible optimal subgraphs and evaluate their performance after actual deployment. This invention comprehensively considers and optimizes both power consumption and time factors in the inference process; it uses a fusion-then-segmentation approach, combining dynamic and static optimization to reduce optimization costs and improve optimization results. In this invention, the deep learning compiler only needs to be developed once to automatically generate optimal code for any device. For example, operators developed for CPUs can be used almost unchanged for GPUs and GPUs, thus significantly reducing costs. This invention can be applied to embedded inference scenarios, supporting inference acceleration for most CNN networks. It is compatible with mainstream deep learning compilers and primarily targets hardware such as FPGAs and GPUs.

[0093] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An automatic optimization method for deep model computation graphs related to hardware characteristics, characterized in that, include: S1. Obtain the deep learning model; The deep learning model is converted into an intermediate representation according to preset rules; the intermediate representation is used to describe the structure of the computation graph. S2. Perform the first operator fusion on the intermediate representation according to the static operator fusion rules to form a composite subgraph; The static operator fusion rules include: Data dependencies exist between the merged operators; Merging should not create directed loops between subgraphs; After fusion, the on-chip cache is sufficient, and there is no need to move data from main memory during the computation process; S3. Parallelize the operators in the composite subgraph according to hardware resources, perform a second operator fusion, and form an aggregated subgraph. The parallelization process includes: abstracting the underlying hardware resources into virtual computing units, scheduling the computation to be performed onto the computing units, and enabling the operators to complete the parallelization process. The parallelization process includes intra-operator parallelism and inter-operator parallelism; The internal parallelism of the operator includes: sorting out the parallel logic between operators when merging operators, and parallelizing the computations that have no data dependency within the merged operator when hardware resources allow; The inter-operator parallelism includes: continuing to perform parallel operations on the fused operators after operator fusion; and simultaneously readjusting the internal parallelism of the operators while the fused operators are in parallel. If, while reducing the internal parallelism of operators, the excess resources can achieve inter-operator parallelism, then the inter-operator parallelism strategy should be given priority. S4. Perform cross-boundary optimization processing on the aggregated subgraph; S5. Based on the records of the first operator fusion and the second operator fusion, form a list of original operators and a list of fused operators; construct a cost model on the target hardware platform based on the list of original operators and the list of fused operators, and evaluate the fusion effect; S6. Based on the evaluation results of the fusion effect, the fusion operator is segmented, and the generated subgraph is used as the input generation code of the operator layer.

2. The method for automatic optimization of hardware characteristic-related deep model computation graphs as described in claim 1, characterized in that, The intermediate representation is used to define the computation type, number of inputs and number of outputs of each operator in the computation graph, as well as the relevant parameters of the computation; the intermediate representation is also used to record the number of constituent operators of the fusion operator, as well as the relevant information of the constituent operators.

3. The method for automatic optimization of hardware characteristic-related deep model computation graphs as described in claim 1, characterized in that, S2 specifically includes: based on the intermediate representation after conversion, traversing the computation graph in reverse order starting from the output, and fusing operators in the computation graph that satisfy the static operator fusion rules; iterating this process multiple times until there are no operators in the computation graph that can be fused; and re-integrating the fused computation graph to form a composite subgraph.

4. The method for automatic optimization of hardware characteristic-related deep model computation graphs as described in claim 1, characterized in that, S4 specifically includes: To traverse the computation graph using topological sorting, first select a node with an in-degree of 0 to execute, then delete that node and the edges connected to it, and then find the next node with an in-degree of 0 to execute. Cross-boundary optimization is performed during the traversal of the computation graph; the cross-boundary optimization includes: constant folding, inlining optimization, and common sub-representation extraction.

5. The method for automatic optimization of a hardware characteristic-related deep model computation graph as described in claim 1, characterized in that, In S5, the cost model is trained in the following manner: Traverse the computation graph, extract the features of all operators in the computation graph at this time, and construct the number of operators and operator types of the fusion operator based on the features; Select a preset number of operators and deploy them on the hardware, then measure the hardware's time and energy consumption as marker data; Based on the semi-supervised algorithm, a model is constructed using labeled data and predictions are made using unlabeled data; Add high-confidence data to the labeled dataset, use it as labeled data, and retrain the model; Training ends when the number of data points in the labeled dataset exceeds a preset proportion, resulting in a well-trained cost model.

6. The method for automatic optimization of hardware characteristic-related deep model computation graphs as described in claim 1, characterized in that, In step S6, if the fusion effect does not meet the preset conditions, the fusion operator is segmented.

Images