Method for fusing operators in computational graph, and corresponding apparatus

Abstract

Provided in the present application is a method for fusing operators in a computational graph, which method is applied to an AI compiling process. The method comprises: acquiring a target graph, wherein the target graph comprises a plurality of nodes and at least one edge, and the nodes are associated with operators in a computational graph; determining computation types of two nodes connected to a first edge, wherein computation logic of two operators corresponding to the two nodes both comprise a loop statement; and on the basis of a first computation type and a second computation type, fusing the loop statement of one operator into the loop statement of the other operator, so as to obtain a fused operator which is obtained by means of fusing the two operators. In the solution provided in the present application, it is not necessary to compress all loop statements of operators into one dimension and recalculate indexes for the operators, instead, two operators can be fused by means of computation types of the two operators, so that a high-performance fused operator can be obtained.

Description

A method and corresponding device for operator fusion in a computational graph

[0001] This application claims priority to a Chinese patent application filed with the State Intellectual Property Office on February 7, 2024, with application number 202410175944.6 and application name “A method for operator fusion in a computational graph and corresponding device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a method for fusion of operators in a computational graph and a corresponding device. Background Art

[0003] The AI ​​compilation optimization framework uses a computation graph to represent the network structure. A computation graph is a directed graph with operators (OPs) as nodes, representing a computation function. A computation function sequentially executes the operators in the directed graph on input tensors, producing a final output tensor. An operator is essentially a computation function. Some operators may also include the computational logic of multiple simple operators. These relatively complex operators can be considered fused operators. During the compilation process, data-dependent producer and consumer operators on the computation graph can be fused into a single fused operator, improving memory locality for intermediate tensors during the computation. This means that intermediate data generated by the producer operator does not need to be moved out to global memory and then back into the consumer operator. Instead, it is directly passed to the next compute unit within the fused operator for computation completion, thus resolving the memory wall problem. This fusion technique is also collectively referred to as "buffer fusion."

[0004] Buffer fusion has long been the mainstream technology for operator fusion. However, in the era of large models, operators are becoming increasingly complex, and many operators contain loops, making operator fusion increasingly difficult. Currently, the fusion of operators containing loops primarily uses InlineFusion technology. This compresses all loops into a single dimension, then uses dependency analysis to calculate the index of the input statement and fuses the input and output statements together to achieve operator fusion.

[0005] A lot of scalar calculations will be introduced in the index calculation process, which will occupy a lot of hardware resources. Moreover, the codes of the fusion operators obtained in this way are compressed, resulting in poor performance of the fusion operators. Summary of the Invention

[0006] This application provides a method for fusion of operators in a computation graph, for improving the performance of fused operators. This application also provides corresponding apparatus, computer-readable storage medium, and computer program product.

[0007] In a first aspect, the present application provides a method for fusing operators in a computation graph, comprising: obtaining a target graph, the target graph comprising multiple nodes and at least one edge; wherein each edge is used to connect two nodes having a dependency relationship among the multiple nodes, and the nodes are associated with operators in the computation graph; determining a first computation type and a second computation type, the first computation type being the computation type of the first node connected by the first edge, the second computation type being the computation type of the second node connected by the first edge, the first edge being included in at least one edge, the first node corresponding to the first operator in the computation graph, the second node corresponding to the second operator in the computation graph, and the computation logic of the first operator and the second operator both including loop statements; and merging the loop statement of the first operator into the loop statement of the second operator according to the first computation type and the second computation type to obtain a fused operator after the first operator and the second operator are fused.

[0008] In this application, the computation graph includes operators and edges. Edges are used to connect two operators with a dependency relationship. For example, if the output of the first operator is the input of the second operator, then the first operator and the second operator have a dependency relationship. An operator can be understood as a computation function. The computation function can be a simple computation function, such as performing an addition or multiplication operation, or a more complex computation function. Some operators also include loop statements in their computation functions. When an operator including a loop statement executes the computation function of the operator, multiple loop iterations will be performed on the portion of the loop statement until the loop end condition is met, at which point the loop is exited.

[0009] In this application, the target graph can be a computation graph or a dependency graph obtained by performing relational dependency analysis on the computation graph. If the target graph is a computation graph, then the nodes are operators, and the edges are edges connecting two operators with a dependency relationship. If the target graph is a dependency graph, then the nodes in the dependency graph correspond to the operators in the computation graph, and the edges correspond to the edges connecting the operators in the computation graph.

[0010] In this application, there can be multiple types of calculations, such as: Elemwise operation, broadcast operation, reshape operation, transpose operation, reduce operation, etc.

[0011] In the present application, the merging of loop statements of two operators can be done directly or after loop transformation.

[0012] In the first aspect above, for two operators with a dependency, the loop statement of one operator can be merged into the loop statement of the other operator based on their computation types. This eliminates the need to compress all operator loop statements into one dimension and recalculate operator indices. Instead, the two operators can be fused based on their computation types, resulting in a high-performance fused operator.

[0013] In one possible implementation, if it is determined by searching a combination table based on the first calculation type and the second calculation type that the loop iteration space of the first operator overlaps with the loop iteration space of the second operator, the loop statement of the first operator can be directly merged into the loop statement of the second operator.

[0014] In this possible implementation, the first calculation type and the second calculation type can be the same calculation type or different calculation types. The loop statement in the operator will have a loop iteration space. The loop iteration space of the first operator and the loop iteration space of the second operator may be the same or different. When the loop iteration spaces are the same, the loop statements can be directly merged.

[0015] The loop iteration space can be understood as a two-dimensional space that includes every iteration instance of the loop statement within the loop. If each iteration instance is represented by a point, the points of all iteration instances within the entire loop can be constructed into a rectangle. This rectangle can be regarded as a polyhedron in two-dimensional space. This space is called the loop iteration space of the calculation. If the iteration space of the first operator and the second operator are the same, then the loop statement of the first operator can be directly merged into the loop statement of the second operator. This can improve both the code performance of the fused operator and the efficiency of the operator fusion.

[0016] In one possible implementation, the above step of: merging the loop statement of the first operator into the loop statement of the second operator according to the first calculation type and the second calculation type includes: determining at least one loop transformation method according to the first calculation type and the second calculation type; processing the loop statement of the first operator according to the at least one loop transformation method to transform the loop iteration space of the first operator to overlap with the loop iteration space of the second operator; and merging the loop statement of the first operator into the loop statement of the second operator.

[0017] In this possible implementation, if at least one loop transformation method is determined based on the first and second calculation types, it indicates that the loop statement of the first operator needs to be processed accordingly to transform the loop iteration space of the first operator to overlap with the loop iteration space of the second operator, and then merge the loop statements. Thus, the solution of the present application can assist in determining the loop transformation method for the first operator through the calculation types of the two operators, thereby enabling high-performance fusion of the two operators.

[0018] In one possible implementation, at least one cyclic transformation mode is obtained by searching a combination table according to the first calculation type and the second calculation type, and the combination table includes a correspondence between a combination of the first calculation type and the second calculation type and the at least one cyclic transformation mode.

[0019] In one possible implementation, the combination table is:

[0020] The first row represents the six calculation types of the first operator, and the first column represents the six calculation types of the second operator. Indicates circular replication, represents a cyclic exchange, Indicates loop segmentation, Indicates cycle compression, Indicates loop backtracking. [] is an empty set, indicating that the loop iteration spaces of the first and second operators overlap. When the crossbar is blank, it indicates that the calculation type of the first and second operators cannot be Reshape and Transpose at the same time.

[0021] In this possible implementation, a table lookup can be used to quickly determine whether the loop iteration spaces of the first operator and the second operator are consistent through the first calculation type and the second calculation type, and the loop transformation method used when they are inconsistent. In this way, the speed of operator fusion can be improved.

[0022] In a possible implementation, the method further includes: determining a calculation type of the fusion operator according to the first calculation type and the second calculation type.

[0023] In this possible implementation, the calculation type of the fusion operator is determined by the first calculation type of the first operator and the second calculation type of the second operator, which facilitates further optimization of the fusion operator.

[0024] In one possible implementation, the calculation type of the fusion operator is obtained by searching a calculation type deduction table based on the first calculation type and the second calculation type. The calculation type deduction table includes a correspondence between a combination of the first calculation type and the second calculation type and the calculation type of the fusion operator.

[0025] In one possible implementation, the calculation type deduction table is:

[0026] The first row represents the six calculation types of the first operator, the first column represents the six calculation types of the second operator, and the calculation type at the intersection position is the calculation type of the fusion operator.

[0027] In this possible implementation, the calculation type of the fusion operator can be quickly determined through the first calculation type and the second calculation type by looking up a table, which is conducive to further optimization of the fusion operator.

[0028] In one possible implementation, the computation graph is a dynamic computation graph or a static computation graph. The shape of a tensor in a dynamic computation graph is described by a variable; the shape of a tensor in a static computation graph is described by a constant.

[0029] In this possible implementation, whether it is a static computation graph or a dynamic computation graph, a high-performance fusion operator can be obtained through loop merging.

[0030] In one possible implementation, the computation graph is a dynamic computation graph, and the method further includes: performing symbolic deduction on the computation graph to determine the relationship between the first variable and the second variable in the computation graph; wherein the relationship between the first variable and the second variable is used to indicate a loop merging method or to indicate the relationship between input data and output data of operators of different computation types, the first variable is a variable used to describe the shape of a first tensor, the second variable is a variable used to describe the shape of a second tensor, and the first tensor and the second tensor are different tensors in the computation graph.

[0031] In this possible implementation, the relationship between the first and second variables is used to indicate the loop merging method or to indicate the relationship between the input data and output data of operators of different computation types. This allows support for dynamic shapes and implicit broadcast loop merging.

[0032] In one possible implementation, when the first node serves as a source node and there are multiple corresponding target nodes, the method further includes: determining a second node with the highest priority among the multiple target nodes; and performing dependency inversion on the target nodes other than the second node among the multiple target nodes and the source node, so that the first node becomes the target node in the dependency relationship between the first node and the target nodes other than the second node.

[0033] In this possible implementation, when there are multiple target nodes, a unique target node can be determined for the source node through dependency inversion, thereby achieving efficient fusion of operators.

[0034] In a possible implementation, the method further includes: determining, according to the loop transformation mode, a loop control parameter corresponding to the loop transformation mode, where the loop control parameter is used to control a loop statement for processing the first operator.

[0035] In this possible implementation, loop input parameters can be determined through loop transformation. For example, in loop replication, loop input parameters can include the number of replicated loops, the loop insertion position, the upper bound (starting position) and lower bound (ending position) of the replicated loops, and the step size. In this way, loop processing accuracy can be improved by controlling loop input parameters.

[0036] In a possible implementation, the method further includes: updating the information of the second operator to information of the fusion operator, where the information of the fusion operator includes a calculation type of the fusion operator.

[0037] In this possible implementation, after the first operator is fused into the second operator, the information of the second operator may be updated, and the calculation type of the second operator may be updated to the calculation type of the fused operator, which is convenient for the next cycle merging.

[0038] In a possible implementation, when there are multiple edges, the method further includes: performing a cyclic merging process on two nodes associated with the second edge.

[0039] In this possible implementation, the process of performing loop merging on the two nodes associated with the second edge can be understood by referring to the loop merging process of the first operator and the second operator above. In this application, multiple rounds of loop merging can be performed on the computation graph, so that the computation graph can be better optimized.

[0040] A second aspect of the present application provides a computer device, comprising:

[0041] An acquisition unit is used to acquire a target graph, the target graph including a plurality of nodes and at least one edge; wherein each edge is used to connect two nodes having a dependency relationship among the plurality of nodes, and the nodes are associated with operators in the computation graph;

[0042] a first processing unit, configured to determine a first calculation type and a second calculation type, wherein the first calculation type is a calculation type of a first node connected to a first edge, and the second calculation type is a calculation type of a second node connected to the first edge, the first edge is included in at least one edge, the first node corresponds to a first operator in a calculation graph, the second node corresponds to a second operator in the calculation graph, and the calculation logic of the first operator and the second operator both include loop statements;

[0043] The second processing unit is used to merge the loop statement of the first operator into the loop statement of the second operator according to the first calculation type and the second calculation type, so as to obtain a fused operator after the first operator and the second operator are fused.

[0044] In one possible implementation, if it is determined by searching a combination table based on the first calculation type and the second calculation type that the loop iteration space of the first operator overlaps with the loop iteration space of the second operator, the loop statement of the first operator can be directly merged into the loop statement of the second operator.

[0045] In one possible implementation, the second processing unit is specifically configured to determine at least one loop transformation method based on the first calculation type and the second calculation type; process the loop statement of the first operator according to the at least one loop transformation method to transform the loop iteration space of the first operator to overlap with the loop iteration space of the second operator; and merge the loop statement of the first operator into the loop statement of the second operator.

[0046] In one possible implementation, at least one cyclic transformation mode is obtained by searching a combination table according to the first calculation type and the second calculation type, and the combination table includes a correspondence between a combination of the first calculation type and the second calculation type and the at least one cyclic transformation mode.

[0047] In one possible implementation, the combination table is:

[0048] The first row represents the six calculation types of the first operator, and the first column represents the six calculation types of the second operator. Indicates circular replication, represents a cyclic exchange, Indicates loop segmentation, Indicates cycle compression, Indicates loop backtracking. [] is an empty set, indicating that the loop iteration spaces of the first and second operators overlap. When the crossbar is blank, it indicates that the calculation type of the first and second operators cannot be Reshape and Transpose at the same time.

[0049] In a possible implementation, the second processing unit is further configured to determine a calculation type of the fusion operator according to the first calculation type and the second calculation type.

[0050] In one possible implementation, the calculation type of the fusion operator is obtained by searching a calculation type deduction table based on the first calculation type and the second calculation type. The calculation type deduction table includes a correspondence between a combination of the first calculation type and the second calculation type and the calculation type of the fusion operator.

[0051] In one possible implementation, the calculation type deduction table is:

[0052] The first row represents the six calculation types of the first operator, the first column represents the six calculation types of the second operator, and the calculation type at the intersection position is the calculation type of the fusion operator.

[0053] In one possible implementation, the computation graph is a dynamic computation graph or a static computation graph. The shape of a tensor in a dynamic computation graph is described by a variable; the shape of a tensor in a static computation graph is described by a constant.

[0054] In one possible implementation, the first processing unit is further used to perform symbolic deduction on the dynamic computation graph to determine the relationship between the first variable and the second variable in the dynamic computation graph; wherein the relationship between the first variable and the second variable is used to indicate a loop merging method or to indicate the relationship between input data and output data of operators of different computation types, the first variable is a variable used to describe the shape of the first tensor, the second variable is a variable used to describe the shape of the second tensor, and the first tensor and the second tensor are different tensors in the dynamic computation graph.

[0055] In one possible implementation, when the first node serves as a source node and there are multiple corresponding target nodes, the first processing unit is also used to: determine the second node with the highest priority among the multiple target nodes; and perform dependency inversion on the target nodes other than the second node among the multiple target nodes and the source node, so that the first node becomes the target node in the dependency relationship between the first node and the target nodes other than the second node.

[0056] In a possible implementation, the second processing unit is further configured to determine, according to the loop transformation mode, a loop control parameter corresponding to the loop transformation mode, where the loop control parameter is used to control a loop statement for processing the first operator.

[0057] In a possible implementation, the second processing unit is further configured to update the information of the second operator to information of the fusion operator, where the information of the fusion operator includes a calculation type of the fusion operator.

[0058] In a possible implementation, the second processing unit is further configured to perform a cyclic merging process on two nodes associated with the second edge when there are multiple edges.

[0059] In a possible implementation, the target graph is a computation graph, the first node is a first operator, the second node is a second operator, and the first edge is an edge used to connect the first operator and the second operator.

[0060] In one possible implementation, the target graph is a dependency graph obtained by performing dependency analysis on the computation graph.

[0061] In a third aspect, the present application provides a computer device comprising a processor and a computer-readable storage medium storing a computer program; the processor is coupled to the computer-readable storage medium, and when the computer program is executed by the processor, the method according to the first aspect or any possible implementation method is implemented.

[0062] A fourth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0063] The fifth aspect of the present application provides a computer program product storing one or more computer-executable instructions. When the computer-executable instructions are executed by the processor, the processor executes the method of the above-mentioned first aspect or any possible implementation of the first aspect.

[0064] A sixth aspect of the present application provides a chip system, which includes a processor for supporting a computer device to implement the functions involved in the first aspect or any possible implementation of the first aspect. In one possible design, the chip system may also include a memory for storing program instructions and data necessary for training nodes. The chip system may be composed of a chip or may include a chip and other discrete devices.

[0065] Among them, the technical effects brought about by the second aspect or any possible implementation method of the second aspect to the sixth aspect can refer to the technical effects brought about by the first aspect or different possible implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] FIG1A is a schematic diagram of an example of a calculation graph provided in an embodiment of the present application;

[0067] FIG1B is a schematic diagram of an example of a loop iteration space provided in an embodiment of the present application;

[0068] FIG2A is a schematic diagram of a structure of a cloud system provided in an embodiment of the present application;

[0069] FIG2B is another schematic diagram of the structure of the cloud system provided in an embodiment of the present application;

[0070] FIG2C is a schematic diagram of a structure of data provided in an embodiment of the present application;

[0071] FIG3 is a schematic diagram of the structure of an AI compilation device provided in an embodiment of the present application;

[0072] FIG4 is a schematic diagram of an AI architecture provided in an embodiment of the present application;

[0073] FIG5 is a schematic diagram of an embodiment of a method for operator fusion in a computation graph provided by an embodiment of the present application;

[0074] FIG6 is a schematic diagram of another embodiment of a method for operator fusion in a computation graph provided in an embodiment of the present application;

[0075] FIG7 is a schematic diagram of an example of a calculation graph provided in an embodiment of the present application;

[0076] FIG8 is a schematic diagram of another example of a calculation graph provided in an embodiment of the present application;

[0077] FIG9 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present application, rather than all the embodiments. Those skilled in the art will appreciate that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0079] The terms "first," "second," and the like in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0080] The present application provides a method for fusion of operators in a computation graph, for improving the performance of fused operators. The present application also provides corresponding apparatus, computer-readable storage medium, and computer program product, etc. These are described in detail below.

[0081] To facilitate understanding, the following briefly introduces the technical terms involved in the embodiments of this application:

[0082] 1. Artificial Intelligence (AI): AI is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theory.

[0083] 2. AI Framework: The core function of an AI framework is to provide developers with an interface (mathematical operations) for building neural networks. It automatically performs reverse differentiation of neural network training, approximating the optimal value, and obtaining a neural network model (approximation function) that can be used to solve classification, regression, and fitting problems, and implement application scenarios such as object classification and speech recognition. Mainstream AI frameworks in the industry include Mindspore, TensorFlow, and PyTorch. AI frameworks generally use computational graphs to represent network structures. AI frameworks are typically designed with a multi-layered intermediate representation (IR), primarily to meet both ease of use and high performance requirements. For example, to facilitate developer use, the front-end (layer) of an AI framework typically uses a high-level IR. This IR abstracts and encapsulates tensor computations as much as possible, allowing developers to focus solely on the logical model and operators. In contrast, the back-end (operator layer) of the AI ​​framework typically uses a low-level IR for operator performance optimization. This IR allows the operator-layer compiler to perform more fine-grained optimizations based on hardware characteristics.

[0084] 3. Computational Graph: A computational graph is a directed graph with operators as nodes, representing a computational function. Within the AI ​​framework, this computational function sequentially calls the operator nodes in the directed graph on the input tensor, producing the final output tensor. An operator is essentially a computational function. For example, Add and Sigmoid are both called operators. However, a complex operator like Sigmoid is mathematically implemented using basic operators such as Neg, Exp, Add, and Reciprocal. Therefore, computational graphs and operators are essentially the same in terms of computational nature. Operators are packaged computational graphs, while computational graphs are unpacked operators. Therefore, in theory, one can first define a small set of "basic operators" and then use one or more basic operators to equivalently represent any existing operator, thereby further expressing any existing computational graph. This computational graph composed of multiple basic operators is called a "fused operator." Alternatively, the term "fused subgraph" or "fused operator" refers to all subgraphs containing different basic operators. From a layer perspective, this is a subgraph, and from an operator perspective, it is a fused operator. The structure of a computational graph can be understood by referring to Figure 1A. As shown in Figure 1A, the computational graph includes operators A, B, C, D, and E. Operators A and B are dependent on each other, with the output of A serving as the input of B. Operators B and C are dependent on each other, with the output of B serving as the input of C. Similarly, operators A and D are dependent on each other, with the output of A serving as the input of D. Operators D and E are dependent on each other, with the output of D serving as the input of E.

[0085] 4. Layer Compiler: Mainstream graph compilers include MindSpore's MindCompiler, TensorFlow's XLA, and TVM's Relay. These focus on non-loop optimizations. In addition to common optimizations found in traditional compilers, such as constant folding, algebraic simplification, and common subexpressions, they also perform layout conversion and operator fusion. By analyzing and optimizing the existing network computation graph logic, they split, reorganize, and fuse the original computation logic to reduce the overhead between operator executions and improve device computing resource utilization, thereby optimizing overall network execution time.

[0086] 5. Operator-level compilers: Mainstream operator compilers include the MindSpore automatic kernel generator (AKG), BaldeDISC, and TVM. These compilers primarily focus on scheduling-related optimizations for low-level IR, such as loop transformation and loop splitting, as well as back-end pass optimizations for hardware intrinsic mapping and memory allocation.

[0087] 6. Operator Fusion: Operator fusion is the most important single-card performance optimization method for AI networks. Operator fusion primarily combines data-dependent producer and consumer operators on the computation graph into a single fused operator, thereby improving the memory locality of intermediate tensor data. Specifically, intermediate data generated by the producer operator does not need to be moved out to global memory and then from global memory to the consumer operator. Instead, it is directly fed to the next computing unit within the fused operator for computation, thus solving the memory wall problem. This fusion technology is also collectively referred to as "memory (buffer) fusion."

[0088] 7. Large model: Also known as large language model (LLM). Language model (LM) refers to a model that models natural language. Usually, given a natural language x 1～t Under the premise of , LM will predict the probability of the next word appearing according to the above context, that is, the modeling distribution P(x t+1 |x 1～t ). A variety of machine learning models can be used to model natural language to form so-called language models, such as hidden Markov chains, naive Bayes models, neural networks, etc. After entering the era of deep learning, neural network models have gradually become mainstream. In particular, after the transformer architecture was proposed, neural networks of this paradigm gradually became the mainstream of LM. The first L in LLM stands for Large. Compared with traditional LM, its characteristic is the huge parameter scale, which can often reach tens of billions or even hundreds of billions or trillions. Current LLMs mostly use transformers as the basic architecture, use massive text data for pre-training, and have powerful language modeling capabilities.

[0089] 8. Polyhedron Model: The operator optimization in the aforementioned AKG can be implemented by processing the polyhedron model. The polyhedron model is an abstract mathematical model that has gradually formed and developed based on linear algebra, linear programming, set theory, mathematical logic, and optimization theory. It describes a program using iteration space, statement instances, access relationships, dependencies, and scheduling, and implements a series of parallelism and locality-related optimizations based on scheduling transformations. During the optimization process, operators in the computational graph can be translated into poly statements used in the polyhedron model. Polyhedron abstract modeling is then performed. Based on full consideration of the characteristics of the underlying architecture, tensor-level loop transformations are implemented through the polyhedron scheduling optimization module. The optimized loops are then task-mapped to hardware-related optimization modules, and code is finally generated for the target hardware.

[0090] 9. Poly statement (referred to as statement): Poly statement is the basic unit of scheduling in the polyhedron model, represented by S_index, for example: S_0, S_1. The above example is described in detail below. An example of a loop nesting is as follows: for(int i=1; i <N;i++) for(int j=1;j<N;j++) A[i,j]=f(A[i-1][j],A[i][j-1]);

[0091] Where N is a constant. Statements within the loop nest update the data at position A[i][j] by referencing the data stored in A[i-1][j] and A[i][j-1]. If each iteration of the statement within the loop is abstracted as a point in space, a two-dimensional space based on (i, j) can be constructed, as shown in Figure 1B. Each black point represents an iteration of the statement that writes to A[i][j]. This allows us to construct a rectangle consisting of all the black points. This rectangle can be considered a polyhedron in two-dimensional space, and this space is called the iteration space of the computation. Furthermore, this two-dimensional polyhedron can be represented by a set in algebra, namely, {[i, j]: 1 <= i <= N-1 and 1 <= j <= N–1}, where [i, j] is a two-tuple and the inequality following the “:” represents the interval of this set. Let this tuple be called S, representing a statement, then the Polyhedron of this statement can be expressed as {S[i, j]: 1<=i<=N-1and1<=j<=N–1}.

[0092] 10. Computational operations: Operator fusion or operator optimization involves multiple computational operations or multiple types of computational operations, including reshape operations, transpose operations, elemwise operations, broadcast operations, and reduce operations.

[0093] 11. Reshape operation: The reshape function is a function in MATLAB that transforms a specified matrix into a matrix of a specific dimension, and the number of elements in the matrix remains unchanged. The function can readjust the number of rows, columns, and dimensions of the matrix. The function syntax is B = reshape(A, size), which means returning an n-dimensional array with the same elements as A, but the size of the reconstructed array dimension is determined by the vector size (size). The reshape operation can be understood as adjusting the shape of a tensor, but the number and relative position of the data in it remain unchanged. For example: a tensor with one row and two columns: [0.2, 0.1], its shape can be expressed as (1, 2), where the data is 0.2, 0.1. After reshaping it to (2, 1), it will become a tensor with two rows and one column:

[0094] [0.2,

[0095] 0.1].

[0096] 12. Transpose operation: The transpose operation can be understood as an operation that interchanges the rows and columns of a matrix or array, thereby changing the arrangement of data without changing the data itself to meet specific calculation or analysis needs.

[0097] 13.Elemwise operation: The Elemet-Wise operator refers to an element-by-element operation on the input tensor. For example, for each value of the input tensor, it is judged whether it is greater than 0. If it is greater than 0, the output is the input, otherwise it is 0.

[0098] 14. Reduce Operation: Reduce is a common operation in functional programming that iterates a data set and returns a single result. This operation applies a binary function to the elements of the data set, gradually reducing the data set to a single value.

[0099] 15. Broadcast: This includes automatic broadcast and implicit broadcast. Automatic broadcast allows the AI ​​compilation framework to automatically expand the shape of tensorA to match that of tensorB, and then perform element-by-element operations (e.g., addition) on tensorA with a shape of (1024, 1) and tensorB with a shape of (1024, 1024). Implicit broadcast is an operation currently supported by all AI frameworks by default. It allows two compatible tensors with different shapes to perform elemwise operations by automatically expanding their shapes. For example, when performing an elemwise operation (such as addition) on tensorA with a shape of (1024, ?) and tensorB with a shape of (1024, 1024), the "?" represents a dynamic shape, so the shape of tensorA cannot be determined until runtime. Consequently, there are two possible shapes for tensorA: (1024, 1) or (1024, 1024). These two situations correspond to the need for automatic broadcasting and the lack of automatic broadcasting, respectively. Therefore, this scenario with a dynamic shape is called implicit broadcasting. Broadcasting scenarios involve dynamic shapes and fixed shapes, which are a pair of concepts. A fixed shape refers to a shape size that is specified at compile time and does not need to be specified at runtime. When developing fixed-shape operators, an operator's source code can support multiple fixed shapes, but the actual shape value must be specified at compile time. Dynamic shapes, on the other hand, do not specify the shape size at compile time; the actual shape size is passed in at runtime. The compiled binary file of the operator supports arbitrary shapes or one or more shape ranges.

[0100] 16. InlineFusion Technology: Inlinefusion is a compiler optimization technique used to inline function calls to their call locations during compilation, reducing function call overhead and improving program performance. In this application, this refers to the operator fusion technology implemented by BladeDISC based on InlineFusion technology. Specifically, it compresses all loops into one dimension and recalculates the indices for different operators. For example:

[0101] Through inline fusion, the calculation process of out1 is inlined into the calculation process of out2. It is necessary to recalculate the index of out1 based on the loop nest where out2 is located:

[0102] 17. Program Dependency Analysis: Program dependency analysis is a static analysis technique used to identify the dependencies between components in a computer program. It analyzes the references and call relationships between variables, functions, modules, or classes in the code, as well as the flow of data and control, to understand the structure and behavior of the program.

[0103] 18. Loop merging: Loop merging is a loop transformation technique that merges multiple loop nests into a unified iteration space, making it easier for the optimizing compiler to implement other optimizations.

[0104] 19. Loop Interchange: Loop Interchange is a loop transformation technique that swaps the nesting order of two loops within a perfectly nested loop.

[0105] 20. Loop segmentation: Loop segmentation is a loop transformation technique that divides the iteration space of a loop into multiple subsets and executes each subset as a scheduling unit.

[0106] 21. Loop compression: Loop compression, also known as loop linearization, is a loop transformation technique that compresses multi-layer loop nesting into a single-layer loop.

[0107] The method for operator fusion in a computational graph provided in an embodiment of the present application can be applied to the model training stage or the model reasoning stage. Whether in the model training stage or the model reasoning stage, the operator fusion process can be executed on a cloud system, an independent server or a terminal device. The following is an introduction using the execution of the operator fusion process on a cloud system as an example.

[0108] FIG2A is a schematic structural diagram of a cloud system provided in an embodiment of the present application.

[0109] As shown in Figure 2A, the cloud system includes an AI compilation device and multiple resource nodes. The AI ​​compilation device can communicate with the multiple resource nodes. The AI ​​compilation device can compile and optimize the computation graph, such as fusing operators in the computation graph to obtain a fused operator. The AI ​​compilation device can then send the optimized computation graph to one or more resource nodes for execution.

[0110] The AI ​​compilation device in Figure 2A can be an independent device, or it can be integrated into the control platform of the cloud system, or integrated into a resource node.

[0111] The functions of the AI ​​compilation device can be implemented through software or hardware.

[0112] As an example of a software functional unit, an AI compilation device may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the AI ​​compilation device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0113] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Inter-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0114] As an example of a hardware functional unit, an AI compilation device may include at least one computing device, such as a server. Alternatively, the AI ​​compilation device may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0115] The multiple computing devices included in the AI ​​compilation apparatus can be distributed in the same region or in different regions. The multiple computing devices included in the AI ​​compilation apparatus can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the AI ​​compilation apparatus can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0116] The cloud system provided in the embodiments of the present application may be a cloud service system. In the cloud service system, as shown in FIG2B , the cloud service system includes a cloud platform and basic resources. The cloud platform includes a cloud platform manager, and the AI ​​compilation device in FIG2A may be the cloud platform manager in FIG2B . The basic resources may include multiple servers, each of which may include multiple resource nodes.

[0117] The resource nodes in Figures 2A and 2B may be computing device cards or virtual machines (VMs), wherein the computing device cards may be at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU).

[0118] The cloud platform manager can compile and optimize the computational graph, such as performing operator fusion.

[0119] The cloud system provided in the embodiments of the present application can be a data center. As shown in FIG2C , the data center includes a data center management platform, an internal data center network, and multiple servers. Each server includes a hardware layer and a software layer. The hardware layer includes memory, a network card, a processor, and a disk, which are connected via a bus. The hardware layer provides the hardware resources required for the virtual machines in the software layer to operate. The software layer includes a host operating system and multiple virtual machines. The host operating system can include a data center management platform client that can interact with the data center management platform.

[0120] Virtualization technology, which mainly consists of computing virtualization and input / output (I / O) virtualization, shares a physical server with multiple tenants at the granularity of virtual machines, allowing tenants to use physical resources conveniently and flexibly under the premise of secure isolation, and can greatly improve the utilization of physical resources.

[0121] Compute virtualization is the process of making a server's computing resources, such as processors and memory, available to virtual instances. For example, virtual machines (VMs) are virtual instances. In other scenarios, these virtual instances are containers or bare metal servers.

[0122] In Figure 2C, each server uses virtualization technology to obtain multiple virtual machines, each of which can be understood as a resource node. The AI ​​compilation device in Figure 2A can be the data center management platform in Figure 2C.

[0123] Among them, virtual machines can also be called cloud servers (Elastic Compute Service, ECS) or elastic instances (different cloud service providers have different names).

[0124] The data center management platform can provide an access interface (such as a user interface or application programming interface (API)). Tenants can use a client to remotely access the access interface, register an account and password on the data center management platform, and log in to the data center management platform. After the data center management platform successfully authenticates the account and password, the tenant can use the client to send computational graph information to the data center management platform. The data center management platform can then optimize the computational graph, obtain the optimized computational graph, and return the optimized computational graph to the tenant.

[0125] Figure 3 is a schematic diagram of a possible logical structure of an AI compilation device provided in an embodiment of the present application. As shown in Figure 3, the AI ​​compilation device 30 provided in an embodiment of the present application includes: a processor 301, a communication interface 302, a memory 303, and a bus 304. The processor 301, the communication interface 302, and the memory 303 are interconnected via the bus 304. In the embodiment of the present application, the processor 301 is used to control and manage the operations of the AI ​​compilation device 30. For example, the processor 301 is used to perform loop merging to perform operator fusion. The communication interface 302 is used to support the communication of the AI ​​compilation device 30. For example, the communication interface 302 can execute the process of obtaining a computation graph. The memory 303 is used to store the program code and data of the AI ​​compilation device 30.

[0126] The processor 301 may be a central processing unit (CPU), a general-purpose processor (GPOR), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device (PLD), a transistor logic device (TLD), a hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. A processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The bus 304 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, for example. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG. 3 shows only one thick line, but this does not imply that there is only one bus or only one type of bus.

[0127] The above-mentioned AI compilation device can execute the operator fusion process provided by the embodiment of the present application under various possible AI frameworks. Please refer to Figure 4. The AI ​​framework may include: a graph-calculation fusion module 401, a symbolic derivation module 402, an AKG module 403 and a target hardware 404. Among them, the AKG module 403 includes a scheduling module 4031 and a machine code generation module 4032. The AI ​​compilation device may include the AKG module 403, or the AI ​​compilation device includes the symbolic derivation module 402 and the AKG module 403. Of course, the AI ​​compilation device may also include the graph-calculation fusion module 401. It is also possible that the AI ​​compilation device includes the graph-calculation fusion module 401, the symbolic derivation module 402, the AKG module 403 and the target hardware 404.

[0128] The graph-computation fusion module 401 can divide a large computation graph into multiple smaller ones. This division can be based on the shape of the tensors in the computation graph, for example, dividing the portion with a fixed shape into a static computation graph and the portion with a dynamic shape (e.g., a shape of (1024, ?)) into a dynamic computation graph. The static computation graph is then input into the scheduling module 4031 in the AKG module 403. The dynamic computation graph is then input into the symbolic derivation module 402.

[0129] The symbolic derivation module 402 can dynamically calculate the semantic information of operators in the graph to deduce the relationship between different dynamic shapes. Then, the symbolic derivation module 402 transmits the dynamic calculation graph and the relationship between the dynamic shapes of different tensors (such as equality, multiples, size, etc.) in the form of symbolic information to the scheduling module 4031 in the AKG module 403.

[0130] A dynamic computation graph represents the shape of tensors using dynamic variables, while a static computation graph uses constants to represent the shape of tensors. Symbolic inference uses the computation type in the computation graph to infer relationships between the variables used for the shapes of different tensors in the dynamic computation graph, such as equality, multiples, and size.

[0131] The scheduling module 4031 is mainly used to process the loop process of the operators in the calculation graph, which can include the following three steps: loop merging, loop segmentation, and loop expansion. The operator fusion process provided in the embodiment of the present application is mainly the loop merging step. After the loop merging is completed, loop segmentation and loop expansion will be performed. In this way, the parallelism and locality of the program can be improved through loop transformation without changing the correctness of the program, thereby improving the program's running performance. After the loop process is completed, the scheduling module 4031 will pass the processed code to the machine code generation module 4032.

[0132] The machine code generation module 4032 may translate the code of the fusion operator into the machine code of the target hardware, and then transfer the machine code to the target hardware 404 for execution.

[0133] Target hardware 404: runs the fusion operator code to obtain the operator operation result.

[0134] Based on the AI ​​architecture described above, the following describes a method for operator fusion in a computation graph provided by an embodiment of the present application. The method involving the execution of an AI compilation device can be executed by the AI ​​compilation device or by a component of the AI ​​compilation device (e.g., a processor, chip, or chip system).

[0135] As shown in FIG5 , an embodiment of a method for operator fusion in a computation graph provided by an embodiment of the present application includes:

[0136] 501. Obtain a target graph, the target graph including multiple nodes and at least one edge; wherein each edge is used to connect two nodes with a dependency relationship among the multiple nodes, and the node association calculates operators in the graph.

[0137] In this application, the target graph can be a computation graph or a dependency graph obtained by performing relational dependency analysis on the computation graph. If the target graph is a computation graph, then the nodes are operators, and the edges are edges connecting two operators with a dependency relationship. If the target graph is a dependency graph, then the nodes in the dependency graph correspond to the operators in the computation graph, and the edges correspond to the edges connecting the operators in the computation graph.

[0138] 502. Determine a first calculation type and a second calculation type, where the first calculation type is the calculation type of the first node connected to the first edge, and the second calculation type is the calculation type of the second node connected to the first edge. The first edge is included in at least one edge, the first node corresponds to the first operator in the calculation graph, and the second node corresponds to the second operator in the calculation graph. The calculation logic of the first operator and the second operator both include loop statements.

[0139] In this application, there may be various types of calculations, such as: Elemwise operation, broadcast operation, reshape operation, transpose operation, reduce operation, etc. For an introduction to these types of calculations, please refer to the introduction to the technical terminology section above.

[0140] In the present application, the first calculation type and the second calculation type may be the same or different.

[0141] 503. According to the first calculation type and the second calculation type, merge the loop statement of the first operator into the loop statement of the second operator to obtain a fused operator after the first operator and the second operator are fused.

[0142] In the present application, the merging of loop statements of two operators may be a direct merging of loop statements, or a merging after loop transformation.

[0143] In this application, the first calculation type and the second calculation type can be the same calculation type or different calculation types. The loop statement in the operator will have a loop iteration space. The loop iteration space of the first operator and the loop iteration space of the second operator may be the same or different. When the loop iteration spaces are the same, the loop statements can be directly merged.

[0144] The loop iteration space can be understood as a two-dimensional space, which includes each iteration instance of the loop statement within the loop. If each iteration instance is represented by a point, the points of all iteration instances in the entire loop can be constructed into a rectangle. This rectangle can be regarded as a polyhedron in the two-dimensional space. This space is called the loop iteration space of the calculation, as shown in Figure 1B introduced above. If the iteration space of the first operator and the second operator is the same, then the loop statement of the first operator can be directly merged into the loop statement of the second operator. In this way, the code performance of the fused operator can be improved, and the efficiency of operator fusion can be improved.

[0145] In this embodiment of the present application, for two operators with a dependency, the loop statement of one operator can be merged into the loop merge statement of the other operator based on the calculation type of the two operators. This eliminates the need to compress all the loop statements of the operators into one dimension and recalculate the indexes of the operators. Instead, the two operators can be fused based on their calculation types, resulting in a high-performance fused operator.

[0146] Optionally, the above step 503 may include: determining at least one loop transformation method according to the first calculation type and the second calculation type; processing the loop statement of the first operator according to the at least one loop transformation method to transform the loop iteration space of the first operator to overlap with the loop iteration space of the second operator; and merging the loop statement of the first operator into the loop statement of the second operator.

[0147] In this application, if at least one loop transformation method is determined based on the first calculation type and the second calculation type, it means that the loop statement of the first operator needs to be processed accordingly to transform the loop iteration space of the first operator to overlap with the loop iteration space of the second operator, and then merge the loop statements. Therefore, it can be seen that the solution of this application can assist in determining the loop transformation method for the first operator through the calculation type of the two operators, thereby enabling the high-performance fusion of the two operators.

[0148] In the embodiment of the present application, the computation graph in step 501 can be a static computation graph or a dynamic computation graph. Regardless of whether it is a dynamic computation graph or a dynamic computation graph, the following steps as shown in Figure 6 can be executed during the operator fusion process:

[0149] 601. Dependency and Data Flow Analysis.

[0150] The process involves abstracting the read, write, and compute operations in the computational graph into nodes, which are then identified by node numbers. Two nodes with a dependency relationship can be represented as a source node and a target node, with the edge between them as an edge. Each edge can also be represented as a tuple consisting of the source and target node numbers. All edges can then be topologically sorted and used as input to loop merging for subsequent operator fusion.

[0151] This process can be understood by referring to FIG7 , which can also be represented as FIG8 , which includes the node number, the original statement in the node, the node type, and the edge.

[0152] Node number, for example, in the first line "Node:0:%0=affine.load%arg2[%arg6,0]:memref<3185x1xf32>", 0 is the node number, and different node numbers are unique. There are five nodes in Figure 7, namely node 0, node 1, node 2, node 3, and node 4.

[0153] The original statement in the node, for example, the first line "%0=affine.load%arg2[%arg6,0]:memref<3185x1xf32>" is the original statement of the node. There are four types of statements:

[0154] Memory loading statement: such as affine.load in the first line;

[0155] Memory storage statement: such as affine.store on line 17;

[0156] Calculation statements: such as arith.mulf in line 5;

[0157] Loop statements: For example, in line 13 of affine.for, the original statements contained in the loop are in lines 14-17. These statements have been previously assigned node numbers (line 14 is Node0, line 15 is Node1, line 16 is Node2, and line 17 is Node3). These can be obtained by checking whether the original statements in the nodes are equal.

[0158] Node type: The statement type of the original statement in the node indicates that both memory load statements and memory store statements are classified as memory access statements. The calculation statement type can be set by querying Table 1 below. The loop statement type is the loop type.

[0159] In this application, the calculation types that may be involved in the calculation graph or dependency graph can be mapped and converted with the calculation types to be used later in this application. The mapping relationship can be understood by referring to Table 1.

[0160] Table 1: Calculation type mapping table

[0161] The possible computation types involved in the target graph include the various possible types listed in the first column of Table 1, such as: Add, And, Cast, Ceil, Cos, Div, Exp, Greater, Less, Log, Mul, Not, Reciprocal, Relu, Sub, Sqrt, and Tanh. These possible computation types can all be mapped to the Elemwise computation type. The Add, Div, Mul, and Sub in the third row of the first column can also be mapped to the Broadcast computation type.

[0162] In the second and third rows of the first column, the Elemwise and Broadcast type mappings overlap because Broadcast itself is a special case of Elemwise calculation. When encountering overlapping network calculation types, the classification standard is to judge the shape of the input tensor of the calculation. If the shape of the input tensor is all static and equal constants, it will be mapped to the Elemwise type; if the shape of the input tensor contains dynamic or static and unequal constants, it will be mapped to the Broadcast type.

[0163] The calculation type after mapping Reshape in the fourth row of the first column is also Reshape, and the calculation type after mapping Transpose in the fifth row of the first column is also Transpose. The sum (ReduceSum), maximum (ReduceMax), and minimum (ReduceMin) in the sixth row of the first column can be mapped to the calculation type Reduce.

[0164] In Figure 7, an edge, such as "OutEdge: 2" in the second row, represents an edge between Node0 and Node2, from Node0 to Node2. This means that Node2 is the target node and Node0 is the source node. Therefore, the edge can be further represented by a tuple of "(source node number, target node number)." That is, in Figure 8, the edge between Node0 and Node2 can be represented by (0, 2). In Figures 7 and 8, there are three edges: (0, 1), (1, 2), and (2, 3). Node 4 represents a loop, starting with nodes 0 and 1.

[0165] The three edges can be sorted in topological order: (0, 1), (1, 2) and (2, 3), and the sorted edges can be used as inputs of loop merging for subsequent operator fusion processing.

[0166] 602. Determine a first calculation type of a first node and a calculation type of a second node associated with a first edge.

[0167] The first edge can be any edge after sorting. Usually the sorted edges are traversed in reverse order, that is, starting from the last edge after sorting.

[0168] The first node and the second node associated with the edge can be determined by the edge, and then the corresponding calculation type can be determined by the above Table 1. This process can determine the first calculation type of the first operator mapped to the first node and the second calculation type of the second operator mapped to the second node.

[0169] In the embodiment of the present application, the first node can be understood as the source node, and the second node can be understood as the target node. Of course, the first node can also be the target node and the second node can be the source node, and the present application does not limit this.

[0170] 603. Determine whether the first computing type and the computing type of the second node are supported types. If so, execute step 604; if not, execute step 608.

[0171] 604. Determine a corresponding cyclic transformation method according to the first calculation type and the calculation type of the second node.

[0172] Regarding the cyclic transformation method in this application, please refer to Table 2 for understanding.

[0173] Table 2: Cyclic transformation method table

[0174] Thus, step 604 can determine the calculation type and cyclic transformation mode by searching a combination table of calculation types and cyclic transformation modes. The combination table includes multiple calculation types in the first row, multiple calculation types in the first column, and the cyclic transformation mode included in the intersection of each calculation type in the first row and each calculation type in the first column. This combination table can be understood by referring to Table 3 below.

[0175] Table 3: Combination table of calculation type and loop transformation method

[0176] The first row represents the six calculation types of the first operator, and the first column represents the six calculation types of the second operator. Indicates circular replication, represents a cyclic exchange, Indicates loop segmentation, Indicates cycle compression, Indicates loop backtracking. [] is an empty set, indicating that the loop iteration spaces of the first and second operators overlap. When the crossbar is blank, it indicates that the calculation type of the first and second operators cannot be Reshape and Transpose at the same time.

[0177] As can be seen from Table 3 above, if the first calculation type of the first operator is Elemwise and the second calculation type of the second operator is also Elemwise, the information in the intersection grid of the two is [], which is an empty set, indicating that the loop iteration spaces of the first operator and the second operator overlap. In this case, the loop statement of the first operator can be directly merged into the loop statement of the second operator.

[0178] If the first calculation type of the first operator is Reshape and the second calculation type of the second operator is Elemwise, then the information in the intersection of the two is It is necessary to first perform loop replication on the first operator, then perform loop segmentation, and then perform loop compaction so that the loop iteration space of the first operator overlaps with the loop iteration space of the second operator, and then merge the loop statement of the first operator into the loop statement of the second operator.

[0179] Table 3 also includes the calculation type Shuffle, which is a combination type. The Shuffle type is an abstract definition of an ordered list of a series of calculation types. By viewing the ordered list saved in the Shuffle type, the historical type of the current calculation can be restored. When the first calculation type of the first operator and the second calculation type of the second operator are both Shuffle, a loop backtracking operation needs to be performed on the first operator.

[0180] Backtracking is an abstract definition of an ordered list of loop transformations. By viewing the ordered list saved in the BackTracking loop transformation, the historical loop transformations of the current calculation can be restored.

[0181] 605. Determine a loop control parameter corresponding to the loop transformation method according to the loop transformation method, where the loop control parameter is used to control the loop statement for processing the first operator.

[0182] In the embodiments of the present application, the derived loop transformation methods can be sequentially applied to the loop nests to which the source node belongs. For example, if the determined loop transformation methods include loop duplication, loop swapping, loop compression, and loop segmentation, the loop control parameters corresponding to each loop transformation method, i.e., loop input parameters, can be first determined. The loop input parameters for different loop transformation methods are different and are described below.

[0183] The loop input parameters of loop copying include: the number of copied loops, the loop insertion position, the upper bound of the copied loop (that is, the starting position of the loop), the lower bound (that is, the ending position of the loop), and the step size (which can be 1).

[0184] The loop input parameters of loop swap include: the positions of the two loops that need to be swapped.

[0185] The cycle input parameters of cycle compaction include: the positions of several cycles that need to be compacted.

[0186] The loop input parameters of loop segmentation include: the position of the loop to be segmented, the number of segments to be segmented, the upper and lower bounds of the loop for each segment after segmentation, and the step size.

[0187] In the embodiment of the present application, the method of determining the loop input parameters of the loop transformation method according to the node type is as follows:

[0188] Loop replication: Check the loop nest containing the loop variable accessing the tensor at the source node of the edge and the loop nest containing the loop variable accessing the tensor at the target node of the edge, and calculate the difference between the two. The size of the difference is the number of loops to be replicated, the position of the difference is the position where the loop is inserted, and the upper bound, lower bound, and step size of the loop of the difference are the upper bound, lower bound, and step size of the loop to be replicated.

[0189] Loop swapping: Based on the semantic information of the transpose operator, the axes to be swapped in the transpose operation are the positions of the two loops to be swapped. Based on the semantic information of the reduce operator, when the loop containing the reduce axis is outside the non-reduce axis, the positions of the reduce axis and the non-reduce axis are swapped until the loop containing the reduce axis is in the innermost layer of the loop nesting. If there is a multi-dimensional reduce operation, the tensors read by the reduce operation are sorted and swapped according to their dimensions.

[0190] Loop compaction: Based on the semantic information of the reshape operator, the difference between the loops corresponding to the dimensions of the tensor read by the reshape operator and the dimensions of the tensor written by the reshape operator is the location of the loop that needs to be compacted.

[0191] Loop segmentation: According to the semantic information of the reshape operator, the difference between the loops corresponding to the dimensions of the tensor read by the reshape operator and the dimensions of the tensor written by the reshape operator is the position of the loop that needs to be segmented. The number of segments is the size of the difference. After segmentation, the upper bound, lower bound, and step size of the loop of each segment are the loops of the tensor dimensions written by the reshape operator.

[0192] 606. Determine the calculation type of the fusion operator according to the first calculation type and the second calculation type.

[0193] This step can be determined using a calculation type derivation table. The first row of the calculation type derivation table includes multiple calculation types, the first column includes multiple calculation types, and the intersection of each calculation type in the first row and each calculation type in the first column contains at least one calculation type. The at least one calculation type is a calculation type of a fusion operator. For more information about this calculation type derivation table, refer to Table 4.

[0194] Table 4: Calculation type derivation table

[0195] The first row represents the six calculation types of the first operator, the first column represents the six calculation types of the second operator, and the calculation type at the intersection position is the calculation type of the fusion operator.

[0196] As can be seen from Table 4 above, the calculation type of the fusion operator can be quickly determined through the first calculation type and the second calculation type by looking up the table, which is conducive to further optimization of the fusion operator.

[0197] 607. Update the target node information.

[0198] In the embodiment of the present application, if the source node is cyclically merged into the target node, the target node information needs to be updated so that the next cyclic process can proceed correctly. The process of updating the target node information may include:

[0199] Update the current calculation type of the target node to the calculation type of the fusion operator. Add the calculation type of the source node to the historical fusion calculation type list of the target node. Add the cyclic transformation method used for the source node in step 605 to the list of cyclic transformation methods used by the source node. Add the source node number to the historical fusion node number list of the target node. Set all edges with the source node as the target node to the edge with the fusion operator as the target node.

[0200] In addition, if the calculation type of the source or target node is Shuffle, the calculation type of the fused node (target node) needs to be added to the Shuffle type and saved at the end of the Shuffle type's ordered list. At the same time, the updated cyclic transformation type of the target node needs to be added to the BackTracking type and saved at the end of the BackTracking type's ordered list.

[0201] When the target node type is Shuffle type, the ordered list stored in the Shuffle type is traversed from the beginning to the end. The traversed calculation type is used as the target node type, and is combined with the source node type to form a tuple. The tuple is added to the historical fusion calculation type list. After the traversal is completed, these calculation types are deleted from the Shuffle type ordered list.

[0202] When the cyclic transformation mode is the BackTracking type, the ordered list stored in the BackTracking type is traversed from the beginning to the end, and the traversed cyclic transformation mode is added to the cyclic transformation combination list used by the source node. After completion, the cyclic transformation is deleted from the ordered list of the BackTracking type.

[0203] 608. Mark the first edge as processed.

[0204] 609. Is the number of unprocessed edges greater than 0? If so, select the second edge and execute step 602; if not, execute step 610.

[0205] The process of performing the loop merging process on the two nodes associated with the second edge may be executed starting from step 602 , except that the first edge is replaced by the second edge.

[0206] 610. End the loop merging process.

[0207] After the loop merging process is completed, you can refer to Figure 4 above to perform loop segmentation and subsequent processes.

[0208] As can be seen from the embodiment introduced in Figure 6 above, the solution provided in the embodiment of the present application can obtain a high-performance fusion operator by cyclically merging the two nodes associated with the edge. Moreover, the above-mentioned cyclic merging process can be executed multiple times, further improving the performance of the operator and optimizing the calculation graph.

[0209] Because the operator fusion solution provided in the embodiment of the present application can be applied to static computation graphs and dynamic computation graphs, it can be seen from the introduction of the above Figure 4 that support for loop merging of dynamic shapes and implicit broadcasts can be achieved through symbolic deduction.

[0210] In an embodiment of the present application, symbolic deduction can be performed on the dynamic computation graph to determine the relationship between the first variable and the second variable in the dynamic computation graph; wherein the relationship between the first variable and the second variable is used to indicate a loop merging method or to indicate the relationship between input data and output data of operators of different computation types, the first variable is a variable used to describe the shape of the first tensor, the second variable is a variable used to describe the shape of the second tensor, and the first tensor and the second tensor are different tensors in the computation graph.

[0211] Symbolic derivation can be used to obtain the dynamic relationship between the upper and lower bounds of different loops, eliminating the need to distinguish between dynamic and static shape scenarios. The specific symbolic derivation process can be:

[0212] Step 1: Use symbolic derivation results and data flow analysis to establish shape constraints;

[0213] In this step, the result of symbolic deduction is to assign a unique identifier to each unknown shape information; the result of data flow analysis is the size relationship between different calculations and data storage; therefore, based on these analysis results, the following constraints can be obtained:

[0214] 1.1, Equality:

[0215] Elemwise type: For a calculation that does not change the input and output shapes (for example, taking the square (Sqrt)), it can be known that the shapes of the input data and output data are equal, so the input symbol can be marked as equal to the output symbol.

[0216] Transpose type: According to the computational semantics of transpose, it does not change the shape size, but it rearranges it. That is, [shape1, shape2] becomes [shape2, shape1] after transpose. Therefore, it can be indicated that the first dimension of the output dimension after transpose is equal to the second dimension of the input shape; the second dimension of the output dimension is equal to the first dimension of the input shape.

[0217] 1.2. Greater than / less than: For example, for implicit broadcast calculations, according to the previous description, the output shape of the calculation is greater than or equal to the input shape. Therefore, it can be indicated that the output shape symbol A is greater than or equal to the input shape symbol B.

[0218] Step (2): perform loop merging based on symbol information;

[0219] For the three situations in step (1), perform cyclic merging respectively;

[0220] 2.1. Equality: Merge the loops containing two equal symbols. You can merge them directly, for example:

[0221] The above is an example of transpose. The symbols i1 and i2 are equal, and j1 and j2 are equal. The loops can be directly merged and the i2 and j2 indexes can be used uniformly.

[0222] 2.2. Greater than / less than: Merge the loops of two symbols with a size relationship. You can merge the statements in the loop of the smaller symbol into the loop of the larger symbol. For example:

[0223] The above is an example of implicit broadcasting. When the symbol j is greater than or equal to i, loop i is merged into loop j, and a bounds check statement is added before the index. Since this scenario only occurs during broadcast calculation, according to the semantic description of broadcast, the symbol i can be either equal to j or equal to 1, so j or 0 is used for indexing.

[0224] From the above introduction, it can be seen that the solution provided by the embodiment of the present application supports the loop merging of dynamic shape operators, and supports the fusion of implicit broadcast through the 2.2 cases of step (2), where the conditional branch of i == j is the non-broadcast fusion code, and the else conditional branch is the broadcast fusion code.

[0225] Furthermore, a source node in a computational graph may correspond to multiple target nodes. In this case, multiple outgoing edges can be processed for the source node. This process can include: determining the second node with the highest priority among the multiple target nodes; and performing dependency inversion on the target nodes other than the second node with the source node, so that the first node becomes the target node in its dependency relationship with the target nodes other than the second node. In other words, when there are multiple target nodes, dependency inversion can be used to determine a unique target node for the source node, thereby achieving efficient operator fusion.

[0226] The process of dependency inversion can include:

[0227] Step 1: Prioritize multiple target nodes of multiple output edges according to the following criteria:

[0228] It can be sorted by calculation type: the order of different calculation priorities from high to low is as follows: [Shuffle, Reduce, Transpose, Reshape, Broadcast, Elemwise].

[0229] When the calculation types are equal, the nodes are sorted from large to small according to their node numbers. Since the numbers are unique, the sorting ends here.

[0230] Step (2): Based on the sorting results of step (1), keep the node with the highest priority as the target node, and perform dependency inversion on the remaining nodes: swap the source and target nodes of the edge, for example:

[0231] If there is an edge [(S,D1),(S,D2),(S,D3)] between the source node S and the target nodes [D1,D2,D3] (sorted by priority), then keep the edge between S and D1 unchanged and reverse the edges of the remaining nodes, so that the final edge is [(S,D1),(D2,S),(D3,S)].

[0232] After dependency inversion, the process of performing operator fusion based on edges as described in the embodiment corresponding to FIG6 is executed.

[0233] The above describes the method of operator fusion in the computation graph. The following describes the computer device provided in the embodiment of the present application with reference to the accompanying drawings.

[0234] As shown in FIG9 , a computer device 90 provided in an embodiment of the present application includes:

[0235] An acquisition unit 901 is configured to acquire a target graph, the target graph including a plurality of nodes and at least one edge; wherein each edge is used to connect two nodes having a dependency relationship among the plurality of nodes, and the nodes are associated with operators in the computation graph;

[0236] A first processing unit 902 is configured to determine a first calculation type and a second calculation type, where the first calculation type is a calculation type of a first node connected to a first edge, and the second calculation type is a calculation type of a second node connected to the first edge, the first edge is included in at least one edge, the first node corresponds to a first operator in a calculation graph, the second node corresponds to a second operator in the calculation graph, and the calculation logic of the first operator and the second operator both include loop statements;

[0237] The second processing unit 903 is configured to merge the loop statement of the first operator into the loop statement of the second operator according to the first calculation type and the second calculation type, so as to obtain a fused operator obtained by fusing the first operator and the second operator.

[0238] The computer device provided in the embodiments of the present application can, for two operators with a dependency, merge the loop statement of one operator into the loop merging statement of the other operator based on the computation types of the two operators. This eliminates the need to compress all loop statements of the operators into one dimension and recalculate the operator indexes. Instead, the two operators can be fused based on their computation types, resulting in a high-performance fused operator.

[0239] Optionally, if it is determined by searching the combination table based on the first calculation type and the second calculation type that the loop iteration space of the first operator overlaps with the loop iteration space of the second operator, the loop statement of the first operator can be directly merged into the loop statement of the second operator.

[0240] Optionally, the second processing unit 903 is specifically used to determine at least one loop transformation method based on the first calculation type and the second calculation type; process the loop statement of the first operator according to the at least one loop transformation method to transform the loop iteration space of the first operator to overlap with the loop iteration space of the second operator; and merge the loop statement of the first operator into the loop statement of the second operator.

[0241] Optionally, the at least one cyclic transformation mode is obtained by searching a combination table according to the first calculation type and the second calculation type, and the combination table includes a correspondence between a combination of the first calculation type and the second calculation type and the at least one cyclic transformation mode.

[0242] Optionally, the combination table is:

[0243] The first row represents the six calculation types of the first operator, and the first column represents the six calculation types of the second operator. Indicates circular replication, represents a cyclic exchange, Indicates loop segmentation, Indicates cycle compression, Indicates loop backtracking. [] is an empty set, indicating that the loop iteration spaces of the first and second operators overlap. When the crossbar is blank, it indicates that the calculation type of the first and second operators cannot be Reshape and Transpose at the same time.

[0244] Optionally, the second processing unit 903 is further configured to determine a calculation type of the fusion operator according to the first calculation type and the second calculation type.

[0245] Optionally, the calculation type of the fusion operator is obtained by searching a calculation type derivation table according to the first calculation type and the second calculation type, and the calculation type derivation table includes a correspondence between a combination of the first calculation type and the second calculation type and the calculation type of the fusion operator.

[0246] Optionally, the calculation type deduction table is:

[0247] The first row represents the six calculation types of the first operator, the first column represents the six calculation types of the second operator, and the calculation type at the intersection position is the calculation type of the fusion operator.

[0248] Optionally, the computation graph is a dynamic computation graph or a static computation graph. The shape of the tensor in the dynamic computation graph is described by variables; the shape of the tensor in the static computation graph is described by constants.

[0249] Optionally, the first processing unit 902 is further used to perform symbolic deduction on the dynamic computation graph to determine the relationship between the first variable and the second variable in the dynamic computation graph; wherein the relationship between the first variable and the second variable is used to indicate a loop merging method or to indicate the relationship between input data and output data of operators of different computation types, the first variable is a variable used to describe the shape of the first tensor, the second variable is a variable used to describe the shape of the second tensor, and the first tensor and the second tensor are different tensors in the dynamic computation graph.

[0250] Optionally, when the first node serves as a source node and there are multiple corresponding target nodes, the first processing unit 902 is also used to: determine the second node with the highest priority among the multiple target nodes; and perform dependency inversion on the target nodes other than the second node among the multiple target nodes and the source node, so that the first node becomes the target node in the dependency relationship between the first node and the target nodes other than the second node.

[0251] Optionally, the second processing unit 903 is further configured to determine, according to the loop transformation mode, a loop control parameter corresponding to the loop transformation mode, where the loop control parameter is used to control a loop statement for processing the first operator.

[0252] Optionally, the second processing unit 903 is further configured to update the information of the second operator to information of the fusion operator, where the information of the fusion operator includes a calculation type of the fusion operator.

[0253] Optionally, the second processing unit 903 is further configured to perform a cyclic merging process on two nodes associated with the second edge when there are multiple edges.

[0254] Optionally, the target graph is a computation graph, the first node is a first operator, the second node is a second operator, and the first edge is an edge used to connect the first operator and the second operator.

[0255] Optionally, the target graph is a dependency graph obtained by performing dependency analysis on the computation graph.

[0256] The functions of the various units of the computer device 90 introduced above can be understood by referring to the introduction of the previous method embodiment part, and will not be repeated here.

[0257] The computer device or computer equipment provided in the embodiment of the present application may also be the AI ​​compilation device introduced in FIG. 3 above.

[0258] In another embodiment of the present application, a computer-readable storage medium is further provided, in which computer-executable instructions are stored. When the processor of the computer device executes the computer-executable instructions, the computer device executes the steps executed by the computer device in Figures 5 to 8 above.

[0259] In another embodiment of the present application, a computer program product is provided. The computer program product includes computer program code. When the computer program code is executed on a computer, the computer device executes the steps executed by the computer device in Figures 5 to 8 above.

[0260] In another embodiment of the present application, a chip system is also provided, which includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected by lines; the interface circuits are used to receive signals from the memory of the computer device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the computer device executes the steps performed by the computer device in Figures 5 to 8 above. In one possible design, the chip system may also include a memory, which is used to store program instructions and data necessary for the control device. The chip system can be composed of chips, or it can include chips and other discrete devices.

[0261] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0262] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0263] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in whole or in part through software, hardware, firmware, or any combination thereof.

[0264] When software is used to implement the integrated unit, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

Claims

1. A method for operator fusion in a computation graph, characterized in that: include: Obtain a target graph, the target graph comprising a plurality of nodes and at least one edge; wherein each edge is used to connect two nodes having a dependency relationship among the plurality of nodes, and the nodes are associated with operators in the computation graph; Determining a first calculation type and a second calculation type, where the first calculation type is a calculation type of a first node connected to a first edge, and the second calculation type is a calculation type of a second node connected to the first edge, the first edge is included in the at least one edge, the first node corresponds to a first operator in the calculation graph, the second node corresponds to a second operator in the calculation graph, and the calculation logic of the first operator and the second operator both include loop statements; According to the first calculation type and the second calculation type, the loop statement of the first operator is merged into the loop statement of the second operator to obtain a fused operator obtained by fusing the first operator and the second operator.

2. The method according to claim 1, characterized in that The merging of the loop statement of the first operator into the loop statement of the second operator according to the first calculation type and the second calculation type includes: determining at least one cyclic transformation mode according to the first calculation type and the second calculation type; Processing the loop statement of the first operator according to the at least one loop transformation manner to transform the loop iteration space of the first operator to overlap with the loop iteration space of the second operator; Merge the loop statement of the first operator into the loop statement of the second operator.

3. The method according to claim 2, characterized in that The at least one cyclic transformation mode is obtained by searching a combination table according to the first calculation type and the second calculation type, and the combination table contains a correspondence between a combination of the first calculation type and the second calculation type and the at least one cyclic transformation mode.

4. The method according to claim 3, characterized in that The combination table is: The first row represents the six calculation types of the first operator, and the first column represents the six calculation types of the second operator. Indicates circular replication, represents a cyclic exchange, Indicates loop segmentation, Indicates cycle compression, Indicates loop backtracking. [] is an empty set, indicating that the loop iteration spaces of the first and second operators overlap. When the crossbar is blank, it indicates that the calculation type of the first and second operators cannot be Reshape and Transpose at the same time.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The calculation type of the fusion operator is determined according to the first calculation type and the second calculation type.

6. The method according to claim 5, characterized in that The calculation type of the fusion operator is obtained by searching a calculation type deduction table according to the first calculation type and the second calculation type. The calculation type deduction table includes a correspondence between a combination of the first calculation type and the second calculation type and the calculation type of the fusion operator.

7. The method according to claim 6, characterized in that The calculation type derivation table is: The first row represents the six calculation types of the first operator, the first column represents the six calculation types of the second operator, and the calculation type at the intersection position is the calculation type of the fusion operator.

8. The method according to any one of claims 1 to 7, characterized in that The computation graph is a dynamic computation graph or a static computation graph. The shape of the tensor in the dynamic computation graph is described by variables; the shape of the tensor in the static computation graph is described by constants.

9. The method according to claim 8, characterized in that The computation graph is a dynamic computation graph, and the method further includes: Symbolic deduction is performed on the dynamic computation graph to determine a relationship between a first variable and a second variable in the dynamic computation graph; wherein the relationship between the first variable and the second variable is used to indicate a loop merging method or to indicate a relationship between input data and output data of operators of different computation types, the first variable is a variable used to describe a shape of a first tensor, the second variable is a variable used to describe a shape of a second tensor, and the first tensor and the second tensor are different tensors in the dynamic computation graph.

10. The method according to claim 9, characterized in that The method is applied to implicit broadcast scenarios.

11. The method according to any one of claims 1 to 10, characterized in that When the first node serves as a source node and there are multiple corresponding target nodes, the method further includes: Determine the second node with the highest priority among the multiple target nodes; Dependency inversion is performed on target nodes other than the second node among the multiple target nodes and the source node, so that the first node becomes a target node in the dependency relationship between the first node and the target nodes other than the second node.

12. The method according to any one of claims 2 to 4, characterized in that: The method further comprises: According to the loop transformation mode, a loop control parameter corresponding to the loop transformation mode is determined, and the loop control parameter is used to control a loop statement for processing the first operator.

13. The method according to any one of claims 5 to 7, characterized in that: The method further comprises: The information of the second operator is updated to the information of the fusion operator, where the information of the fusion operator includes the calculation type of the fusion operator.

14. The method according to claim 13, characterized in that When there are multiple edges, the method further includes: Perform a loop merge on the two nodes associated with the second edge.

15. The method according to any one of claims 1 to 14, characterized in that The target graph is the computation graph, the first node is the first operator, the second node is the second operator, and the first edge is an edge used to connect the first operator and the second operator.

16. The method according to any one of claims 1 to 14, characterized in that The target graph is a dependency graph obtained by performing dependency analysis on the computation graph.

17. A computer device comprising: An acquisition unit, configured to acquire a target graph, the target graph comprising a plurality of nodes and at least one edge; wherein each edge is used to connect two nodes having a dependency relationship among the plurality of nodes, and the nodes are associated with operators in the computation graph; a first processing unit, configured to determine a first calculation type and a second calculation type, wherein the first calculation type is a calculation type of a first node connected to a first edge, and the second calculation type is a calculation type of a second node connected to the first edge, the first edge is included in the at least one edge, the first node corresponds to a first operator in the calculation graph, the second node corresponds to a second operator in the calculation graph, and the calculation logic of the first operator and the second operator both include loop statements; The second processing unit is used to merge the loop statement of the first operator into the loop statement of the second operator according to the first calculation type and the second calculation type, so as to obtain a fused operator after the first operator and the second operator are fused.

18. A computing device, characterized in that comprising a processor and a computer-readable storage medium storing a computer program; The processor is coupled to the computer-readable storage medium, and when the computer program is executed by the processor, the method according to any one of claims 1 to 16 is implemented.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 16 is implemented.

20. A computer program product, characterized in that The computer program product comprises a computer program code, which, when executed on a computer device, causes the computer device to perform the method according to any one of claims 1 to 16.

21. A chip system, characterized in that: The method comprises a processor, wherein the processor is called to execute the method according to any one of claims 1 to 16.

Images