Computational graph optimization method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a calculation graph optimization method and device, electronic equipment and a storage medium. The method comprises the steps that a first calculation graph is obtained, and a first operator corresponding to each first node is pre-configured with at least two candidate kernel functions matched with different types of hardware calculation cores; determining a target kernel function container from a kernel function container list corresponding to the first operator according to the attribute information of the first operator; according to the target kernel function container and the attribute information corresponding to each first operator, determining whether to execute a local graph optimization operation on a first node corresponding to the first operator; for the to-be-optimized nodes which are determined to execute the local graph optimization operation, executing the local graph optimization operation according to the corresponding target kernel function container to obtain a second calculation graph; and according to the second calculation graph, generating an executable code corresponding to the first calculation graph. According to the method, optimal balance can be achieved between generalization and extreme performance, redundant calculation overhead is prevented from being introduced, and high flexibility is achieved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of artificial intelligence technology, and in particular, to a computation graph optimization method and device, an electronic device, and a storage medium. BACKGROUND

[0002] In the field of artificial intelligence (AI) technology, the processing efficiency of an AI framework on a computation graph directly affects the model execution performance. Current mainstream AI frameworks generally adopt a processing flow of “global graph optimization first, then operator selection”. This flow covers three stages of graph optimization, operator selection, and execution in sequence. In the graph optimization stage, predefined general optimization operations such as constant folding, operator fusion, and layout transformation are performed on the input computation graph. In the operator selection stage, specific kernel functions are matched for each operator in the computation graph according to the shape, layout, and other characteristic information of the input data (such as tensors). In the execution stage, the final executable code is generated based on the selected kernel functions, and the computation graph structure is fixed and cannot be modified again.

[0003] In the AI framework, high-performance kernel functions (such as Tensor Core accelerated convolution) have strict constraints on the shape, layout, and other characteristics of the input data, and need to be solved by inserting specific operators (or auxiliary nodes encapsulating specific operators). However, such optimization operations only have gain effects for specific kernel functions. Due to the processing flow of “global graph optimization first, then operator selection”, the AI framework cannot predict the selected kernel functions in the subsequent graph optimization stage. That is, performing graph optimization operations in advance will introduce redundant overhead for general kernel function scenarios. Moreover, after the kernel function is determined, the computation graph structure cannot be modified again to supplement targeted optimization. This makes it difficult for the AI framework to achieve an optimal balance between generalization and extreme performance. SUMMARY

[0004] The method comprises: obtaining a first computation graph, wherein the first computation graph comprises at least one first node, each first node corresponds to a first operator, each first operator is preconfigured with at least two candidate kernel functions, and the at least two candidate kernel functions are respectively adapted to different types of hardware computing cores; for each first operator, determining a kernel function container list corresponding to the first operator according to attribute information of the first operator, and determining a target kernel function container from the kernel function container list corresponding to the first operator, wherein the kernel function container list comprises at least two candidate kernel function containers, each candidate kernel function container comprises at least one candidate kernel function preconfigured for the first operator, and the target kernel function container is a candidate kernel function container with the lowest execution time in the at least two candidate kernel function containers; determining whether to perform a local graph optimization operation on the first node corresponding to the first operator according to the target kernel function container corresponding to each first operator and the attribute information; for a to-be-optimized node determined to perform the local graph optimization operation, performing the local graph optimization operation according to the target kernel function container corresponding to the to-be-optimized node to obtain a second computation graph; and generating executable code corresponding to the first computation graph according to the second computation graph.

[0005] For example, in the method provided in at least one embodiment of the present disclosure, in response to the hardware computing core adapted to a candidate kernel function having a hardware running constraint, the candidate kernel function container further comprises a candidate optimization list corresponding to the candidate kernel function, and the candidate optimization list comprises at least one operator for satisfying the hardware running constraint of the candidate kernel function.

[0006] For example, in the method provided in at least one embodiment of the present disclosure, the attribute information of the first operator at least comprises: an operator type of the first operator and shape information of input data of the first operator, wherein the shape information comprises at least one of a shape size of the input data and an arrangement format of the input data; determining the kernel function container list corresponding to the first operator according to the attribute information of the first operator, and determining the target kernel function container from the kernel function container list corresponding to the first operator comprises: determining a first kernel function container list corresponding to the first operator according to the operator type of the first operator; for each candidate kernel function container in the first kernel function container list, obtaining an execution time of the candidate kernel function container according to at least the shape information of the input data of the first operator and the candidate kernel function in the candidate kernel function container; and taking the candidate kernel function container with the lowest execution time as the target kernel function container of the first operator.

[0007] For example, in the method provided in at least one embodiment of the present disclosure, the execution time of the candidate kernel function container comprises a time required for the adapted hardware computing core to execute the candidate kernel function in the candidate kernel function container according to the attribute information of the first operator.

[0008] For example, in the method provided by at least one embodiment of the present disclosure, in response to the candidate kernel function container further comprising a candidate optimization list corresponding to the candidate kernel function, and the shape information of the input data of the first operator not satisfying the hardware running constraint of the adaptive hardware computing core, the execution time consumption of the candidate kernel function container further comprises: execution time of at least one operator in the candidate optimization list.

[0009] For example, in the method provided by at least one embodiment of the present disclosure, each candidate kernel function container further comprises a cost estimation function, and the cost estimation function is configured to determine the execution time consumption of the corresponding candidate kernel function container. For each candidate kernel function container in the first kernel function container list, the execution time consumption of the candidate kernel function container is obtained according to at least the shape information of the input data of the first operator and the candidate kernel function in the candidate kernel function container, comprising: inputting the shape information into the cost estimation function included in each candidate kernel function container in the first kernel function container list to obtain the execution time consumption of each candidate kernel function container in the first kernel function container list.

[0010] For example, in the method provided by at least one embodiment of the present disclosure, in response to the candidate kernel function container further comprising a candidate optimization list corresponding to the candidate kernel function, the cost estimation function is further configured according to the candidate optimization list.

[0011] For example, in the method provided by at least one embodiment of the present disclosure, the attribute information of the first operator at least comprises the shape information of the input data of the first operator, the hardware running constraint comprises a constraint condition for the shape information, and the determination of whether to perform the local graph optimization operation on the first node corresponding to the first operator comprises: in response to the target kernel function container not comprising the candidate optimization list, determining not to perform the local graph optimization operation on the first node, in response to the target kernel function container comprising the candidate optimization list and the shape information satisfying the constraint condition, determining not to perform the local graph optimization operation on the first node, and in response to the target kernel function container comprising the candidate optimization list and the shape information not satisfying the constraint condition, determining to perform the local graph optimization operation on the first node and taking the first node as the to-be-optimized node.

[0012] For example, in the method provided by at least one embodiment of the present disclosure, for the to-be-optimized node determined to perform the local graph optimization operation, the local graph optimization operation is performed according to the target kernel function container corresponding to the to-be-optimized node to obtain a second computing graph, comprising: for each to-be-optimized node, at least one auxiliary node is added at a corresponding position adjacent to the to-be-optimized node in the first computing graph according to at least one operator included in the candidate optimization list for satisfying the hardware running constraint of one candidate kernel function, to obtain the second computing graph.

[0013] For example, the method provided in at least one embodiment of the present disclosure further includes: binding the first operator with a target kernel function container; and generating executable code corresponding to the first calculation graph according to the second calculation graph includes: traversing each node in the second calculation graph, in response to the current operator corresponding to the current node being a non-first operator, selecting a kernel function for the current operator according to the attribute information of the current operator; in response to the current operator corresponding to the current node being a first operator of any operator type, taking the candidate kernel function in the target kernel function container bound by the first operator of any operator type as the kernel function of the current operator; and generating executable code corresponding to the first calculation graph according to the kernel functions of the operators corresponding to each node in the second calculation graph.

[0014] For example, in the method provided in at least one embodiment of the present disclosure, obtaining the first calculation graph includes: obtaining an original calculation graph; and performing a general graph optimization operation on the second node in the original calculation graph to obtain the first calculation graph, wherein the operator corresponding to the second node is a non-first operator.

[0015] For example, the method provided in at least one embodiment of the present disclosure further includes: respectively configuring a corresponding kernel function container list for the first operators of different operator types.

[0016] For example, in the method provided in at least one embodiment of the present disclosure, respectively configuring a corresponding kernel function container list for the first operators of different operator types includes: for any operator type in the first operators: constructing N candidate kernel function containers in the kernel function container list corresponding to the any operator type, wherein N is an integer greater than or equal to 2, and the N candidate kernel function containers correspond to different hardware computing cores; encapsulating N candidate kernel functions respectively implemented by different hardware computing cores and pre-configured for the operators of the any operator type into the N candidate kernel function containers; and for any one of the N candidate kernel functions, in response to the hardware computing core implementing the any one candidate kernel function having a hardware running constraint, configuring a candidate optimization list associated with the any one candidate kernel function in the corresponding candidate kernel function container, wherein the associated candidate optimization list includes at least one operator for satisfying the hardware running constraint.

[0017] For example, the method provided in at least one embodiment of the present disclosure further includes: constructing a corresponding cost estimation function for each candidate kernel function container, wherein the cost estimation function is constructed according to the candidate kernel function encapsulated in each candidate kernel function container, and in response to the candidate kernel function container further including a candidate optimization list, the cost estimation function is further constructed according to the candidate optimization list.

[0018] The at least one embodiment of the present disclosure further provides a computing graph optimization apparatus, which comprises an obtaining module, an inquiring module, an optimizing module and a code generating module. The obtaining module is configured to obtain a first computing graph, wherein the first computing graph comprises at least one first node, each first node corresponds to a first operator, each first operator is pre-configured with at least two candidate kernel functions, and the at least two candidate kernel functions are respectively adapted to different types of hardware computing cores. The inquiring module is configured to, for each first operator, determine a kernel function container list corresponding to the first operator according to attribute information of the first operator, and determine a target kernel function container from the kernel function container list corresponding to the first operator, wherein the kernel function container list comprises at least two candidate kernel function containers, each candidate kernel function container comprises at least one candidate kernel function pre-configured for the first operator, and the target kernel function container is a candidate kernel function container with the lowest execution time consumption in the at least two candidate kernel function containers. The optimizing module is configured to determine whether to perform a local graph optimization operation on the first node corresponding to each first operator according to the target kernel function container and the attribute information corresponding to the first operator, perform the local graph optimization operation on a to-be-optimized node determined to perform the local graph optimization operation according to the target kernel function container corresponding to the to-be-optimized node, and obtain a second computing graph. The code generating module is configured to generate executable code corresponding to the first computing graph according to the second computing graph.

[0019] The at least one embodiment of the present disclosure further provides an electronic device comprising at least one processor and at least one memory, wherein the at least one memory stores computer readable instructions, and the computer readable instructions, when executed by the at least one processor, cause the at least one processor to perform the computing graph optimization method provided in any one of the above embodiments.

[0020] The at least one embodiment of the present disclosure further provides a computer readable storage medium, which stores computer readable instructions, and the computer readable instructions, when executed by at least one processor, cause the at least one processor to perform the computing graph optimization method provided in any one of the above embodiments.

[0021] The computing graph optimization method provided by at least one embodiment of the present disclosure innovatively proposes a dynamic graph optimization mechanism supporting kernel function awareness, so that the AI compiler in the AI framework can automatically trade off the comprehensive overhead of the "general kernel function", the "high-performance kernel function", and the "high-performance kernel function + auxiliary nodes introduced by local graph optimization operations" in the graph optimization stage according to the attribute information of the operator and the execution time of the candidate kernel function container, so as to predict the kernel function (i.e., the candidate kernel function in the target kernel function container) selected in the subsequent operator selection stage; and can dynamically judge whether to perform targeted local graph optimization operations on the corresponding node in the graph optimization stage according to the predicted target kernel function container and the attribute information of the operator, so as to insert the operator (i.e., the auxiliary node) that removes the hardware running constraint of the actually selected kernel function in the computing graph, and avoid inserting invalid auxiliary nodes for "general kernel functions" or "high-performance kernel functions" that have met the hardware running constraint, so as to avoid introducing redundant computing overhead. In addition, the computing graph optimization method can be compatible with the current AI framework (such as TensorFlow, PyTorch, MindSpore, TVM, etc.), can be integrated into the AI framework as an independent plug-in module, and does not need to reconstruct the overall architecture or modify the code compilation process, which is beneficial to the extension of new hardware backends. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only related to some embodiments of the present disclosure, but not limited to the present disclosure.

[0023] Figure 1 A flowchart of a computing graph optimization method according to at least one embodiment of the present disclosure is shown.

[0024] Figure 2 A flowchart of configuring a kernel function container list according to at least one embodiment of the present disclosure is shown.

[0025] Figure 3 A schematic diagram of performing local graph optimization according to at least one embodiment of the present disclosure is shown.

[0026] Figure 4 A schematic block diagram of a computing graph optimization apparatus according to at least one embodiment of the present disclosure is shown.

[0027] Figure 5 A schematic block diagram of an electronic device according to at least one embodiment of the present disclosure is shown.

[0028] Figure 6 A schematic block diagram of a computer readable storage medium according to at least one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0029] The technical solutions of the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art without any creative effort belong to the protection scope of the present disclosure.

[0030] Unless otherwise defined, technical terms or scientific terms used in the present disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which the present disclosure pertains. The terms “first”, “second”, and similar terms used in the present disclosure do not denote any order, quantity, or importance, but are used to distinguish different components. Similarly, the terms “one”, “a”, or “the” do not denote quantity limitation, but mean that there is at least one. The terms “include” or “contain” and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, without excluding other elements or objects. The terms “connect” or “connected” and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “up”, “down”, “left”, “right”, and the like are used only to represent relative positional relationships, and when the absolute positions of the described objects are changed, the relative positional relationships can also be changed accordingly.

[0031] At present, artificial intelligence chips are developing rapidly and have been widely applied in various fields, from intelligent driving to medical diagnosis, to smart home, etc. One of the core functions of artificial intelligence chips is to efficiently process neural network models. Neural network models can be represented by a computational graph, and the compilation process of the neural network model can be completed by optimizing and reconstructing the computational graph, and mapping the optimized computational graph to executable code.

[0032] For example, a computational graph is usually organized in the form of a directed acyclic graph (DAG). The computational graph includes nodes and edges. The nodes in the computational graph represent specific computing operations, such as convolution, matrix multiplication, activation function, etc., and each node corresponds to an operator. The edges in the computational graph represent the data dependency relationship between nodes, for example, a tensor flows from the output of one operator to the input of another operator.

[0033] As described above, the current mainstream AI framework (e.g., TensorFlow, PyTorch, MindSpore, TVM, etc.) generally adopts the processing procedure of "global graph optimization first, and then operator selection" for the computation graph. High-performance kernel functions (such as Tensor Core accelerated convolution, etc.) have strict constraints on the shape size, arrangement format, etc. of the input data. For example, some high-performance kernel functions explicitly require the input data (such as tensors) to adopt the NHWC arrangement format instead of the conventional NCHW arrangement format. Such constraints can usually be removed by inserting auxiliary nodes (i.e., nodes encapsulating operators such as Transpose, Reorder, etc. for arrangement format conversion, Padding, or Reshape operators, which can also be referred to as "Transpose nodes", "Reorder nodes", "Padding nodes", and "Reshape nodes" hereinafter), but such optimization operations only have gain value for specific high-performance kernel functions.

[0034] In the foregoing processing procedure, the graph optimization phase is executed before the operator selection phase, and the AI framework cannot predict whether such high-performance kernel functions will be adopted in the subsequent operator selection phase when performing global graph optimization, which makes it difficult for the AI framework to achieve the optimal balance between generalization and extreme performance.

[0035] For example, in the case where a specific optimization operation (e.g., inserting a Reorder node or a Padding node, etc. at the adjacent positions before and after the node) is performed on a node in the graph optimization phase, and a generalization kernel function is selected for the operator encapsulated in the node in the subsequent operator selection phase, since the generalization kernel function has no specific constraints on the shape size, arrangement format, etc. of the input data, the optimization operation introduces redundant computational overhead.

[0036] For example, in the case where no specific optimization operation is performed on a node in the graph optimization phase, and a high-performance kernel function is selected for the operator encapsulated in the node in the subsequent operator selection phase, since the computation graph enters a frozen state after the operator selection phase ends, the AI framework cannot insert a node for meeting the constraint condition for the node according to the selected high-performance kernel function, which results in the high-performance kernel function being executed under a non-ideal data arrangement format or shape, and the computational efficiency is significantly reduced.

[0037] To address the above problems, one or more embodiments of the present disclosure provide a computation graph optimization method, a computation graph optimization device, an electronic device, a storage medium and a program product. The computation graph optimization method comprises: obtaining a first computation graph, wherein the first computation graph comprises at least one first node, each first node corresponds to a first operator (or each first node encapsulates a first operator), each first operator is pre-configured with at least two candidate kernel functions, and the at least two candidate kernel functions are respectively adapted to different types of hardware computing cores; for each first operator, determining a kernel function container list corresponding to the first operator according to attribute information of the first operator, and determining a target kernel function container from the kernel function container list corresponding to the first operator, wherein the kernel function container list comprises at least two candidate kernel function containers, each candidate kernel function container comprises at least one candidate kernel function pre-configured for the first operator, and the target kernel function container is a candidate kernel function container with the lowest execution time among the at least two candidate kernel function containers; determining whether to perform a local graph optimization operation on the first node corresponding to the first operator according to the target kernel function container and the attribute information corresponding to each first operator; for a to-be-optimized node determined to perform the local graph optimization operation, performing the local graph optimization operation according to the target kernel function container corresponding to the to-be-optimized node to obtain a second computation graph; and generating executable code corresponding to the first computation graph according to the second computation graph.

[0038] The computation graph optimization method provided by at least one embodiment of the present disclosure innovatively proposes a dynamic graph optimization mechanism supporting kernel function awareness, so that the AI compiler in the AI framework can automatically weigh the comprehensive overheads of the "general kernel function", the "high-performance kernel function" and the "high-performance kernel function + auxiliary node introduced by the local graph optimization operation" according to the attribute information of the operator and the execution time of the candidate kernel function container in the graph optimization stage, thereby predicting the kernel function (i.e., the candidate kernel function in the target kernel function container) selected in the subsequent operator selection stage; and can dynamically determine whether to perform a targeted local graph optimization operation on the corresponding node according to the predicted target kernel function container and the attribute information of the operator in the graph optimization stage, thereby inserting the operator (i.e., the auxiliary node) that removes the hardware running constraint for the actually selected kernel function in the computation graph, and avoiding inserting invalid auxiliary nodes for the "general kernel function" or the "high-performance kernel function" that has met the hardware running constraint, so as to avoid introducing redundant computing overheads.

[0039] In addition, the computation graph optimization method provided by at least one embodiment of the present disclosure can be compatible with current AI frameworks (such as TensorFlow, PyTorch, MindSpore, TVM, etc.), can be integrated into the AI framework as an independent plug-in module, and does not need to reconstruct the overall architecture or modify the code compilation process, which is conducive to extending new hardware backends.

[0040] The present disclosure is described below with reference to several specific embodiments. In order to keep the following description of embodiments of the present disclosure clear and concise, detailed descriptions of known functions and components will be omitted. When any component of the embodiments of the present disclosure appears in more than one figure, the component is denoted by the same or similar reference signs in each figure.

[0041] Some embodiments of the present disclosure and examples thereof are described in detail below with reference to the accompanying drawings.

[0042] Figure 1 A flowchart of a computing graph optimization method according to at least one embodiment of the present disclosure is shown.

[0043] As Figure 1 shown, the computing graph optimization method provided by at least one embodiment of the present disclosure includes steps S110 to S150.

[0044] Step S110: Obtain a first computing graph, wherein the first computing graph includes at least one first node, each first node corresponds to a first operator, each first operator is pre-configured with at least two candidate kernel functions, and the at least two candidate kernel functions are respectively adapted to different types of hardware computing cores.

[0045] Step S120: For each first operator, determine a kernel function container list corresponding to the first operator according to attribute information of the first operator, and determine a target kernel function container from the kernel function container list corresponding to the first operator, wherein the kernel function container list includes at least two candidate kernel function containers, each candidate kernel function container includes at least one candidate kernel function pre-configured for the first operator, and the target kernel function container is a candidate kernel function container with the lowest execution time among the at least two candidate kernel function containers.

[0046] Step S130: Determine whether to perform a local graph optimization operation on the first node corresponding to the first operator according to the target kernel function container and the attribute information corresponding to each first operator.

[0047] Step S140: For the to-be-optimized node determined to perform the local graph optimization operation, perform the local graph optimization operation according to the target kernel function container corresponding to the to-be-optimized node to obtain a second computing graph.

[0048] Step S150: Generate executable code corresponding to the first computing graph according to the second computing graph.

[0049] The computing graph optimization method provided by the embodiments of the present disclosure can be applied to various scenarios.

[0050] For example, the computing graph optimization method can be applied to an AI compiler (e.g., as a functional module of the AI compiler), and when the AI compiler compiles a high-level model (e.g., a neural network model) into executable code, the computing graph corresponding to the high-level model can be optimized, thereby achieving more efficient hardware resource utilization, lower inference delay, and better energy efficiency.

[0051] For example, the computing graph optimization method can also be applied to an AI processing module of a smartphone (e.g., as a built-in function of an NPU in a phone chip), and when the smartphone runs AI applications such as photo beautification and real-time translation, the computing graph behind the applications can be optimized at the same time, thereby enabling these functions to start faster and run more power-efficiently.

[0052] For example, the computing graph optimization method can also be applied to a home smart camera, and when the camera performs human movement detection and abnormal behavior recognition, the computing graph of the detection algorithm can be optimized, thereby reducing the power consumption of the camera while ensuring the real-time and accuracy of the recognition results. The present disclosure does not limit this.

[0053] For example, the hardware computing core in the embodiments of the present disclosure includes but is not limited to a vector processing core (Vector Core), a tensor processing core (Tensor Core), a neural network processing core (AI Core / NPU Core), etc. The vector processing core has good generality (generalization), can support multiple data arrangement formats (Layout) and shapes, but has relatively low computing performance. The tensor processing core has high computing performance, but has poor generality, i.e., the tensor processing core is only suitable for some operator types, and has hardware running constraints on the arrangement format, shape, and alignment of input data, etc., for example, the Tensor Core requires input data to adopt the NHWC arrangement format, and the channel dimension needs to be aligned to a multiple of 8, etc. The neural network processing core is a special computing unit customized for deep learning inference / training tasks, and has excellent performance in adaptive scenarios, but has hardware running constraints on the computing graph structure, data arrangement format, channel alignment, batch size, etc.

[0054] For example, the first node in the embodiments of the present disclosure refers to a node whose corresponding operator has multiple kernel function (Kernel) implementations, and each kernel function implementation is adapted to different types of hardware computing cores. For example, the first node can be a convolution node, and the corresponding first operator of the first node is a convolution (Conv) operator, which is pre-configured with at least two candidate kernel functions (for example, Kernel_1 and Kernel_2), wherein the candidate kernel function Kernel_1 can be executed by a vector processing core (Vector Core), and the candidate kernel function Kernel_2 can be executed by a tensor processing core (Tensor Core). For example, the first node can also be a matrix multiplication (MMA) node, and the present disclosure does not limit this.

[0055] For example, the first node in the first computing graph can be manually specified. For example, the Conv node and the MMA node in the first computing graph can be specified as the "first node", and the present disclosure does not limit this.

[0056] For example, the obtaining step of the first computing graph in the embodiments of the present disclosure can include: obtaining an original computing graph (i.e., a computing graph without a graph optimization operation); and performing a general graph optimization operation on a second node in the original computing graph to obtain the first computing graph, wherein the operator corresponding to the second node is not the first operator, that is, the second node is a node other than the first node in the first computing graph.

[0057] For example, the first computing graph in the embodiments of the present disclosure can also be an original computing graph. In this case, after performing the local graph optimization operation on the to-be-optimized node (i.e., the first node in the first computing graph that is determined to perform the local graph optimization operation) in the first computing graph, the general graph optimization operation is also performed on the nodes other than the first node in the first computing graph, and then the second computing graph is obtained.

[0058] It should be noted that the "local graph optimization operation" in the present disclosure refers to a graph optimization operation that takes a single first node as an object, and the purpose is to adapt to the hardware running constraints (such as data arrangement format, shape alignment, etc.) of a specific hardware computing core. After performing the local graph optimization operation, the total number of nodes of the first computing graph remains unchanged or increases. For example, the local graph optimization operation can include: inserting a reordering (Reorder) node, a padding (Padding) node, or a precision conversion (Cast) node, etc. in front of and behind the target first node to generate input data that meets the high-performance kernel function execution condition (i.e., meets the hardware running constraints of the input data of the hardware computing core).

[0059] In contrast, the "general graph optimization operation" in the present disclosure refers to a graph optimization operation that takes a second node of one or more non-first nodes as the object, and aims to improve overall computing efficiency, reduce redundant computation, or reduce kernel startup overhead. After performing the general graph optimization operation, the total number of nodes of the computation graph can increase, decrease, or remain unchanged. For example, the general graph optimization operation can include operator fusion, constant folding, dead code elimination, etc.

[0060] For the above step S120, for each first operator in the first computation graph, according to the attribute information of the first operator, a list of kernel function containers corresponding to the first operator is determined, and a candidate kernel function container with the lowest execution time is selected from the list of kernel function containers corresponding to the first operator as the target kernel function container of the first operator.

[0061] For example, in at least one embodiment of the present disclosure, the attribute information of the first operator at least includes the operator type of the first operator and the shape information of the input data of the first operator, wherein the shape information includes at least one of the shape size of the input data and the arrangement format of the input data.

[0062] For example, the operator type of the first operator includes but is not limited to a matrix multiplication (MMA) operator, a convolution (Conv) operator, a Fourier transform (FFT) operator, etc.

[0063] For example, the input data can be a tensor, which can be represented as NxCxHxW, where N represents the batch size, C represents the number of channels, H represents the height of the corresponding tensor, and W represents the width of the corresponding tensor. The shape size (also referred to as "dimension size") of the tensor is determined by the size of each dimension (i.e., the values of N, C, H, and W). The arrangement format of the tensor includes but is not limited to "NCHW", "NHWC", "NC / xHWx (interleaved mode)", etc., where x can take values such as 8, 16, 32, etc. as needed. NC / xHWx is similar to NHWC, and the difference between the two is that in the memory layout of NC / xHWx, the C channels are divided into C / x groups, each group has x channels: the first group consists of channels c=0 to c=x-1, the second group consists of channels c=x to c=2x-1, and each group is arranged in NHWC format.

[0064] For example, in at least one embodiment of the present disclosure, a kernel function container represents a container that encapsulates at least a kernel function, and each kernel function container encapsulates only one kernel function. The first operator is pre-configured with at least two candidate kernel functions, and the list of kernel function containers corresponding to the first operator includes at least two candidate kernel function containers.

[0065] For example, in at least one embodiment of the present disclosure, in response to the hardware computing core to which the candidate kernel function container encapsulates any candidate kernel function is adapted having a hardware running constraint, the candidate kernel function container further includes a candidate optimization list corresponding to the candidate kernel function, and the candidate optimization list includes at least one operator for satisfying the hardware running constraint.

[0066] For example, the operator included in the candidate optimization list can be referred to as an "auxiliary operator" (for example, a Reorder operator, a Padding operator, etc.), and the node corresponding to the auxiliary operator is also referred to as an "auxiliary node". By inserting the auxiliary node corresponding to the auxiliary operator at the corresponding position in the computation graph, a data transformation operation is performed on the input data of the first operator, so as to satisfy the hardware running constraint of the hardware computing core (for example, Tensor Core) on the data arrangement format or shape size of the input data, thereby ensuring that the selected high-performance kernel function can be executed correctly and efficiently.

[0067] For example, the execution time consumption of the candidate kernel function container includes the time required for the adapted hardware computing core to execute the candidate kernel function in the candidate kernel function container according to the attribute information of the first operator. The execution time consumption of the candidate kernel function container is the predicted execution time consumption.

[0068] For example, in response to the candidate kernel function container further including the candidate optimization list corresponding to the candidate kernel function, and the shape information of the input data of the first operator not satisfying the hardware running constraint of the adapted hardware computing core, the execution time consumption of the candidate kernel function container further includes the execution time of at least one operator in the candidate optimization list.

[0069] For example, before the execution step S120, the computation graph optimization method provided by at least one embodiment of the present disclosure can further include: respectively configuring a corresponding kernel function container list for the first operator of different operator types.

[0070] For example, Figure 2 A flowchart of configuring a kernel function container list according to at least one embodiment of the present disclosure is shown.

[0071] As Figure 2 shown, for any operator type in the first operator, the specific configuration steps of the corresponding kernel function container list can include steps S210 to S230.

[0072] In step S210, N candidate kernel function containers are constructed in the kernel function container list corresponding to any operator type, where N is an integer greater than or equal to 2, and the N candidate kernel function containers correspond to different hardware computing cores.

[0073] In step S220, N candidate kernel function containers are respectively encapsulated with N candidate kernel functions pre-configured with any operator type of operator and implemented by different hardware computing cores.

[0074] In step S230, for any one of the N candidate kernel functions, a candidate optimization list associated with the any one of the N candidate kernel functions is configured in the corresponding candidate kernel function container in response to the hardware computing core implementing the any one of the N candidate kernel functions having a hardware running constraint, wherein the associated candidate optimization list comprises at least one operator for satisfying the hardware running constraint.

[0075] The following takes the first operator as a Conv operator and the Conv operator pre-configured with two candidate kernel functions (Kernel_1 and Kernel_2) as an example to specifically introduce the configuration process of the kernel function container list.

[0076] For example, for the above step S210, since the two candidate kernel functions Kernel_1 and Kernel_2 pre-configured with the Conv operator are respectively adapted to the hardware computing core Vector Core and the hardware computing core Tensor Core, two candidate kernel function containers (for example, Conv_KernelEngine_1() and Conv_KernelEngine_2()) are constructed in the kernel function container list (for example, Conv_KernelEngine_List()) corresponding to the Conv operator.

[0077] For example, for the above step S220, the candidate kernel function Kernel_1 is encapsulated into the candidate kernel function container Conv_KernelEngine_1(), and the candidate kernel function Kernel_2 is encapsulated into the candidate kernel function container Conv_KernelEngine_2().

[0078] For example, for the above step S230, in response to the hardware computing core VectorCore of the candidate kernel function Kernel_1 having no hardware running constraint, the corresponding candidate optimization list does not need to be configured in the candidate kernel function container Conv_KernelEngine_1(). In response to the hardware computing core Tensor Core of the candidate kernel function Kernel_2 having a hardware running constraint (for example, requiring the shape size of the input data of the Conv operator to be a multiple of 16 and the arrangement format to be NHWC), the candidate optimization list (for example, PassList()) associated with the candidate kernel function Kernel_2 is configured in the candidate kernel function container Conv_KernelEngine_2(), wherein the PassList() associated with the candidate kernel function Kernel_2 includes a Padding operator (for satisfying the hardware running constraint that the shape size of the input data of the Conv operator is a multiple of 16) and a Reorder operator (for satisfying the hardware running constraint that the arrangement format of the input data of the Conv operator is NHWC).

[0079] For example, after the above steps S210, S220 and S230 are executed, the obtained kernel function container list for the Conv operator can be represented as: Conv_KernelEngine_List{ Conv_KernelEngine_1(Kernel_1 ), Conv_KernelEngine_2( Kernel_2, PassList( AutoReoder(),AutoPadding() ) )}. “AutoReoder()” represents the optimization function corresponding to the Reorder operator, and by calling AutoReoder() in the kernel function container list, an auxiliary node (i.e., a Reorder node) corresponding to the Reorder operator can be automatically inserted at the corresponding position adjacent to the Conv node. “AutoPadding()” represents the optimization function corresponding to the Padding operator, and by calling AutoPadding() in the kernel function container list, an auxiliary node (i.e., a Padding node) corresponding to the Padding operator can be automatically inserted at the corresponding position adjacent to the Conv node.

[0080] For example, the configuration process of the kernel function container list corresponding to the first operator of other operator types (for example, the MMA operator) can refer to the configuration process of the kernel function container list Conv_KernelEngine_List corresponding to the Conv operator, which will not be described here.

[0081] For example, the implementation of the step S120 can include: determining a first kernel function container list corresponding to the first operator according to the operator type of the first operator; obtaining an execution time of each candidate kernel function container in the first kernel function container list according to at least the shape information of the input data of the first operator and the candidate kernel function in the candidate kernel function container; and taking the candidate kernel function container with the lowest execution time as the target kernel function container of the first operator.

[0082] The determination of the target kernel function container will be described below by means of specific embodiments.

[0083] In an embodiment of the present disclosure, the first operator is a Conv operator, the input data of the Conv operator is a one-dimensional tensor, the shape size (Shape) of the input data is 1, and the kernel function container list corresponding to the Conv operator is Conv_KernelEngine_List{ Conv_KernelEngine_1( Kernel_1 ), Conv_KernelEngine_2(Kernel_2, PassList( AutoPadding() ) )} (for details, please refer to the above description), wherein the hardware computing core of the candidate kernel function Kernel_1 is Vector Core, the hardware running constraint of the hardware computing core Tensor Core of the candidate kernel function Kernel_2 is “the shape size of the input data of the Conv operator must be a multiple of 16”, and the operation speed of the Tensor Core is 10 times that of the Vector Core.

[0084] For example, for the candidate kernel function container Conv_KernelEngine_1( Kernel_1 ), the hardware computing core Vector Core of the candidate kernel function Kernel_1 has good generalization, and thus has no hardware running constraint, so the execution time of the candidate kernel function container Conv_KernelEngine_1 only includes the time required for the Vector Core to execute the candidate kernel function Kernel_1 according to the attribute information of the first operator. For example, the Shape of the input data of the first operator is 1, and the calculation amount of the Vector Core can be represented as vcore(Shape) = vcore(1) = 1.

[0085] For example, for the candidate kernel container Conv_KernelEngine_2(Kernel_2, PassList(AutoPadding())), since the Shape(1) of the input data of the Conv operator does not satisfy the hardware running constraint of the hardware computing core TensorCore (i.e., requiring the Shape to be a multiple of 16), in addition to the time required for Tensor Core to execute Kernel_2, the execution of the candidate kernel container Conv_KernelEngine_2 further includes the execution time of the Padding operator in the candidate optimization list PassList. Specifically, the execution time of the Padding operator includes: the time required for the Padding operator to adjust the Shape of the input data from 1 to 16 before TensorCore executes Kernel_2; and the time required for the Padding operator to restore the Shape of the input data from 16 to 1 after Tensor Core executes Kernel_2. For example, the amount of computation of Tensor Core can be represented as: padding(1->16) + tcore(16) + padding(16 ->1) = 15 + 16 / 10 + 15 = 31.6.

[0086] For example, in the case where the time required for the hardware computing core to execute the corresponding candidate kernel is proportional to the amount of computation, and the time required for Vector Core and Tector Core to execute a unit amount of computation is the same (e.g., both T), the execution time of the candidate kernel container Conv_KernelEngine_1 can be represented as 1T, and the execution time of the candidate kernel container Conv_KernelEngine_2 can be represented as 31.6T, then the candidate kernel container Conv_KernelEngine_1 is selected as the target kernel container of the Conv node in this embodiment.

[0087] In another embodiment of the present disclosure, the first operator is a Conv operator, the input data of the Conv operator is a one-dimensional tensor, the shape of the input data is 15, and the kernel function container list corresponding to the Conv operator is Conv_KernelEngine_List{ Conv_KernelEngine_1( Kernel_1 ), Conv_KernelEngine_2(Kernel_2, PassList( AutoPadding() ) )}, wherein the hardware computing core of the candidate kernel function Kernel_1 is a Vector Core, the hardware running constraint of the hardware computing core Tensor Core of the candidate kernel function Kernel_2 is “the shape of the input data of the Conv operator must be a multiple of 16”, and the operation speed of the Tensor Core is 10 times that of the Vector Core.

[0088] For example, for the candidate kernel function container Conv_KernelEngine_1( Kernel_1 ), the shape of the input data of the first operator is 15, and the calculation amount of the Vector Core adapted by Kernel_1 can be represented as vcore(Shape) = 15.

[0089] For example, for the candidate kernel function container Conv_KernelEngine_2( Kernel_2, PassList(AutoPadding() ) ), since the shape (15) of the input data of the Conv operator does not satisfy the hardware running constraint (i.e., the shape must be a multiple of 16) of the hardware computing core Tensor Core of Kernel_2, in addition to the time required for the Tensor Core to execute Kernel_2, the execution time of the candidate kernel function container Conv_KernelEngine_2 further includes the execution time of the padding operator in the candidate optimization list PassList. For example, the calculation amount of the Tensor Core can be represented as: padding(15->16)+ tcore(16)+ padding (16 ->15) = 1 + 16 / 10 + 1 = 3.6.

[0090] For example, in a case where the time required for the hardware computing core to execute the corresponding candidate kernel function is proportional to the amount of computation, and the time required for the Vector Core and the Tensor Core to execute the unit amount of computation is the same (for example, both are T), the execution time of the candidate kernel function container Conv_KernelEngine_1 can be represented as 15T, and the execution time of the candidate kernel function container Conv_KernelEngine_2 can be represented as 3.6T. In this embodiment, the candidate kernel function container Conv_KernelEngine_2 is selected as the target kernel function container of the Conv node.

[0091] In another embodiment of the present disclosure, the first operator is a Conv operator, the input data of the Conv operator is a tensor NxCxHxW, the arrangement format is NCHW, and the kernel function container list corresponding to the Conv operator is Conv_KernelEngine_List{Conv_KernelEngine_1(Kernel_1), Conv_KernelEngine_2(Kernel_2, PassList(AutoPadding(), AutoReorder()))}, wherein the hardware computing core of the candidate kernel function Kernel_1 is a Vector Core, the hardware running constraint of the hardware computing core Tensor Core of the candidate kernel function Kernel_2 is “the shape size of the input data of the Conv operator is required to be a multiple of 8, and the arrangement format is NHWC”, and the operation speed of the Tensor Core is 10 times that of the Vector Core.

[0092] For example, for the candidate kernel function container Conv_KernelEngine_1(Kernel_1), the Shape of the input data of the first operator is NxCxHxW, and the amount of computation of the Vector Core adapted by Kernel_1 can be represented as vcore(Shape)=NxCxHxW.

[0093] For example, for the candidate kernel container Conv_KernelEngine_2 (Kernel_2, PassList (AutoPadding ( ), AutoReorder ( ) ) ), if the Shape (NxCxHxW) of the input data of the Conv operator satisfies the "requirement that the Shape is a multiple of 8" in the hardware running constraints of the Tensor Core, but the arrangement format (NCHW) of the input data of the Conv operator does not satisfy the "requirement that the arrangement format of the input data of the Conv operator is NHWC" in the hardware running constraints of the Tensor Core, therefore, in addition to the time required for the Tensor Core to execute Kernel_2, the execution time of the candidate kernel container Conv_KernelEngine_2 further includes the execution time of the Reorder operator in the candidate optimization list PassList. For example, the calculation amount of the Tensor Core can be represented as: reorder (NCHW->NHWC) + tcore (NHWC) + reorder (NHWC->NCHW) = NxCxHxW + NxCxHxW / 10 + NxCxHxW = 2.1x (NxCxHxW).

[0094] For example, in the case where the time required for the hardware computing core to execute the corresponding candidate kernel is proportional to the calculation amount, and the time required for the Vector Core and the Tector Core to execute a unit calculation amount is the same (for example, both are T), the execution time of the candidate kernel container Conv_KernelEngine_1 can be represented as 1Tx (NxCxHxW), and the execution time of the candidate kernel container Conv_KernelEngine_2 can be represented as 2.1Tx (NxCxHxW). Therefore, in this embodiment, the candidate kernel container Conv_KernelEngine_1 is selected as the target kernel container of the Conv node.

[0095] For example, in at least one embodiment of the present disclosure, the execution time of the corresponding candidate kernel container can be determined using a cost estimation function, and the cost estimation function is at least configured according to the candidate kernel in the corresponding candidate kernel container.

[0096] For example, the cost estimation function can be configured (encapsulated) in the corresponding candidate kernel function container, so that the candidate kernel function container can return the execution time estimation of its associated candidate kernel function when queried. For example, the cost estimation function can be centrally managed in a separate cost estimation model module, which dynamically calculates the execution time of each candidate kernel function container according to the attribute information of the operator (e.g., the shape of the input data, the data arrangement format, etc.). For example, the cost estimation function can be cached in the form of performance profiling data in the runtime database, and the measured or fitted execution time of each candidate kernel function container is obtained through table lookup during the deployment phase. The present disclosure does not make any limitation in this regard.

[0097] For example, in the case where the candidate kernel function container in the first kernel function container list includes a cost estimation function, when performing the step of obtaining the execution time of each candidate kernel function container in the first kernel function container list according to at least the shape information of the input data of the first operator and the candidate kernel function in the candidate kernel function container, the shape information can be input into the cost estimation function included in each candidate kernel function container in the first kernel function container list. The cost estimation function returns the calculated execution time of the corresponding candidate kernel function container.

[0098] For example, in response to the corresponding candidate kernel function container further including a candidate optimization list corresponding to the candidate kernel function, the cost estimation function is further configured according to the candidate optimization list.

[0099] For example, for the candidate kernel function container Conv_KernelEngine_1( Kernel_1 ), the hardware computing core adapted by the candidate kernel function Kernel_1 is Vector Core, and the cost estimation function Cost_1 set in the kernel function container can be configured according to the time required for Vector Core to execute the candidate kernel function Kernel_1, for example, Cost_1 = T1 × (vcore(Shape) ), where T1 is the time reference required for Vector Core to execute a unit of computation.

[0100] For example, in response to the corresponding candidate kernel function container further including a candidate optimization list corresponding to the candidate kernel function, the cost estimation function is further configured according to the candidate optimization list.

[0101] For example, for the candidate kernel container Conv_KernelEngine_2(Kernel_2, PassList(AutoPadding())), the hardware computing core to which the candidate kernel Kernel_2 is adapted is a Tensor Core, and the hardware running constraint of the Tensor Core is "the Shape of the input data is required to be a multiple of 16". The cost estimation function Cost_2 set in the kernel container can be configured according to the time required for the Tensor Core to execute the candidate kernel Kernel_2 and the execution time of the Padding operator in the candidate optimization list, for example, Cost_2 = T2 x (padding(Shape->16) + tcore(16) + padding(16->Shape)), where T2 is a time benchmark required for the Tector Core to execute a unit of calculation.

[0102] It should be noted that the configuration manner of the cost estimation function described above is only illustrative and does not limit the technical solutions of the present disclosure.

[0103] For example, other manners can also be adopted to predict the execution time consumption, and the present disclosure does not make specific limitations in this regard.

[0104] In the process of determining the predicted execution time consumption, only cost prediction is performed without modifying the graph structure, and after the target kernel container is determined, the calculation graph structure is modified by the local graph optimization operation.

[0105] For the step S130 described above, whether the local graph optimization operation is performed on the first node corresponding to the first operator can be determined according to whether the hardware computing core to which the candidate kernel in the target kernel container is adapted has a hardware running constraint, and in the case where the hardware running constraint exists, whether the attribute information of the first operator satisfies the hardware running constraint.

[0106] For example, in the case where the attribute information of the first operator at least includes the shape information of the input data of the first operator, and the hardware running constraint includes a constraint condition on the shape information, the specific implementation manner of the step S130 can include: in response to the target kernel container not including the candidate optimization list, determining not to perform the local graph optimization operation on the first node; in response to the target kernel container including the candidate optimization list and the shape information satisfying the constraint condition, determining not to perform the local graph optimization operation on the first node; and in response to the target kernel container including the candidate optimization list and the shape information not satisfying the constraint condition, determining to perform the local graph optimization operation on the first node and taking the first node as the to-be-optimized node.

[0107] For example, in response to the target kernel function container (e.g., Conv_KernelEngine_1(Kernel_1)) corresponding to the Conv operator not including the candidate optimization list, it is determined that no local graph optimization operation is performed on the Conv node corresponding to the Conv operator.

[0108] For example, in response to the target kernel function container (e.g., Conv_KernelEngine_2(Kernel_2, PassList(AutoPadding())) ) corresponding to the Conv operator including the candidate optimization list and the shape information satisfying the constraint condition (e.g., the Shape of the input data of the Conv operator is 16, and the constraint condition is “require the Shape of the input data to be a multiple of 16”), it is determined that no local graph optimization operation is performed on the node corresponding to the Conv operator.

[0109] For example, in response to the target kernel function container (e.g., Conv_KernelEngine_2(Kernel_2, PassList(AutoPadding())) ) corresponding to the Conv operator including the candidate optimization list and the shape information not satisfying the constraint condition (e.g., the Shape of the input data of the Conv operator is 1, and the constraint condition is “require the Shape of the input data to be a multiple of 16”), it is determined that the local graph optimization operation is performed on the node corresponding to the Conv operator, and the node corresponding to the Conv operator is taken as the to-be-optimized node.

[0110] For example, the specific implementation of the above step S140 can include: for each to-be-optimized node, according to at least one operator included in the candidate optimization list for satisfying the hardware running constraint of one candidate kernel function, at least one auxiliary node is added at the corresponding position adjacent to the to-be-optimized node in the first computation graph, to obtain the second computation graph.

[0111] For example, the to-be-optimized node is the first node determined to perform the local graph optimization operation in step S130.

[0112] For example, the “auxiliary node” in the present disclosure refers to a non-original calculation logic node dynamically inserted for satisfying a specific constraint (such as a hardware running constraint) in the process of performing the local graph optimization operation. The auxiliary node itself does not change the mathematical semantics of the model, and is only used for data preprocessing or post-processing. For example, the auxiliary node can include a reorder (Reorder) node, a padding (Padding) node, a cast (Cast) node, etc., which are not limited in the present disclosure.

[0113] For example, since the local graph optimization operation in the embodiments of the present disclosure is performed on a single first node, the insertion position of the auxiliary node is only related to the to-be-optimized node served by the auxiliary node, and the next node of the to-be-optimized node does not need to be considered.

[0114] For ease of description, the candidate kernel function and the candidate optimization list in the target kernel function container corresponding to the to-be-optimized node are referred to as a "target kernel function" and a "target optimization list", respectively.

[0115] For example, Figure 3 A schematic diagram of performing local graph optimization is shown according to at least one embodiment of the present disclosure.

[0116] As Figure 3 shown, the first computation graph includes six nodes, i.e., nodes A to F. After the steps S110 to S130, it is determined that node B (for example, a Conv node) is the first node (i.e., the to-be-optimized node) to perform local graph optimization, and the target optimization list corresponding to node B is PassList( AutoPadding(), AutoReorder() ), the constraint condition of the adapted hardware computing core on node B is "requiring the Shape of the input data to be a multiple of 16 and the arrangement format of the input data to be NHWC", and the Shape of the input data of node B is 1 and the arrangement format is NHWC. In this case, the target optimization list includes the Padding operator and the Reorder operator for satisfying the constraint condition, and the input data of node B does not satisfy the constraint condition of "requiring the Shape of the input data to be a multiple of 16", so a Padding node (for example, as Figure 3 shown) can be inserted at the front and rear positions adjacent to node B in the first computation graph, so that the Shape of the input data is expanded to a multiple of 16 before entering node B, and the Shape of the data output by node B is restored to 1 after leaving node B.

[0117] For the above step S150, the second computation graph is the computation graph obtained after performing the local graph optimization operation on the to-be-optimized node in the first computation graph, and the second computation graph has stronger hardware adaptability. Based on the second computation graph after the local graph optimization, the code generation operation is performed to generate executable code which is functionally equivalent to the first computation graph and is adapted to the target hardware computing core.

[0118] For example, the generated executable code includes, but is not limited to, device-specific kernel function call sequences, memory management instructions, and execution scheduling logic, which can be deployed on hardware platforms such as graphics processing units (GPUs), neural network processing units (NPUs), or deep learning processing units (DPUs) to perform inference or training tasks.

[0119] For example, after determining the target kernel function container of each first operator in the first computation graph, the computation graph optimization method provided by at least one embodiment of the present disclosure further includes: establishing a binding relationship between the first operator and the determined target kernel function container. For example, whether or not to perform a local graph optimization operation on the first node, after determining the target kernel function container of each first operator in the first computation graph, a binding relationship is established between the first operator and the determined target kernel function container, so that in the subsequent operator selection stage, a candidate kernel function in the target kernel function container can be directly selected as the kernel function implementation of the first operator.

[0120] For example, through the established binding relationship, the following steps can be performed in the operator selection stage and the execution stage to implement the above step S150: traversing each node in the second computation graph, in response to the current operator corresponding to the current node being a non-first operator, selecting a kernel function for the current operator according to the attribute information of the current operator; in response to the current operator corresponding to the current node being a first operator of any operator type, selecting a candidate kernel function in the target kernel function container bound by the first operator of any operator type as the kernel function of the current operator; and generating executable code corresponding to the first computation graph according to the kernel functions of the operators corresponding to each node in the second computation graph.

[0121] For example, for non-first operators (such as addition operators, activation function operators, etc.), a default general kernel function can be directly used. For example, a specific kernel function implementation is selected for the operator according to the tensor shape information of the operator. Of course, other selection methods can also be used, which are not limited by the present disclosure. For example, for a first operator of any operator type, a candidate kernel function in the target kernel function container of the first operator can be used as the specific kernel function implementation of the first operator. In this way, the selection of the kernel function implementation of the operator corresponding to each node in the second computation graph is completed.

[0122] After that, the executable code corresponding to the first computation graph (or the original computation graph) can be generated based on the kernel functions selected by the operators corresponding to each node in the second computation graph, and the entire graph optimization process is completed.

[0123] The computational graph optimization method provided in at least one embodiment of this disclosure innovatively proposes a dynamic graph optimization mechanism that supports kernel function awareness. This allows the AI ​​compiler in the AI ​​framework to automatically weigh the combined overhead of "generalized kernel function", "high-performance kernel function" and "high-performance kernel function + auxiliary nodes introduced by local graph optimization operations" during the graph optimization stage, based on the attribute information of the operators and the execution time of the candidate kernel function container. This predicts the kernel function to be selected in the subsequent operator selection stage (i.e., the candidate kernel function in the target kernel function container). Furthermore, during the graph optimization stage, based on the predicted target kernel function container and the attribute information of the operators, it can dynamically determine whether to perform targeted local graph optimization operations on the corresponding nodes. This allows for the insertion of operators (i.e., auxiliary nodes) in the computational graph that remove the hardware operation constraints of the actually selected kernel function, and avoids prematurely inserting invalid auxiliary nodes for "generalized kernel function" or "high-performance kernel function" that already meets the hardware operation constraints, thus avoiding the introduction of redundant computational overhead.

[0124] Furthermore, this computation graph optimization method is compatible with current AI frameworks (such as TensorFlow, PyTorch, MindSpore, TVM, etc.) and can be integrated into AI frameworks as independent plug-in modules without refactoring the overall architecture or modifying the code compilation process, which is beneficial for expanding to new hardware backends.

[0125] For example, Figure 4 A schematic block diagram of a computational graph optimization apparatus 400 according to at least one embodiment of the present disclosure is shown.

[0126] like Figure 4 As shown, the computational graph optimization device 400 includes an acquisition module 410, a query module 420, an optimization module 430, and a code generation module 440.

[0127] The acquisition module 410 is configured to acquire a first computation graph, wherein the first computation graph includes at least one first node, each first node corresponds to a first operator, each first operator is pre-configured with at least two candidate kernel functions, and the at least two candidate kernel functions are adapted to different types of hardware computing cores respectively.

[0128] The query module 420 is configured to, for each first operator, determine the kernel function container list corresponding to the first operator based on the attribute information of the first operator, and determine the target kernel function container from the kernel function container list corresponding to the first operator. The kernel function container list includes at least two candidate kernel function containers, each candidate kernel function container includes at least one candidate kernel function pre-configured by the first operator, and the target kernel function container is the candidate kernel function container with the lowest execution time among the at least two candidate kernel function containers.

[0129] The optimization module 430 is configured to: determine whether to perform a local graph optimization operation on the first node corresponding to the first operator based on the target kernel function container and attribute information corresponding to each first operator; and perform a local graph optimization operation on the node to be optimized based on the target kernel function container corresponding to the node to be optimized to obtain a second computation graph.

[0130] The code generation module 440 is configured to generate executable code corresponding to the first computation graph based on the second computation graph.

[0131] It should be noted that the acquisition module 410 can be used to implement Figure 1 The step S110 shown; the query module 420 can be used to implement Figure 1 The step S120 shown; optimization module 430 can be used to implement Figure 1 Steps S130 and S140 are shown; the code generation module can be used to implement... Figure 1 The steps are shown in step S150. For a detailed description of the functions that the acquisition module 410 can perform, please refer to the description of step S110 in the embodiments of the computational graph optimization method described above. For a detailed description of the functions that the query module 420 can perform, please refer to the description of step S120 in the embodiments of the computational graph optimization method described above. For a detailed description of the functions that the optimization module 430 can perform, please refer to the descriptions of steps S130 and S140 in the embodiments of the computational graph optimization method described above. For a detailed description of the functions that the code generation module 440 can perform, please refer to the description of step S150 in the embodiments of the computational graph optimization method described above. Repeated descriptions will not be repeated here. Furthermore, the computational graph optimization device 400 can achieve similar technical effects to the aforementioned computational graph optimization method, and will not be described further here.

[0132] It should be noted that, in at least one embodiment of this disclosure, the computational graph optimization device 400 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited; it can be constructed from analog devices, digital chips, or other suitable methods according to circuit principles.

[0133] For example, the computational graph optimization device 400 can be implemented in hardware, software, or a combination of hardware and software, and this disclosure does not impose any specific limitations on it.

[0134] The computational graph optimization method and apparatus provided in at least one embodiment of this disclosure can be applied to different systems or devices, such as those used in... Figure 5An electronic device 500 is shown. The electronic device 500 can be a terminal, such as a mobile phone terminal, a tablet computer, a notebook computer, an AR device, a VR device, a vehicle-mounted terminal, etc., and can also be a server, etc. For example, the hardware architecture of the electronic device 500 can include a graphics processor. The computing graph optimization method provided by at least one embodiment of the present disclosure can be applied to scenarios involving CPU, high performance computing (HPC), and artificial intelligence (AI) in the electronic device 500. Of course, the present disclosure is not limited thereto, and any scenario, device, apparatus, etc. involving a computing graph and model running can adopt the computing graph optimization method or computing graph optimization apparatus provided by at least one embodiment of the present disclosure.

[0135] In some embodiments, the computing graph optimization apparatus provided by at least one embodiment of the present disclosure can be a chip, for example, the chip is a system-on-a-chip (SoC). The system-on-a-chip includes a processor, which can be a single-core processor or a multi-core processor, a memory, an I / O interface, etc.

[0136] Figure 5 A schematic block diagram of an electronic device according to at least one embodiment of the present disclosure is shown. As Figure 5 shown, the electronic device 500 is suitable for implementing the computing graph optimization method provided by the embodiments of the present disclosure, for example. It should be noted that Figure 5 The components of the electronic device 500 shown are only exemplary and are not limiting, and the electronic device 500 can also have other components according to actual application needs.

[0137] As Figure 5 shown, the electronic device 500 can include a processing apparatus 501 (such as a central processor, a graphics processor, etc.), which can perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in a memory to achieve various functions.

[0138] For example, when the computer-readable instructions are executed by the processing apparatus 501, one or more steps of the computing graph optimization method according to any of the above embodiments can be performed. It should be noted that the detailed description of the processing process of the computing graph optimization method can refer to the related description in the above embodiments of the computing graph optimization method.

[0139] For example, the memory can include any combination of one or more computer program products, which can include various forms of computer-readable storage media, for example, volatile memory and / or non-volatile memory. Volatile memory, for example, can include random access memory (RAM) 503 and / or cache memory and / or the like, for example, computer-readable instructions can be loaded from the storage 508 into the random access memory (RAM) 503 to run the computer-readable instructions. Non-volatile memory, for example, can include read-only memory (ROM) 502, a hard disk, an erasable programmable read-only memory (EPROM), a compact disc read-only memory (CD-ROM), a USB memory, a flash memory, and the like. Various application programs and various data, for example, style images, and various data used and / or generated by the application programs, and the like, can also be stored in the computer-readable storage medium.

[0140] For example, the processing device 501, the read-only memory (ROM) 502, and the random access memory (RAM) 503 are connected to each other through the bus 504. The input / output (I / O) interface 505 is also connected to the bus 504.

[0141] Generally, the following devices can be connected to the input / output (I / O) interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 508 including, for example, a magnetic tape, a hard disk, a flash memory, and the like; and communication devices 509. The communication devices 509 can allow the electronic device 500 to communicate wirelessly or wired with other electronic devices to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or possessed, and the electronic device 500 can instead be implemented or possessed with more or fewer devices. For example, the processing device 501 can control other components in the electronic device 500 to perform desired functions. The processing device 501 can be a central processing unit (CPU), a tensor processor (TPU), or a graphics processor GPU, and the like, which has data processing capability and / or program execution capability. The central processing unit (CPU) can be X86, ARM, RISC-V architecture, and the like. The GPU can be directly integrated into the SOC, directly integrated into the mainboard, or built into the north bridge chip of the mainboard.

[0142] Figure 6 A schematic block diagram of a computer-readable storage medium according to at least one embodiment of the present disclosure is shown.

[0143] For example, as Figure 6As shown, one or more computer-readable instructions 601 can be stored non-transitorily on the computer-readable storage medium 600. The computer-readable instructions 601 may, for example, execute one or more steps of the computational graph optimization method according to the above description when executed by at least one processor.

[0144] For example, the storage medium 600 can be applied in the electronic device 500, for example, the storage medium 600 can include the storage 508 in the electronic device 500.

[0145] For example, the storage can include any combination of one or more computer program products. The computer program product can include various forms of computer-readable storage media for storing information that is in a form readable by a computer. For example, the computer-readable storage media can include volatile storage media (e.g., random access memory (RAM)) and / or non-volatile storage media (e.g., read-only memory (ROM), hard disk drives, solid state drives, USB flash drives, flash memory, etc.). One or more computer-readable instructions can be stored on the computer-readable storage medium. The computer-readable instructions may, for example, implement various functions of the processor. Various application programs and various data can also be stored in the storage medium.

[0146] For example, the storage medium can include a memory card of a smart phone, a cache component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a compact disc read-only memory (CD-ROM), a flash memory, or any combination of the above storage media, or other applicable storage medium.

[0147] The flow diagrams and the block diagrams in the drawings are illustrations of possible architectures, functions, and operations for systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0148] The units described in the embodiments of the present disclosure can be implemented in the form of software, or can be implemented in the form of hardware. In some cases, the name of the unit does not constitute a limitation on the unit itself.

[0149] The functions described above in the present document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0150] The above description is merely the preferred embodiments of the present disclosure and the explanation of the applied technical principles. It should be understood by those skilled in the art that the disclosed scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and also covers other technical solutions formed by any combinations of the above technical features or equivalent features without departing from the disclosed concept. For example, the technical solutions formed by replacing the above features with the technical features disclosed in the present disclosure (but not limited to) having similar functions.

[0151] In addition, although each operation is depicted in a particular order, this should not be understood as requiring the operations to be performed in the particular order shown or in a sequential order. In certain circumstances, multitasking and parallel processing can be advantageous. Similarly, although several implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be combined in a single embodiment. Conversely, various features described in the context of a single embodiment can also be separated and implemented in multiple embodiments. Although the subject matter has been described in terms of particular structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the particular features or acts described above. Rather, the particular features and acts described above are merely illustrative of the example forms of implementing the claims.

[0152] In addition to the above exemplary descriptions, the present disclosure has the following points to be explained:

[0153] (1) The drawings of the embodiments of the present disclosure only involve the structures involved in the embodiments of the present disclosure, and other structures can be referred to the general design.

[0154] (2) In the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined to obtain new embodiments.

[0155] The above-described exemplary embodiments of the present disclosure are merely for the purpose of illustration and are not intended to limit the scope of the present disclosure, which is defined by the appended claims.

Claims

1. A method of optimizing a computational graph, the method comprising: The method comprises: obtaining a first calculation graph, wherein the first calculation graph comprises at least one first node, each first node corresponds to a first operator, each first operator is pre-configured with at least two candidate kernel functions, and the at least two candidate kernel functions are adapted to different types of hardware computing cores respectively; for each first operator, determining a kernel function container list corresponding to the first operator according to attribute information of the first operator, and determining a target kernel function container from the kernel function container list corresponding to the first operator, wherein the kernel function container list comprises at least two candidate kernel function containers, each candidate kernel function container comprises at least one pre-configured candidate kernel function of the first operator, and the target kernel function container is a candidate kernel function container with the lowest execution time in the at least two candidate kernel function containers; determining whether to perform a local graph optimization operation on the first node corresponding to the first operator according to the target kernel function container corresponding to each first operator and the attribute information; for a to-be-optimized node determined to perform the local graph optimization operation, performing the local graph optimization operation according to the target kernel function container corresponding to the to-be-optimized node to obtain a second calculation graph; and generating executable code corresponding to the first calculation graph according to the second calculation graph.

2. The method of claim 1, wherein, In response to the hardware computing core adapted to the one candidate kernel function having a hardware running constraint, the candidate kernel function container further comprises a candidate optimization list corresponding to the one candidate kernel function, and the candidate optimization list comprises at least one operator for satisfying the hardware running constraint of the one candidate kernel function.

3. The method of claim 1, wherein, The attribute information of the first operator at least comprises an operator type of the first operator and shape information of input data of the first operator, wherein the shape information comprises at least one of a shape size of the input data and an arrangement format of the input data, The determination of the kernel function container list corresponding to the first operator and the target kernel function container from the kernel function container list corresponding to the first operator according to the attribute information of the first operator comprises: determining a first kernel function container list corresponding to the first operator according to the operator type of the first operator; for each candidate kernel function container in the first kernel function container list, obtaining an execution time of the candidate kernel function container according to at least the shape information of the input data of the first operator and a candidate kernel function in the candidate kernel function container; and taking the candidate kernel function container with the lowest execution time as the target kernel function container of the first operator.

4. The method of claim 3, wherein, The execution time of the candidate kernel function container comprises a time required by the adapted hardware computing core to execute the candidate kernel function in the candidate kernel function container according to the attribute information of the first operator.

5. The method of claim 4, wherein, In response to the candidate kernel function container further comprising a candidate optimization list corresponding to the candidate kernel function, and the shape information of the input data of the first operator not satisfying the hardware running constraint of the adapted hardware computing core, the execution time of the candidate kernel function container further comprises an execution time of at least one operator in the candidate optimization list.

6. The method of claim 3, wherein, Each candidate kernel container further comprises a cost estimation function configured to determine an execution time of the corresponding candidate kernel container, and the cost estimation function is configured at least according to the candidate kernel in the corresponding candidate kernel container, The execution time of each candidate kernel container in the first kernel container list is determined at least according to shape information of input data of the first operator and the candidate kernel in the candidate kernel container. The shape information is input into the cost estimation function included in each candidate kernel container in the first kernel container list to obtain the execution time of each candidate kernel container in the first kernel container list.

7. The method of claim 6, wherein, The cost estimation function is further configured according to the candidate optimization list corresponding to the candidate kernel in response to the corresponding candidate kernel container further comprising the candidate optimization list corresponding to the candidate kernel.

8. The method of claim 2, wherein, The attribute information of the first operator at least includes shape information of input data of the first operator, and the hardware running constraint includes a constraint condition for the shape information, The target kernel container corresponding to each first operator and the attribute information are used to determine whether to perform a local graph optimization operation on the first node corresponding to the first operator, including: In response to the target kernel container not comprising the candidate optimization list, it is determined that the local graph optimization operation is not performed on the first node, In response to the target kernel container comprising the candidate optimization list and the shape information satisfying the constraint condition, it is determined that the local graph optimization operation is not performed on the first node, and In response to the target kernel container comprising the candidate optimization list and the shape information not satisfying the constraint condition, it is determined that the local graph optimization operation is performed on the first node, and the first node is taken as the to-be-optimized node.

9. The method of claim 8, wherein, The local graph optimization operation is performed on the to-be-optimized node determined to perform the local graph optimization operation according to the target kernel container corresponding to the to-be-optimized node to obtain a second computing graph, including: For each to-be-optimized node, at least one auxiliary node is added at a corresponding position adjacent to the to-be-optimized node in the first computing graph according to at least one operator included in the candidate optimization list for satisfying the hardware running constraint of the one candidate kernel to obtain the second computing graph.

10. The method of claim 1, wherein, The method further comprises: Binding the first operator and the target kernel container; Wherein, according to the second computing graph, the executable code corresponding to the first computing graph is generated, including: Traversing each node in the second computing graph, In response to the current operator corresponding to the current node being the first operator, a kernel function of the current operator is selected according to attribute information of the current operator; In response to the current operator corresponding to the current node being the first operator of any operator type, the candidate kernel in the target kernel container bound by the first operator of the any operator type is taken as the kernel function of the current operator; According to a kernel function of an operator corresponding to each node in the second calculation graph, executable code corresponding to the first calculation graph is generated.

11. The method of claim 1, wherein, The first calculation graph is obtained by: obtaining an original calculation graph; performing a general graph optimization operation on a second node in the original calculation graph to obtain the first calculation graph, wherein the operator corresponding to the second node is not the first operator.

12. The method of claim 1, wherein, The method further comprises: configuring a corresponding kernel function container list for each first operator of different operator types.

13. The method of claim 12, wherein, The method further comprises: for each operator type in the first operator: constructing N candidate kernel function containers in the kernel function container list corresponding to the operator type, wherein N is an integer greater than or equal to 2, and the N candidate kernel function containers correspond to different hardware computing cores; encapsulating N candidate kernel functions of the operator type pre-configured by different hardware computing cores into the N candidate kernel function containers; for each candidate kernel function in the N candidate kernel functions, in response to a hardware computing core implementing the candidate kernel function having a hardware running constraint, configuring a candidate optimization list associated with the candidate kernel function in the corresponding candidate kernel function container, wherein the associated candidate optimization list includes at least one operator for satisfying the hardware running constraint.

14. The method of claim 13, wherein, The method further comprises: constructing a corresponding cost estimation function for each candidate kernel function container, wherein the cost estimation function is constructed according to the candidate kernel function encapsulated in each candidate kernel function container, wherein, in response to the candidate kernel function container further including the candidate optimization list, the cost estimation function is further constructed according to the candidate optimization list.

15. A computational graph optimization apparatus, comprising: The device comprises: an obtaining module configured to obtain a first calculation graph, wherein the first calculation graph includes at least one first node, each first node corresponds to a first operator, each first operator is pre-configured with at least two candidate kernel functions, and the at least two candidate kernel functions are adapted to different types of hardware computing cores; a querying module configured to, for each first operator, determine a kernel function container list corresponding to the first operator according to attribute information of the first operator, and determine a target kernel function container from the kernel function container list corresponding to the first operator, wherein the kernel function container list includes at least two candidate kernel function containers, each candidate kernel function container includes at least one candidate kernel function pre-configured for the first operator, and the target kernel function container is a candidate kernel function container with the lowest execution time in the at least two candidate kernel function containers; an optimization module configured to determine whether to perform a local graph optimization operation on a first node corresponding to each first operator according to the target kernel function container corresponding to the first operator and the attribute information; for a to-be-optimized node determined to perform the local graph optimization operation, perform the local graph optimization operation according to the target kernel function container corresponding to the to-be-optimized node to obtain a second calculation graph; and A code generation module configured to generate executable code corresponding to the first computation graph according to the second computation graph.

16. An electronic device, comprising: The electronic device includes at least one processor and at least one memory, wherein the at least one memory has computer readable instructions stored therein, and the computer readable instructions, when executed by the at least one processor, cause the at least one processor to perform the computation graph optimization method of any one of claims 1-14.

17. A computer-readable storage medium, characterized in that, The computer readable storage medium has computer readable instructions stored thereon, and the computer readable instructions, when executed by at least one processor, cause the at least one processor to perform the computation graph optimization method of any one of claims 1-14.

Images