Computing graph optimization method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present disclosure provide a computing graph optimization method and device, electronic equipment and storage medium. The method comprises: obtaining a first computing graph, wherein each first node corresponds to a first operator pre-configured with at least two candidate kernel functions adapted to different types of hardware computing cores; determining a target kernel function container from a kernel function container list corresponding to the first operator according to attribute information of the first operator; determining whether to perform a local graph optimization operation on the first node corresponding to the first operator according to the target kernel function container and the attribute information corresponding to each first operator; for the to-be-optimized node determined to perform the local graph optimization operation, performing the local graph optimization operation according to the corresponding target kernel function container to obtain a second computing graph; and generating executable code corresponding to the first computing graph according to the second computing graph. The method can achieve an optimal balance between generalization and extreme performance, avoid introducing redundant computing overhead, and has strong flexibility.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of artificial intelligence technology, and in particular to a computational graph optimization method, a computational graph optimization apparatus, an electronic device, and a storage medium. Background Technology

[0002] In the field of Artificial Intelligence (AI) technology, the efficiency of AI frameworks in processing computation graphs directly affects model execution performance. Current mainstream AI frameworks generally adopt a processing flow of "global graph optimization first, then operator selection," which sequentially covers three stages: graph optimization, operator selection, and execution. Specifically, the graph optimization stage performs predefined general optimization operations such as constant folding, operator fusion, and layout transformation on the input computation graph. The operator selection stage matches specific kernel functions to each operator in the computation graph based on the shape and layout characteristics of the input data (such as tensors). The execution stage generates the final executable code based on the selected kernel functions; at this stage, the computation graph structure is fixed and cannot be modified secondaryly.

[0003] In AI frameworks, high-performance kernel functions (such as Tensor Cores for accelerated convolution) impose strict constraints on the shape, size, and arrangement of input data, requiring the insertion of specific operators (or auxiliary nodes encapsulating specific operators) to address these constraints. However, such optimization operations only provide a gain effect for specific kernel functions. Limited by the "global graph optimization first, operator selection later" processing flow, AI frameworks cannot predict the subsequently selected kernel function during the graph optimization stage. That is, performing graph optimization operations prematurely introduces redundant overhead for generalizable kernel function scenarios. Furthermore, once the kernel function is determined, the computation graph structure cannot be modified again to supplement targeted optimizations. This makes it difficult for AI frameworks to achieve the optimal balance between generalization and extreme performance. Summary of the Invention

[0004] At least one embodiment of this disclosure provides a computation graph optimization method, comprising: obtaining a first computation graph, wherein the first computation graph includes at least one first node, each first node corresponds to a first operator, each first operator is pre-configured with at least two candidate kernel functions, and the at least two candidate kernel functions are adapted to different types of hardware computing cores; for each first operator, determining a kernel function container list corresponding to the first operator based on the attribute information of the first operator, and determining a target kernel function container from the kernel function container list corresponding to the first operator, wherein the kernel function container list includes at least two candidate kernel function containers, each candidate kernel function container includes at least one candidate kernel function pre-configured by the first operator, and the target kernel function container is the candidate kernel function container with the lowest execution time among the at least two candidate kernel function containers; determining whether to perform a local graph optimization operation on the first node corresponding to the first operator based on the target kernel function container and attribute information corresponding to each first operator; performing a local graph optimization operation on the node to be optimized based on the target kernel function container corresponding to the node to be optimized to obtain a second computation graph; and generating executable code corresponding to the first computation graph based on the second computation graph.

[0005] For example, in the method provided in at least one embodiment of this disclosure, the hardware computing core adapted to a candidate kernel function has hardware operating constraints, and the candidate kernel function container further includes a candidate optimization list corresponding to the candidate kernel function. The candidate optimization list includes at least one operator for satisfying the hardware operating constraints of a candidate kernel function.

[0006] For example, in the method provided in at least one embodiment of this disclosure, the attribute information of the first operator includes at least: the operator type of the first operator and the shape information of the input data of the first operator, wherein the shape information includes at least one of the shape size of the input data and the layout format of the input data; determining a list of kernel function containers corresponding to the first operator based on the attribute information of the first operator, and determining a target kernel function container from the list of kernel function containers corresponding to the first operator, includes: determining a list of first kernel function containers corresponding to the first operator based on the operator type of the first operator; for each candidate kernel function container in the list of first kernel function containers, obtaining the execution time of the candidate kernel function container based at least on the shape information of the input data of the first operator and the candidate kernel functions in the candidate kernel function container; and selecting the candidate kernel function container with the lowest execution time as the target kernel function container of the first operator.

[0007] For example, in the method provided in at least one embodiment of this disclosure, the execution time of the candidate kernel function container includes: the time required for the adapted hardware computing core to execute the candidate kernel functions in the candidate kernel function container according to the attribute information of the first operator.

[0008] For example, in the method provided in at least one embodiment of this disclosure, in response to the candidate kernel function container further including a candidate optimization list corresponding to the candidate kernel function, and the shape information of the input data of the first operator not satisfying the hardware operation constraints of the adapted hardware computing core, the execution time of the candidate kernel function container further includes: the execution time of at least one operator in the candidate optimization list.

[0009] For example, in the method provided in at least one embodiment of this disclosure, each candidate kernel function container further includes a cost estimation function. The cost estimation function is used to determine the execution time of the corresponding candidate kernel function container, and the cost estimation function is configured at least according to the candidate kernel functions in the corresponding candidate kernel function container. For each candidate kernel function container in the first kernel function container list, the execution time of the candidate kernel function container is obtained at least according to the shape information of the input data of the first operator and the candidate kernel functions in the candidate kernel function container. This includes: inputting the shape information into the cost estimation function included in each candidate kernel function container in the first kernel function container list to obtain the execution time of each candidate kernel function container in the first kernel function container list.

[0010] For example, in the method provided in at least one embodiment of this disclosure, in response to the corresponding candidate kernel function container, a candidate optimization list corresponding to the candidate kernel function is also included, and the cost estimation function is further configured according to the candidate optimization list.

[0011] For example, in the method provided in at least one embodiment of this disclosure, the attribute information of the first operator includes at least the shape information of the input data of the first operator, and the hardware operation constraints include constraints on the shape information. Based on the target kernel function container and attribute information corresponding to each first operator, determining whether to perform a local graph optimization operation on the first node corresponding to the first operator includes: in response to the target kernel function container not including the candidate optimization list, determining not to perform a local graph optimization operation on the first node; in response to the target kernel function container including the candidate optimization list and the shape information satisfying the constraints, determining not to perform a local graph optimization operation on the first node; and in response to the target kernel function container including the candidate optimization list and the shape information not satisfying the constraints, determining to perform a local graph optimization operation on the first node and designating the first node as the node to be optimized.

[0012] For example, in the method provided in at least one embodiment of this disclosure, for a node to be optimized that is determined to perform a local graph optimization operation, a local graph optimization operation is performed according to the target kernel function container corresponding to the node to be optimized to obtain a second computation graph, including: for each node to be optimized, according to at least one operator included in the candidate optimization list for satisfying the hardware operation constraints of a candidate kernel function, at least one auxiliary node is added at the corresponding position adjacent to the node to be optimized in the first computation graph to obtain a second computation graph.

[0013] For example, at least one embodiment of this disclosure provides a method that further includes: establishing a binding relationship between a first operator and a target kernel function container; wherein, generating executable code corresponding to the first computation graph according to the second computation graph includes: traversing each node in the second computation graph; in response to the current operator corresponding to the current node not being the first operator, selecting a kernel function for the current operator based on the attribute information of the current operator; in response to the current operator corresponding to the current node being a first operator of any operator type, using the candidate kernel function in the target kernel function container bound to the first operator of any operator type as the kernel function of the current operator; and generating executable code corresponding to the first computation graph according to the kernel function of the operator corresponding to each node in the second computation graph.

[0014] For example, in a method provided in at least one embodiment of this disclosure, obtaining a first computation graph includes: obtaining an original computation graph; performing a general graph optimization operation on a second node in the original computation graph to obtain the first computation graph, wherein the operator corresponding to the second node is not the first operator.

[0015] For example, at least one embodiment of the method provided in this disclosure further includes: configuring corresponding kernel function container lists for first operators of different operator types.

[0016] For example, in the method provided in at least one embodiment of this disclosure, configuring corresponding kernel function container lists for first operators of different operator types includes: for any operator type in the first operator: constructing N candidate kernel function containers in the kernel function container list corresponding to any operator type, where N is an integer greater than or equal to 2, and the N candidate kernel function containers correspond to different hardware computing cores; encapsulating the N candidate kernel functions pre-configured for any operator type and implemented by different hardware computing cores into the N candidate kernel function containers respectively; for any candidate kernel function among the N candidate kernel functions, in response to the hardware computing core implementing any candidate kernel function having hardware operation constraints, configuring the candidate optimization list associated with any candidate kernel function in the corresponding candidate kernel function container, wherein the associated candidate optimization list includes at least one operator for satisfying the hardware operation constraints.

[0017] For example, at least one embodiment of the present disclosure provides a method that further includes: constructing a corresponding cost estimation function for each candidate kernel function container, wherein the cost estimation function is constructed based on the candidate kernel functions encapsulated in each candidate kernel function container, wherein, in response to the candidate kernel function container further including a candidate optimization list, the cost estimation function is also constructed based on the candidate optimization list.

[0018] At least one embodiment of this disclosure also provides a computational graph optimization apparatus, which includes: an acquisition module, a query module, an optimization module, and a code generation module. The acquisition module is configured to acquire a first computation graph, wherein the first computation graph includes at least one first node, each first node corresponds to a first operator, each first operator is pre-configured with at least two candidate kernel functions, and the at least two candidate kernel functions are adapted to different types of hardware computing cores; the query module is configured to, for each first operator, determine a list of kernel function containers corresponding to the first operator based on the attribute information of the first operator, and determine a target kernel function container from the list of kernel function containers corresponding to the first operator, wherein the list of kernel function containers includes at least two candidate kernel function containers, each candidate kernel function container includes at least one candidate kernel function pre-configured for the first operator, and the target kernel function container is the candidate kernel function container with the lowest execution time among the at least two candidate kernel function containers; the optimization module is configured to, based on the target kernel function container and attribute information corresponding to each first operator, determine whether to perform a local graph optimization operation on the first node corresponding to the first operator; for the node to be optimized that is determined to perform a local graph optimization operation, perform a local graph optimization operation based on the target kernel function container corresponding to the node to be optimized to obtain a second computation graph; and the code generation module is configured to, based on the second computation graph, generate executable code corresponding to the first computation graph.

[0019] At least one embodiment of this disclosure also provides an electronic device, which includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions that, when executed by the at least one processor, cause the at least one processor to perform the computational graph optimization method provided in any of the above embodiments.

[0020] At least one embodiment of this disclosure also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by at least one processor, cause the at least one processor to perform the computation graph optimization method provided in any of the above embodiments.

[0021] The computational graph optimization method provided in at least one embodiment of this disclosure innovatively proposes a dynamic graph optimization mechanism that supports kernel function awareness. This allows the AI ​​compiler in the AI ​​framework to automatically weigh the combined overhead of "generalized kernel function", "high-performance kernel function" and "high-performance kernel function + auxiliary nodes introduced by local graph optimization operations" during the graph optimization stage, based on the attribute information of the operators and the execution time of the candidate kernel function container. This predicts the kernel function to be selected in the subsequent operator selection stage (i.e., the candidate kernel function in the target kernel function container). Furthermore, during the graph optimization stage, based on the predicted target kernel function container and the attribute information of the operators, it can dynamically determine whether to perform targeted local graph optimization operations on the corresponding nodes. This allows for the insertion of operators (i.e., auxiliary nodes) in the computational graph that remove the hardware operation constraints of the actually selected kernel function, and avoids prematurely inserting invalid auxiliary nodes for "generalized kernel function" or "high-performance kernel function" that already meets the hardware operation constraints, thus avoiding the introduction of redundant computational overhead. Furthermore, this computation graph optimization method is compatible with current AI frameworks (such as TensorFlow, PyTorch, MindSpore, TVM, etc.) and can be integrated into AI frameworks as independent plug-in modules without refactoring the overall architecture or modifying the code compilation process, which is beneficial for expanding to new hardware backends. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0023] Figure 1 A flowchart of a computational graph optimization method according to at least one embodiment of the present disclosure is shown.

[0024] Figure 2 A flowchart illustrating a list of configuration kernel function containers according to at least one embodiment of the present disclosure is shown.

[0025] Figure 3 A schematic diagram illustrating the implementation of local graph optimization according to at least one embodiment of the present disclosure is shown.

[0026] Figure 4 A schematic block diagram of a computational graph optimization apparatus according to at least one embodiment of the present disclosure is shown.

[0027] Figure 5 A schematic block diagram of an electronic device according to at least one embodiment of the present disclosure is shown.

[0028] Figure 6 A schematic block diagram of a computer-readable storage medium provided according to at least one embodiment of the present disclosure is shown. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0030] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “including,” “comprising,” or “containing,” and similar terms mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. The terms “connected,” “linked,” or similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “upper,” “lower,” “left,” and “right,” etc., are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0031] Currently, artificial intelligence chips are developing rapidly and have been widely applied in various fields, from autonomous driving to medical diagnosis and smart homes. One of the core functions of artificial intelligence chips is to efficiently process neural network models. Neural network models can be represented by computational graphs. By optimizing and reconstructing the computational graphs, and mapping the optimized computational graphs into executable code, the compilation process of the neural network model is completed.

[0032] For example, a computational graph is typically organized as a Directed Acyclic Graph (DAG). A computational graph consists of nodes and edges. Nodes in a computational graph represent specific computational operations, such as convolution, matrix multiplication, and activation functions; each node corresponds to an operator. Edges in a computational graph represent data dependencies between nodes; for example, a tensor flows from the output of one operator to the input of another.

[0033] As mentioned earlier, current mainstream AI frameworks (such as TensorFlow, PyTorch, MindSpore, TVM, etc.) generally adopt a "global graph optimization first, operator selection later" processing flow for computation graphs. High-performance kernel functions (such as Tensor Cores for accelerated convolution) impose strict constraints on the shape, size, and layout of input data. For example, some high-performance kernel functions explicitly require input data (such as tensors) to use the NHWC layout format instead of the conventional NCHW layout format. These constraints can usually be relieved by inserting auxiliary nodes (i.e., nodes encapsulating layout transformation operators such as transpose, reorder, padding, or reshape operators, which can also be referred to as "transpose nodes," "reorder nodes," "padding nodes," and "reshape nodes" below). However, such optimization operations only have a gain value for specific high-performance kernel functions.

[0034] In the aforementioned processing flow, the graph optimization phase is executed before the operator selection phase. When performing global graph optimization, the AI ​​framework cannot predict whether the subsequent operator selection phase will adopt such a high-performance kernel function. This makes it difficult for the AI ​​framework to achieve the optimal balance between generalization and extreme performance.

[0035] For example, if a targeted optimization operation is performed on a node in the computational graph during the graph optimization phase (e.g., inserting a Reorder node or Padding node in adjacent positions before or after the node), and a generalized kernel function is selected for the operator encapsulated in the node during the subsequent operator selection phase, the optimization operation introduces redundant computational overhead because the generalized kernel function does not have specific constraints on the shape, size, or layout of the input data.

[0036] For example, if a specific optimization operation is not performed on a node in the computation graph during the graph optimization phase, and a high-performance kernel function is selected for the operator encapsulated in that node in the subsequent operator selection phase, the computation graph will be frozen after the operator selection phase ends. The AI ​​framework will then be unable to insert a node to satisfy the constraints based on the selected high-performance kernel function. This will cause the high-performance kernel function to be executed under a non-ideal data arrangement or shape, resulting in a significant reduction in computational efficiency.

[0037] To address the above shortcomings, this disclosure provides one or more embodiments of a computational graph optimization method, a computational graph optimization apparatus, an electronic device, a storage medium, and a program product. The computation graph optimization method includes: obtaining a first computation graph, wherein the first computation graph includes at least one first node, each first node corresponds to a first operator (or each first node encapsulates a first operator), each first operator is pre-configured with at least two candidate kernel functions, and the at least two candidate kernel functions are adapted to different types of hardware computing cores; for each first operator, based on the attribute information of the first operator, determining a list of kernel function containers corresponding to the first operator, and determining a target kernel function container from the list of kernel function containers corresponding to the first operator, wherein the list of kernel function containers includes at least two candidate kernel function containers, each candidate kernel function container includes at least one candidate kernel function pre-configured by the first operator, and the target kernel function container is the candidate kernel function container with the lowest execution time among the at least two candidate kernel function containers; based on the target kernel function container and attribute information corresponding to each first operator, determining whether to perform a local graph optimization operation on the first node corresponding to the first operator; for the node to be optimized that is determined to perform a local graph optimization operation, performing a local graph optimization operation based on the target kernel function container corresponding to the node to be optimized to obtain a second computation graph; and generating executable code corresponding to the first computation graph based on the second computation graph.

[0038] The computational graph optimization method provided in at least one embodiment of this disclosure innovatively proposes a dynamic graph optimization mechanism that supports kernel function awareness. This allows the AI ​​compiler in the AI ​​framework to automatically weigh the combined overhead of "generalized kernel function", "high-performance kernel function" and "high-performance kernel function + auxiliary nodes introduced by local graph optimization operations" during the graph optimization stage, based on the attribute information of the operators and the execution time of the candidate kernel function container. This predicts the kernel function to be selected in the subsequent operator selection stage (i.e., the candidate kernel function in the target kernel function container). Furthermore, during the graph optimization stage, based on the predicted target kernel function container and the attribute information of the operators, it can dynamically determine whether to perform targeted local graph optimization operations on the corresponding nodes. This allows for the insertion of operators (i.e., auxiliary nodes) in the computational graph that remove the hardware operation constraints of the actually selected kernel function, and avoids prematurely inserting invalid auxiliary nodes for "generalized kernel function" or "high-performance kernel function" that already meets the hardware operation constraints, thus avoiding the introduction of redundant computational overhead.

[0039] Furthermore, the computation graph optimization method provided in at least one embodiment of this disclosure is compatible with current AI frameworks (such as TensorFlow, PyTorch, MindSpore, TVM, etc.), and can be integrated into AI frameworks as independent plug-in modules without reconstructing the overall architecture or modifying the code compilation process, which is beneficial for expanding new hardware backends.

[0040] The present disclosure will now be described through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and components may be omitted. When any component of the embodiments of the present disclosure appears in more than one drawing, the component is represented by the same or similar reference numerals in each drawing.

[0041] The following detailed description, with reference to the accompanying drawings, describes some embodiments and examples of this disclosure.

[0042] Figure 1 A flowchart of a computational graph optimization method according to at least one embodiment of the present disclosure is shown.

[0043] like Figure 1 As shown, the computational graph optimization method provided in at least one embodiment of this disclosure includes steps S110 to S150.

[0044] Step S110: Obtain a first computation graph, wherein the first computation graph includes at least one first node, each first node corresponds to a first operator, each first operator is pre-configured with at least two candidate kernel functions, and the at least two candidate kernel functions are adapted to different types of hardware computing cores respectively.

[0045] Step S120: For each first operator, determine the kernel function container list corresponding to the first operator based on the attribute information of the first operator, and determine the target kernel function container from the kernel function container list corresponding to the first operator. The kernel function container list includes at least two candidate kernel function containers, each candidate kernel function container includes at least one candidate kernel function pre-configured by the first operator, and the target kernel function container is the candidate kernel function container with the lowest execution time among the at least two candidate kernel function containers.

[0046] Step S130: Based on the target kernel function container and attribute information corresponding to each first operator, determine whether to perform local graph optimization operation on the first node corresponding to the first operator.

[0047] Step S140: For the node to be optimized that is determined to perform local graph optimization operation, perform local graph optimization operation according to the target kernel function container corresponding to the node to be optimized to obtain the second computation graph.

[0048] Step S150: Generate executable code corresponding to the first computation graph based on the second computation graph.

[0049] The computational graph optimization method provided in this disclosure can be applied to various scenarios.

[0050] For example, this computation graph optimization method can be applied to an AI compiler (e.g., as a functional module of the AI ​​compiler). When the AI ​​compiler compiles high-level models (e.g., neural network models) into executable code, it can optimize the computation graph corresponding to the high-level models, thereby achieving more efficient hardware resource utilization, lower inference latency, and better energy efficiency.

[0051] For example, this computation graph optimization method can also be applied to the AI ​​processing module of smartphones (e.g., as a built-in function of the NPU in a mobile phone chip). When a smartphone is running AI applications such as photo beautification and real-time translation, the computation graph behind the application can be optimized at the same time, so that these functions can start up faster and run more power-efficiently.

[0052] For example, this computational graph optimization method can also be applied to home smart cameras. When the camera performs human motion detection and abnormal behavior recognition, the computational graph of the detection algorithm can be optimized, thereby reducing camera power consumption while ensuring the real-time performance and accuracy of the recognition results. This disclosure does not impose any limitations on this.

[0053] For example, the hardware computing cores in this disclosure include, but are not limited to, vector processing cores, tensor processing cores, and neural network processing cores (AI Core / NPU Core). Vector processing cores possess good versatility (generalization), supporting various data layout formats and shapes, but their computational performance is relatively low. Tensor processing cores offer high computational performance, but their generalization is poor; that is, they are only applicable to certain operator types and have hardware constraints on the layout format, shape, or alignment of the input data. For example, the Tensor Core requires the input data to use the NHWC layout format and the channel dimensions to be aligned to multiples of 8. Neural network processing cores are dedicated computing units customized for deep learning inference / training tasks, exhibiting excellent performance in suitable scenarios, but they also have hardware constraints on the computation graph structure, data layout format, channel alignment, batch size, etc.

[0054] For example, in this embodiment, the first node refers to a node whose corresponding operator has multiple kernel implementations, each of which is adapted to different types of hardware computing cores. For example, the first node can be a convolution node, in which case its corresponding first operator is a convolution (Conv) operator, which is pre-configured with at least two candidate kernel functions (e.g., Kernel_1 and Kernel_2). Candidate kernel function Kernel_1 can be executed by a vector core, and candidate kernel function Kernel_2 can be executed by a tensor core. For example, the first node can also be a matrix multiplication (MMA) node; this disclosure does not limit this.

[0055] For example, the first node in the first computation graph can be manually specified. For example, the Conv node and the MMA node in the first computation graph can be specified as the "first node", and this disclosure does not limit this.

[0056] For example, the step of obtaining the first computation graph in this embodiment may include: obtaining the original computation graph (i.e., the computation graph without graph optimization operation); performing a general graph optimization operation on the second node in the original computation graph to obtain the first computation graph, wherein the operator corresponding to the second node is not the first operator, that is, the second node is another node in the first computation graph other than the first node.

[0057] For example, the first computation graph in this embodiment can also be the original computation graph. In this case, after performing local graph optimization operations on the nodes to be optimized in the first computation graph (i.e., the first node in the first computation graph to be subjected to local graph optimization operations), general graph optimization operations are also performed on the other nodes in the first computation graph other than the first node, and then a second computation graph is obtained.

[0058] It is important to note that the "local graph optimization operation" in this disclosure refers to a graph optimization operation that targets a single first node, with the aim of adapting to the hardware operational constraints (such as data layout format, shape alignment, etc.) of a specific hardware computing core. After performing the local graph optimization operation, the total number of nodes in the first computation graph remains unchanged or increases. For example, the local graph optimization operation may include: inserting reorder nodes, padding nodes, or cast nodes in the adjacent positions before and after the target first node, etc., to generate input data that satisfies the execution conditions of the high-performance kernel function (i.e., satisfies the hardware operational constraints of the hardware computing core on the input data).

[0059] In contrast, the "general graph optimization operation" in this disclosure refers to a graph optimization operation that targets one or more second nodes that are not first nodes, with the aim of improving overall computational efficiency, reducing redundant computations, or lowering kernel startup overhead. After performing the general graph optimization operation, the total number of nodes in the computation graph may increase, decrease, or remain unchanged. For example, the general graph optimization operation may include operator fusion, constant folding, dead code elimination, etc.

[0060] For step S120 above, for each first operator in the first computation graph, a list of kernel function containers corresponding to the first operator is determined based on the attribute information of the first operator, and the candidate kernel function container with the lowest execution time is selected from the list of kernel function containers corresponding to the first operator as the target kernel function container of the first operator.

[0061] For example, in at least one embodiment of this disclosure, the attribute information of the first operator includes at least the operator type of the first operator and the shape information of the input data of the first operator, wherein the shape information includes at least one of the shape size of the input data and the layout format of the input data.

[0062] For example, the operator types of the first operator include, but are not limited to: matrix multiplication (MMA) operator, convolution (Conv) operator, Fourier transform (FFT) operator, etc.

[0063] For example, input data can be tensors, which can be represented as N×C×H×W, where N represents the batch size, C represents the number of channels, H represents the height of the tensor, and W represents the width of the tensor. The shape and size of a tensor (also called "dimensional dimensions") are determined by the magnitude of its dimensions (i.e., the values ​​of N, C, H, and W). Tensor layout formats include, but are not limited to, "NCHW", "NHWC", and "NC / xHWx (interleaving mode)", where x can take values ​​such as 8, 16, and 32 as needed. NC / xHWx is similar to NHWC, but the difference is that in the memory layout of NC / xHWx, the C channels are divided into C / x groups, with x channels in each group: the first group consists of channels c=0 to c=x-1, the second group consists of channels c=x to c=2x-1, and each group is arranged in NHWC format.

[0064] For example, in at least one embodiment of this disclosure, a kernel function container represents a container that encapsulates at least one kernel function, and each kernel function container encapsulates only one kernel function. A first operator is pre-configured with at least two candidate kernel functions, and the kernel function container list corresponding to the first operator includes at least two candidate kernel function containers.

[0065] For example, in at least one embodiment of this disclosure, in response to the hardware computing core adapted to the candidate kernel function encapsulated in any candidate kernel function container having hardware operation constraints, the candidate kernel function container further includes a candidate optimization list corresponding to the candidate kernel function, and the candidate optimization list includes at least one operator for satisfying the hardware operation constraints.

[0066] For example, operators included in the candidate optimization list can be called "auxiliary operators" (e.g., reorder operators, padding operators, etc.), and the nodes corresponding to the auxiliary operators are also called "auxiliary nodes." By inserting auxiliary nodes corresponding to the auxiliary operators at the corresponding positions in the computation graph, data transformation operations are performed on the input data of the first operator, thereby satisfying the hardware operation constraints of the hardware computing core (e.g., Tensor Core) on the data arrangement format or shape and size of the input data, thus ensuring that the selected high-performance kernel function can be executed correctly and efficiently.

[0067] For example, the execution time of the candidate kernel function container includes the time required for the adapted hardware computing core to execute the candidate kernel functions in the candidate kernel function container based on the attribute information of the first operator. The execution time of this candidate kernel function container is the predicted execution time.

[0068] For example, in response to the candidate kernel function container also including a candidate optimization list corresponding to the candidate kernel function, and the shape information of the input data of the first operator does not meet the hardware operation constraints of the adapted hardware computing core, the execution time of the candidate kernel function container further includes: the execution time of at least one operator in the candidate optimization list.

[0069] For example, before performing step S120, the computation graph optimization method provided in at least one embodiment of this disclosure may further include: configuring corresponding kernel function container lists for first operators of different operator types.

[0070] For example, Figure 2 A flowchart illustrating a list of configuration kernel function containers according to at least one embodiment of the present disclosure is shown.

[0071] like Figure 2 As shown, for any type of operator in the first operator, the specific configuration steps for the corresponding kernel function container list may include steps S210 to S230.

[0072] In step S210, N candidate kernel function containers are constructed from the kernel function container list corresponding to any operator type, where N is an integer greater than or equal to 2, and the N candidate kernel function containers correspond to different hardware computing cores.

[0073] In step S220, N candidate kernel functions pre-configured by operators of any operator type and implemented by different hardware computing cores are encapsulated into N candidate kernel function containers.

[0074] In step S230, for any one of the N candidate kernel functions, in response to the hardware computing core implementing any one candidate kernel function having hardware operation constraints, the candidate optimization list associated with any one candidate kernel function is configured in the corresponding candidate kernel function container, wherein the associated candidate optimization list includes at least one operator for satisfying the hardware operation constraints.

[0075] The following example, using the first operator as the Conv operator and the Conv operator pre-configured with two candidate kernel functions (Kernel_1 and Kernel_2), illustrates the configuration process of the kernel function container list.

[0076] For example, in step S210 above, since the two candidate kernel functions Kernel_1 and Kernel_2 pre-configured by the Conv operator are adapted to the hardware computing core Vector Core and the hardware computing core Tensor Core respectively, two candidate kernel function containers (e.g., Conv_KernelEngine_List()) are constructed in the kernel function container list corresponding to the Conv operator (e.g., Conv_KernelEngine_List()).

[0077] For example, in step S220 above, the candidate kernel function Kernel_1 is encapsulated into the candidate kernel function container Conv_KernelEngine_1(), and the candidate kernel function Kernel_2 is encapsulated into the candidate kernel function container Conv_KernelEngine_2().

[0078] For example, in step S230 above, since the hardware computing core VectorCore of candidate kernel function Kernel_1 has no hardware operating constraints, it is not necessary to configure the corresponding candidate optimization list in the candidate kernel function container Conv_KernelEngine_1(). Since the hardware computing core Tensor Core of candidate kernel function Kernel_2 has hardware operating constraints (e.g., requiring the shape and size of the input data of the Conv operator to be a multiple of 16 and the layout format to be NHWC), the candidate optimization list associated with candidate kernel function Kernel_2 (e.g., PassList()) is configured in the candidate kernel function container Conv_KernelEngine_2(). The PassList() associated with candidate kernel function Kernel_2 includes a Padding operator (used to satisfy the hardware operating constraint "the shape and size of the input data of the Conv operator is a multiple of 16") and a Reorder operator (used to satisfy the hardware operating constraint "the layout format of the input data of the Conv operator is NHWC").

[0079] For example, after executing steps S210, S220, and S230, the kernel function container list for the Conv operator can be represented as: Conv_KernelEngine_List{ Conv_KernelEngine_1(Kernel_1 ), Conv_KernelEngine_2( Kernel_2, PassList( AutoReoder(),AutoPadding() ) )}. "AutoReoder()" represents the optimization function corresponding to the Reoder operator. By calling AutoReoder() in the kernel function container list, the auxiliary node (i.e., the Reoder node) corresponding to the Reoder operator can be automatically inserted at the corresponding position adjacent to the Conv node. "AutoPadding()" represents the optimization function corresponding to the Padding operator. By calling AutoPadding() in the kernel function container list, the auxiliary node (i.e., the Padding node) corresponding to the Padding operator can be automatically inserted at the corresponding position adjacent to the Conv node.

[0080] For example, the configuration process of the kernel function container list corresponding to the first operator of other operator types (e.g., the MMA operator) can refer to the configuration process of the kernel function container list Conv_KernelEngine_List corresponding to the Conv operator mentioned above, and will not be repeated here.

[0081] For example, a specific implementation of step S120 above may include: determining a list of first kernel function containers corresponding to the first operator based on the operator type of the first operator; for each candidate kernel function container in the list of first kernel function containers, obtaining the execution time of the candidate kernel function container based at least on the shape information of the input data of the first operator and the candidate kernel functions in the candidate kernel function container; and selecting the candidate kernel function container with the lowest execution time as the target kernel function container of the first operator.

[0082] The process of determining the kernel target kernel function container is described below through specific examples.

[0083] In one embodiment of this disclosure, the first operator is a Conv operator. The input data of the Conv operator is a one-dimensional tensor with a shape size of 1. The kernel function container list corresponding to the Conv operator is Conv_KernelEngine_List{ Conv_KernelEngine_1( Kernel_1 ), Conv_KernelEngine_2(Kernel_2, PassList( AutoPadding() ) )} (see above for details). The hardware computing core of the candidate kernel function Kernel_1 is a Vector Core, and the hardware operation constraint of the hardware computing core TensorCore of the candidate kernel function Kernel_2 is that "the shape size of the input data of the Conv operator must be a multiple of 16". The computing speed of the TensorCore is 10 times that of the Vector Core.

[0084] For example, for the candidate kernel function container Conv_KernelEngine_1(Kernel_1), the hardware computation core Vector Core of the candidate kernel function Kernel_1 has good generalization and therefore has no hardware execution constraints. Thus, the execution time of the candidate kernel function container Conv_KernelEngine_1 only includes the time required for Vector Core to execute the candidate kernel function Kernel_1 based on the attribute information of the first operator. For example, if the Shape of the input data of the first operator is 1, the computational cost of VectorCore can be expressed as vcore(Shape) = vcore(1) = 1.

[0085] For example, for the candidate kernel function container Conv_KernelEngine_2( Kernel_2, PassList(AutoPadding() ) ), since the Shape(1) of the input data of the Conv operator does not meet the hardware operation constraints of the TensorCore (i.e., the Shape is required to be a multiple of 16), the execution time of the candidate kernel function container Conv_KernelEngine_2, in addition to the time required for TensorCore to execute Kernel_2, also includes the execution time of the Padding operator in the candidate optimization list PassList. Specifically, the execution time of the Padding operator includes: the time required for the Padding operator to adjust the Shape of the input data from 1 to 16 before TensorCore executes Kernel_2; and the time required for the Padding operator to restore the Shape of the input data from 16 to 1 after TensorCore executes Kernel_2. For example, the computational cost of a Tensor Core can be expressed as: padding(1->16) + tcore(16) + padding(16 ->1) = 15 + 16 / 10 + 15 = 31.6.

[0086] For example, if the time required for the hardware computing core to execute the corresponding candidate kernel function is proportional to the amount of computation, and the time base required for the Vector Core and Tector Core to execute a unit of computation is the same (e.g., both are T), the execution time of the candidate kernel function container Conv_KernelEngine_1 can be expressed as 1T, and the execution time of the candidate kernel function container Conv_KernelEngine_2 can be expressed as 31.6T. In this embodiment, the candidate kernel function container Conv_KernelEngine_1 is selected as the target kernel function container of the Conv node.

[0087] In another embodiment of this disclosure, the first operator is a Conv operator. The input data of the Conv operator is a one-dimensional tensor with a shape of 15. The kernel function container list corresponding to the Conv operator is Conv_KernelEngine_List{ Conv_KernelEngine_1( Kernel_1 ), Conv_KernelEngine_2(Kernel_2, PassList( AutoPadding() ) )}. The hardware computing core of the candidate kernel function Kernel_1 is a Vector Core, and the hardware operation constraint of the hardware computing core Tensor Core of the candidate kernel function Kernel_2 is that "the shape of the input data of the Conv operator must be a multiple of 16". The computing speed of the Tensor Core is 10 times that of the Vector Core.

[0088] For example, for the candidate kernel function container Conv_KernelEngine_1(Kernel_1), the shape of the input data of the first operator is 15, and the computational cost of the Vector Core adapted by Kernel_1 can be expressed as vcore(Shape)=15.

[0089] For example, for the candidate kernel function container Conv_KernelEngine_2( Kernel_2, PassList(AutoPadding() ) ), since the Shape (15) of the input data of the Conv operator does not satisfy the hardware operation constraints of the Tensor Core of Kernel_2 (i.e., the Shape is required to be a multiple of 16), the execution time of the candidate kernel function container Conv_KernelEngine_2 further includes the execution time of the Padding operator in the candidate optimization list PassList, in addition to the time required for the Tensor Core to execute Kernel_2. For example, the computational cost of the Tensor Core can be expressed as: padding(15->16) + tcore(16) + padding (16 ->15) = 1 + 16 / 10 + 1 = 3.6.

[0090] For example, if the time required for the hardware computing core to execute the corresponding candidate kernel function is proportional to the amount of computation, and the time base required for the Vector Core and Tector Core to execute a unit of computation is the same (e.g., both are T), the execution time of the candidate kernel function container Conv_KernelEngine_1 can be expressed as 15T, and the execution time of the candidate kernel function container Conv_KernelEngine_2 can be expressed as 3.6T. In this embodiment, the candidate kernel function container Conv_KernelEngine_2 is selected as the target kernel function container of the Conv node.

[0091] In another embodiment of this disclosure, the first operator is a Conv operator. The input data of the Conv operator is a tensor N×C×H×W with an arrangement format of NCHW. The kernel function container list corresponding to the Conv operator is Conv_KernelEngine_List{ Conv_KernelEngine_1( Kernel_1 ), Conv_KernelEngine_2(Kernel_2, PassList( AutoPadding(), AutoReorder() ) )}. The hardware computing core of the candidate kernel function Kernel_1 is a Vector Core, and the hardware operation constraint of the hardware computing core TensorCore of the candidate kernel function Kernel_2 is that "the shape and size of the input data of the Conv operator must be a multiple of 8, and the arrangement format is NHWC". The operation speed of the TensorCore is 10 times that of the Vector Core.

[0092] For example, for the candidate kernel function container Conv_KernelEngine_1(Kernel_1), the shape of the input data of the first operator is N×C×H×W, and the computational cost of the Vector Core adapted by Kernel_1 can be expressed as vcore(Shape)=N×C×H×W.

[0093] For example, for the candidate kernel function container Conv_KernelEngine_2( Kernel_2, PassList(AutoPadding(), AutoReorder() ) ), if the shape (N×C×H×W) of the input data of the Conv operator satisfies the "requires the shape to be a multiple of 8" requirement in the Tensor Core's hardware operation constraints, but the layout format (NCHW) of the input data of the Conv operator does not satisfy the "requires the layout format of the input data of the Conv operator to be NHWC" requirement in the Tensor Core's hardware operation constraints, then in addition to the time required for the Tensor Core to execute Kernel_2, the execution time of the candidate kernel function container Conv_KernelEngine_2 also includes the execution time of the Reorder operator in the candidate optimization list PassList. For example, the computational cost of a Tensor Core can be expressed as: reorder(NCHW->NHWC) + tcore(NHWC) + reorder (NHWC->NCHW) = N×C×H×W + N×C×H×W / 10 + N×C×H×W = 2.1×(N×C×H×W).

[0094] For example, if the time required for the hardware computing core to execute the corresponding candidate kernel function is proportional to the amount of computation, and the time base required for the Vector Core and Tector Core to execute a unit of computation is the same (e.g., both are T), the execution time of the candidate kernel function container Conv_KernelEngine_1 can be expressed as 1T×(N×C×H×W), and the execution time of the candidate kernel function container Conv_KernelEngine_2 can be expressed as 2.1T×(N×C×H×W). In this embodiment, the candidate kernel function container Conv_KernelEngine_1 is selected as the target kernel function container of the Conv node.

[0095] For example, in at least one embodiment of this disclosure, a cost estimation function can be used to determine the execution time of a corresponding candidate kernel function container, and the cost estimation function is configured at least according to the candidate kernel functions in the corresponding candidate kernel function container.

[0096] For example, cost estimation functions can be configured (encapsulated) within corresponding candidate kernel function containers, enabling these containers to return estimated execution times for their associated candidate kernel functions when queried. Alternatively, cost estimation functions can be centrally managed within a separate cost estimation model module, which dynamically calculates the execution time of each candidate kernel function container based on operator attribute information (e.g., the shape and format of the input data). Or, cost estimation functions can be cached in a runtime database as profiling data, allowing the measured or fitted execution times of each candidate kernel function container to be retrieved during deployment via table lookups. This disclosure does not impose any limitations on these methods.

[0097] For example, if the candidate kernel function containers in the first kernel function container list include a cost estimation function, when performing the step of obtaining the execution time of each candidate kernel function container in the first kernel function container list based at least on the shape information of the input data of the first operator and the candidate kernel functions in the candidate kernel function containers, the shape information can be input into the cost estimation function included in each candidate kernel function container in the first kernel function container list, and the cost estimation function returns the calculated execution time of the corresponding candidate kernel function container.

[0098] For example, in response to the corresponding candidate kernel function container, there is also a candidate optimization list corresponding to the candidate kernel function, and the cost estimation function is further configured according to the candidate optimization list.

[0099] For example, for the candidate kernel function container Conv_KernelEngine_1(Kernel_1), the hardware computing core adapted to the candidate kernel function Kernel_1 is Vector Core. Then, the cost estimation function Cost_1 set in this kernel function container can be configured according to the time required for Vector Core to execute the candidate kernel function Kernel_1. For example, Cost_1 = T1 × (vcore(Shape)), where T1 is the time base required for Vector Core to perform a unit of computation.

[0100] For example, in response to the corresponding candidate kernel function container, there is also a candidate optimization list corresponding to the candidate kernel function, and the cost estimation function is further configured according to the candidate optimization list.

[0101] For example, for the candidate kernel function container Conv_KernelEngine_2( Kernel_2, PassList(AutoPadding() ) ), the hardware computing core adapted to the candidate kernel function Kernel_2 is Tensor Core. The hardware running constraint of Tensor Core is "the input data shape must be a multiple of 16". Then the cost estimation function Cost_2 set in this kernel function container can be configured according to the time required for Tensor Core to execute the candidate kernel function Kernel_2 and the execution time of the Padding operator in the candidate optimization list. For example, Cost_2 = T2 × (padding(Shape->16) + tcore(16) + padding (16 -> Shape)), where T2 is the time base required for Tensor Core to execute a unit of computation.

[0102] It should be noted that the configuration of the above cost estimation function is merely illustrative and does not limit the technical solution disclosed herein.

[0103] For example, other methods can be used to predict execution time, and this disclosure does not impose specific restrictions on this.

[0104] In determining the execution time of the prediction, only cost prediction is performed without modifying the graph structure. After determining the target kernel function container, the computation graph structure is modified in a targeted manner through local graph optimization operations.

[0105] For step S130 above, it can be determined whether to perform local graph optimization operation on the first node corresponding to the first operator based on whether the hardware computing core adapted to the candidate kernel function in the target kernel function container has hardware operation constraints, and if there are hardware operation constraints, it can be determined whether the attribute information of the first operator satisfies the hardware operation constraints.

[0106] For example, when the attribute information of the first operator includes at least the shape information of the input data of the first operator, and the hardware operation constraints include constraints on the shape information, the specific implementation of the above step S130 may include: in response to the target kernel function container not including the candidate optimization list, determining not to perform local graph optimization operation on the first node; in response to the target kernel function container including the candidate optimization list and the shape information satisfying the constraints, determining not to perform local graph optimization operation on the first node; and in response to the target kernel function container including the candidate optimization list and the shape information not satisfying the constraints, determining to perform local graph optimization operation on the first node, and designating the first node as the node to be optimized.

[0107] For example, in response to the fact that the target kernel function container corresponding to the Conv operator (e.g., Conv_KernelEngine_1(Kernel_1)) does not include a candidate optimization list, it is determined that no local graph optimization operation will be performed on the Conv node corresponding to the Conv operator.

[0108] For example, if the target kernel function container corresponding to the Conv operator (e.g., Conv_KernelEngine_2(Kernel_2, PassList( AutoPadding() ) )) includes a list of candidate optimizations and the shape information satisfies the constraints (e.g., the shape of the input data of the Conv operator is 16, and the constraint is "the shape of the input data must be a multiple of 16"), then it is determined not to perform local graph optimization operations on the nodes corresponding to the Conv operator.

[0109] For example, if the target kernel function container corresponding to the Conv operator (e.g., Conv_KernelEngine_2(Kernel_2, PassList( AutoPadding() ) )) includes a candidate optimization list and the shape information does not meet the constraints (e.g., the shape of the input data of the Conv operator is 1, and the constraint is "the shape of the input data must be a multiple of 16"), then it is determined to perform a local graph optimization operation on the node corresponding to the Conv operator, and the node corresponding to the Conv operator is taken as the node to be optimized.

[0110] For example, a specific implementation of step S140 above may include: for each node to be optimized, according to at least one operator included in the candidate optimization list for satisfying the hardware operation constraints of a candidate kernel function, adding at least one auxiliary node at the corresponding position adjacent to the node to be optimized in the first computation graph to obtain a second computation graph.

[0111] For example, the node to be optimized is the first node determined in step S130 to perform the local graph optimization operation.

[0112] For example, in this disclosure, "auxiliary nodes" refer to non-original computational logic nodes that are dynamically inserted during the execution of local graph optimization operations to satisfy specific constraints (such as hardware operation constraints). Auxiliary nodes themselves do not change the mathematical semantics of the model and are only used for data preprocessing or post-processing. For example, auxiliary nodes may include: reorder nodes, padding nodes, cast nodes, etc., and this disclosure does not impose any limitations on them.

[0113] For example, since the local graph optimization operation in this embodiment is for a single first node, the insertion position of the auxiliary node is only related to the node to be optimized that it serves, without needing to consider the next node of the node to be optimized.

[0114] For ease of description, the candidate kernel function and the candidate optimization list in the target kernel function container corresponding to the node to be optimized will be referred to as the "target kernel function" and the "target optimization list", respectively.

[0115] For example, Figure 3 A schematic diagram illustrating the implementation of local graph optimization according to at least one embodiment of the present disclosure is shown.

[0116] like Figure 3 As shown, the first computation graph includes six nodes, from node A to node F. After steps S110 to S130, node B (e.g., a Conv node) is determined as the first node to be optimized (i.e., the node to be optimized) for local graph optimization. The target optimization list for node B is PassList(AutoPadding(), AutoReorder()). The hardware computing core's constraint on node B is that "the input data shape must be a multiple of 16, and the input data layout format is NHWC." The input data of node B has a shape of 1 and a layout format of NHWC. In this case, the target optimization list includes Padding and Reorder operators to satisfy the constraints. Since the input data of node B does not satisfy the constraint that "the input data shape must be a multiple of 16," a Padding node can be inserted in the first computation graph at the positions before and after node B (e.g., ...). Figure 3 As shown in the figure, the shape of the input data is expanded to a multiple of 16 before entering node B, and the shape of the data output by node B is restored to 1 after leaving node B.

[0117] In step S150 above, the second computation graph is obtained by performing local graph optimization operations on the nodes to be optimized in the first computation graph. The second computation graph has stronger hardware adaptability. Based on the second computation graph after local graph optimization, a code generation operation is performed to generate executable code that is functionally equivalent to the first computation graph and adapted to the target hardware computing core.

[0118] For example, the generated executable code includes, but is not limited to, device-specific kernel function call sequences, memory management instructions, and execution scheduling logic, which can be deployed on hardware platforms such as graphics processing units (GPUs), neural network processing units (NPUs), or deep learning processing units (DPUs) to perform inference or training tasks.

[0119] For example, after determining the target kernel function container for each first operator in the first computation graph, the computation graph optimization method provided in at least one embodiment of this disclosure further includes: establishing a binding relationship between the first operator and the determined target kernel function container. For example, regardless of whether a local graph optimization operation is performed on the first node, after determining the target kernel function container for each first operator in the first computation graph, the first operator is bound to the determined target kernel function container, so that in the subsequent operator selection stage, the candidate kernel function in the target kernel function container can be directly selected as the kernel function implementation of the first operator.

[0120] For example, through the established binding relationship, the following steps can be performed in the operator selection phase and the execution phase to achieve the above step S150: traverse each node in the second computation graph; in response to the current operator corresponding to the current node not being the first operator, select a kernel function for the current operator based on the attribute information of the current operator; in response to the current operator corresponding to the current node being the first operator of any operator type, use the candidate kernel function in the target kernel function container bound to the first operator of any operator type as the kernel function of the current operator; generate the executable code corresponding to the first computation graph based on the kernel function of the operator corresponding to each node in the second computation graph.

[0121] For example, for operators that are not the first operator (such as addition operators, activation function operators, etc.), their default general kernel functions can be used directly. For example, a specific kernel function implementation can be selected for the operator based on the tensor shape information of the operator. Of course, other methods can also be used for selection, and this disclosure does not limit this. For example, for the first operator of any operator type, the candidate kernel functions in its target kernel function container can be used as the specific kernel function implementation of the operator. In this way, the selection of kernel function implementations for the operators corresponding to each node in the second computation graph is completed.

[0122] Then, based on the kernel function selected by the operator corresponding to each node in the second computation graph, executable code corresponding to the first computation graph (or the original computation graph) can be generated to complete the entire graph optimization process.

[0123] The computational graph optimization method provided in at least one embodiment of this disclosure innovatively proposes a dynamic graph optimization mechanism that supports kernel function awareness. This allows the AI ​​compiler in the AI ​​framework to automatically weigh the combined overhead of "generalized kernel function", "high-performance kernel function" and "high-performance kernel function + auxiliary nodes introduced by local graph optimization operations" during the graph optimization stage, based on the attribute information of the operators and the execution time of the candidate kernel function container. This predicts the kernel function to be selected in the subsequent operator selection stage (i.e., the candidate kernel function in the target kernel function container). Furthermore, during the graph optimization stage, based on the predicted target kernel function container and the attribute information of the operators, it can dynamically determine whether to perform targeted local graph optimization operations on the corresponding nodes. This allows for the insertion of operators (i.e., auxiliary nodes) in the computational graph that remove the hardware operation constraints of the actually selected kernel function, and avoids prematurely inserting invalid auxiliary nodes for "generalized kernel function" or "high-performance kernel function" that already meets the hardware operation constraints, thus avoiding the introduction of redundant computational overhead.

[0124] Furthermore, this computation graph optimization method is compatible with current AI frameworks (such as TensorFlow, PyTorch, MindSpore, TVM, etc.) and can be integrated into AI frameworks as independent plug-in modules without refactoring the overall architecture or modifying the code compilation process, which is beneficial for expanding to new hardware backends.

[0125] For example, Figure 4 A schematic block diagram of a computational graph optimization apparatus 400 according to at least one embodiment of the present disclosure is shown.

[0126] like Figure 4 As shown, the computational graph optimization device 400 includes an acquisition module 410, a query module 420, an optimization module 430, and a code generation module 440.

[0127] The acquisition module 410 is configured to acquire a first computation graph, wherein the first computation graph includes at least one first node, each first node corresponds to a first operator, each first operator is pre-configured with at least two candidate kernel functions, and the at least two candidate kernel functions are adapted to different types of hardware computing cores respectively.

[0128] The query module 420 is configured to, for each first operator, determine the kernel function container list corresponding to the first operator based on the attribute information of the first operator, and determine the target kernel function container from the kernel function container list corresponding to the first operator. The kernel function container list includes at least two candidate kernel function containers, each candidate kernel function container includes at least one candidate kernel function pre-configured by the first operator, and the target kernel function container is the candidate kernel function container with the lowest execution time among the at least two candidate kernel function containers.

[0129] The optimization module 430 is configured to: determine whether to perform a local graph optimization operation on the first node corresponding to the first operator based on the target kernel function container and attribute information corresponding to each first operator; and perform a local graph optimization operation on the node to be optimized based on the target kernel function container corresponding to the node to be optimized to obtain a second computation graph.

[0130] The code generation module 440 is configured to generate executable code corresponding to the first computation graph based on the second computation graph.

[0131] It should be noted that the acquisition module 410 can be used to implement Figure 1 The step S110 shown; the query module 420 can be used to implement Figure 1 The step S120 shown; optimization module 430 can be used to implement Figure 1 Steps S130 and S140 are shown; the code generation module can be used to implement... Figure 1 The steps are shown in step S150. For a detailed description of the functions that the acquisition module 410 can perform, please refer to the description of step S110 in the embodiments of the computational graph optimization method described above. For a detailed description of the functions that the query module 420 can perform, please refer to the description of step S120 in the embodiments of the computational graph optimization method described above. For a detailed description of the functions that the optimization module 430 can perform, please refer to the descriptions of steps S130 and S140 in the embodiments of the computational graph optimization method described above. For a detailed description of the functions that the code generation module 440 can perform, please refer to the description of step S150 in the embodiments of the computational graph optimization method described above. Repeated descriptions will not be repeated here. Furthermore, the computational graph optimization device 400 can achieve similar technical effects to the aforementioned computational graph optimization method, and will not be described further here.

[0132] It should be noted that, in at least one embodiment of this disclosure, the computational graph optimization device 400 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited; it can be constructed from analog devices, digital chips, or other suitable methods according to circuit principles.

[0133] For example, the computational graph optimization device 400 can be implemented in hardware, software, or a combination of hardware and software, and this disclosure does not impose any specific limitations on it.

[0134] The computational graph optimization method and apparatus provided in at least one embodiment of this disclosure can be applied to different systems or devices, such as those used in... Figure 5The illustrated electronic device 500 can be a terminal, such as a mobile phone, tablet computer, laptop computer, AR device, VR device, vehicle terminal, etc., or it can be a server, etc. For example, the hardware architecture of the electronic device 500 may include a graphics processor. The computational graph optimization method provided in at least one embodiment of this disclosure can be applied to scenarios involving CPU, high-performance computing (HPC), and artificial intelligence (AI) in the electronic device 500. Of course, this disclosure is not limited to this; any scenario, device, or apparatus involving computational graphs and model operation can employ the computational graph optimization method or apparatus provided in at least one embodiment of this disclosure.

[0135] In some embodiments, the computational graph optimization apparatus provided in at least one embodiment of this disclosure can be a chip, such as a system-on-a-chip (SoC). A SoC includes a processor, which can be a single-core or multi-core processor, memory, and I / O interfaces, etc.

[0136] Figure 5 A schematic block diagram of an electronic device according to at least one embodiment of the present disclosure is shown. Figure 5 As shown, the electronic device 500 is, for example, suitable for implementing the computational graph optimization method provided in the embodiments of this disclosure. It should be noted that... Figure 5 The components of the electronic device 500 shown are merely exemplary and not limiting. The electronic device 500 may have other components as needed for the actual application.

[0137] like Figure 5 As shown, the electronic device 500 may include a processing unit 501 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in memory to achieve various functions.

[0138] For example, when the computer-readable instruction processing device 501 executes, it can perform one or more steps in the computational graph optimization method according to any of the above embodiments. It should be noted that a detailed description of the processing procedure of the computational graph optimization method can be found in the relevant descriptions in the embodiments of the above-described computational graph optimization method.

[0139] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 503 and / or cache memory, etc., whereby computer-readable instructions can be loaded from storage device 508 into RAM 503 to execute. Non-volatile memory may include, for example, read-only memory (ROM) 502, hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB storage, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by the applications, may also be stored in the computer-readable storage medium.

[0140] For example, the processing device 501, the read-only memory (ROM) 502, and the random access memory (RAM) 503 are interconnected via a bus 504. The input / output (I / O) interface 505 is also connected to the bus 504.

[0141] Typically, the following devices can be connected to the input / output (I / O) interface 505: input devices 506 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 508 including, for example, magnetic tape, hard disk, flash memory, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown, and the electronic device 500 may alternatively implement or possess more or fewer devices. For example, a processing device 501 can control other components in the electronic device 500 to perform desired functions. The processing device 501 may be a device with data processing capabilities and / or program execution capabilities, such as a central processing unit (CPU), a tensor processor (TPU), or a graphics processing unit (GPU). The central processing unit (CPU) may be an x86, ARM, RISC-V architecture, etc. The GPU may be directly integrated into the SOC, directly integrated into the motherboard, or built into the northbridge chip of the motherboard.

[0142] Figure 6 A schematic block diagram of a computer-readable storage medium provided according to at least one embodiment of the present disclosure is shown.

[0143] For example, such as Figure 6As shown, one or more computer-readable instructions 601 may be stored non-transitorily on the computer-readable storage medium 600. For example, when the computer-readable instructions 601 are executed by at least one processor, one or more steps in the computational graph optimization method described above may be performed.

[0144] For example, the storage medium 600 can be used in an electronic device 500, such as the storage medium 600 including the storage device 508 in the electronic device 500.

[0145] For example, a storage device may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB storage, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and a processor may execute these instructions to perform various functions of the processor. Various application programs and various data may also be stored in the storage medium.

[0146] For example, the storage medium may include a memory card for a smartphone, a cache component for a tablet computer, a hard disk for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0148] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0149] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0150] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0151] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely exemplary forms of implementing the claims.

[0152] In addition to the exemplary descriptions above, the following points also need to be noted in this disclosure:

[0153] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0154] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0155] The above description is merely an exemplary embodiment of this disclosure and is not intended to limit the scope of protection of this disclosure, which is determined by the appended claims.

Claims

1. A computational graph optimization method, characterized in that, The method includes: Obtain a first computation graph, wherein the first computation graph includes at least one first node, each first node corresponds to a first operator, each first operator is pre-configured with at least two candidate kernel functions, and the at least two candidate kernel functions are adapted to different types of hardware computing cores respectively; For each first operator, a kernel function container list corresponding to the first operator is determined based on the attribute information of the first operator, and a target kernel function container is determined from the kernel function container list corresponding to the first operator. The kernel function container list includes at least two candidate kernel function containers, each candidate kernel function container includes at least one candidate kernel function pre-configured by the first operator, and the target kernel function container is the candidate kernel function container with the lowest execution time among the at least two candidate kernel function containers. Based on the target kernel function container corresponding to each first operator and the attribute information, determine whether to perform a local graph optimization operation on the first node corresponding to the first operator; For the node to be optimized that is determined to undergo the local graph optimization operation, the local graph optimization operation is performed according to the target kernel function container corresponding to the node to be optimized, so as to obtain a second computational graph; and Based on the second computation graph, the executable code corresponding to the first computation graph is generated.

2. The method as described in claim 1, characterized in that, In response to the hardware computing core adapted to the candidate kernel function having hardware operating constraints, the candidate kernel function container also includes a candidate optimization list corresponding to the candidate kernel function, and the candidate optimization list includes at least one operator for satisfying the hardware operating constraints of the candidate kernel function.

3. The method as described in claim 1, characterized in that, The attribute information of the first operator includes at least: the operator type of the first operator and the shape information of the input data of the first operator, wherein the shape information includes at least one of the shape and size of the input data and the layout format of the input data. The step of determining the list of kernel function containers corresponding to the first operator based on the attribute information of the first operator, and determining the target kernel function container from the list of kernel function containers corresponding to the first operator, includes: Based on the operator type of the first operator, determine the list of first kernel function containers corresponding to the first operator; For each candidate kernel function container in the first kernel function container list, the execution time of the candidate kernel function container is obtained at least based on the shape information of the input data of the first operator and the candidate kernel functions in the candidate kernel function container; and The candidate kernel function container with the lowest execution time is selected as the target kernel function container for the first operator.

4. The method as described in claim 3, characterized in that, The execution time of the candidate kernel function container includes the time required for the adapted hardware computing core to execute the candidate kernel functions in the candidate kernel function container according to the attribute information of the first operator.

5. The method as described in claim 4, characterized in that, In response to the candidate kernel function container further including a candidate optimization list corresponding to the candidate kernel function, and the shape information of the input data of the first operator not satisfying the hardware operation constraints of the adapted hardware computing core, the execution time of the candidate kernel function container further includes: the execution time of at least one operator in the candidate optimization list.

6. The method as described in claim 3, characterized in that, Each candidate kernel function container also includes a cost estimation function, which is used to determine the execution time of the corresponding candidate kernel function container, and the cost estimation function is configured based at least on the candidate kernel functions in the corresponding candidate kernel function container. The step of obtaining the execution time of each candidate kernel function container in the first kernel function container list, based at least on the shape information of the input data of the first operator and the candidate kernel functions in the candidate kernel function container, includes: The shape information is input into the cost estimation function included in each candidate kernel function container in the first kernel function container list to obtain the execution time of each candidate kernel function container in the first kernel function container list.

7. The method as described in claim 6, characterized in that, In response to the corresponding candidate kernel function container, which also includes a candidate optimization list corresponding to the candidate kernel function, the cost estimation function is further configured according to the candidate optimization list.

8. The method as described in claim 2, characterized in that, The attribute information of the first operator includes at least the shape information of the input data of the first operator, and the hardware operation constraints include constraints on the shape information. The step of determining whether to perform a local graph optimization operation on the first node corresponding to the first operator based on the target kernel function container corresponding to each first operator and the attribute information includes: Since the target kernel function container does not include the candidate optimization list, it is determined that the local graph optimization operation will not be performed on the first node. In response to the target kernel function container including the candidate optimization list and the shape information satisfying the constraints, it is determined that the local graph optimization operation will not be performed on the first node, and In response to the target kernel function container including the candidate optimization list and the shape information not satisfying the constraint, it is determined that the local graph optimization operation will be performed on the first node, and the first node will be selected as the node to be optimized.

9. The method as described in claim 8, characterized in that, The step of performing the local graph optimization operation on the node to be optimized, based on the target kernel function container corresponding to the node to be optimized, to obtain a second computational graph, includes: For each node to be optimized, based on at least one operator included in the candidate optimization list for satisfying the hardware operation constraints of the candidate kernel function, at least one auxiliary node is added at the corresponding position adjacent to the node to be optimized in the first computation graph to obtain the second computation graph.

10. The method as described in claim 1, characterized in that, The method further includes: Establish a binding relationship between the first operator and the target kernel function container; Specifically, generating executable code corresponding to the first computation graph based on the second computation graph includes: Traverse each node in the second computation graph. In response to the fact that the current operator corresponding to the current node is not the first operator, a kernel function is selected for the current operator based on the attribute information of the current operator; In response to the current operator corresponding to the current node being a first operator of any operator type, the candidate kernel function in the target kernel function container bound to the first operator of any operator type is taken as the kernel function of the current operator; Based on the kernel function of the operator corresponding to each node in the second computation graph, the executable code corresponding to the first computation graph is generated.

11. The method as described in claim 1, characterized in that, The process of obtaining the first computation graph includes: Obtain the original computation graph; A general graph optimization operation is performed on the second node of the original computation graph to obtain the first computation graph, wherein the operator corresponding to the second node is not the first operator.

12. The method as described in claim 1, characterized in that, The method further includes: Configure a list of kernel function containers for the first operator of different operator types.

13. The method as described in claim 12, characterized in that, The first operator of different operator types is configured with a corresponding kernel function container list, including: For any type of operator in the first operator: N candidate kernel function containers are constructed from the kernel function container list corresponding to any operator type, where N is an integer greater than or equal to 2, and the N candidate kernel function containers correspond to different hardware computing cores; The N candidate kernel functions pre-configured for any operator type and implemented by different hardware computing cores are respectively encapsulated into the N candidate kernel function containers; For any one of the N candidate kernel functions, in response to the hardware computing core implementing the candidate kernel function having hardware operation constraints, the candidate optimization list associated with the candidate kernel function is configured in the corresponding candidate kernel function container, wherein the associated candidate optimization list includes at least one operator for satisfying the hardware operation constraints.

14. The method as described in claim 13, characterized in that, The method further includes: A corresponding cost estimation function is constructed for each candidate kernel function container, wherein the cost estimation function is constructed based on the candidate kernel functions encapsulated in each candidate kernel function container. The candidate kernel function container also includes the candidate optimization list, and the cost estimation function is constructed based on the candidate optimization list.

15. A computational graph optimization device, characterized in that, The device includes: The acquisition module is configured to acquire a first computation graph, wherein the first computation graph includes at least one first node, each first node corresponds to a first operator, each first operator is pre-configured with at least two candidate kernel functions, and the at least two candidate kernel functions are adapted to different types of hardware computing cores respectively. The query module is configured to, for each first operator, determine the kernel function container list corresponding to the first operator based on the attribute information of the first operator, and determine the target kernel function container from the kernel function container list corresponding to the first operator. The kernel function container list includes at least two candidate kernel function containers, each candidate kernel function container includes at least one candidate kernel function pre-configured by the first operator, and the target kernel function container is the candidate kernel function container with the lowest execution time among the at least two candidate kernel function containers. The optimization module is configured to determine, based on the target kernel function container corresponding to each first operator and the attribute information, whether to perform a local graph optimization operation on the first node corresponding to the first operator; for the node to be optimized that is determined to undergo the local graph optimization operation, to perform the local graph optimization operation based on the target kernel function container corresponding to the node to be optimized, so as to obtain a second computation graph; and The code generation module is configured to generate executable code corresponding to the first computation graph based on the second computation graph.

16. An electronic device, characterized in that, The electronic device includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions that, when executed by the at least one processor, cause the at least one processor to perform the computational graph optimization method as described in any one of claims 1-14.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions that, when executed by at least one processor, cause the at least one processor to perform the computational graph optimization method as described in any one of claims 1-14.

Images