Operator processing method and operator processing apparatus, electronic device, and readable storage medium

Abstract

An operator processing method, an operator processing apparatus, an electronic device, and a computer-readable storage medium are disclosed. The operator processing method includes: acquiring a computation graph containing M operator nodes and the connection relationships between the M operator nodes, each operator node corresponding to at least one operator; splitting the computation graph into N subgraphs, each subgraph containing at least one operator node, the N subgraphs including a first subgraph, the first subgraph corresponding to K first fusion operators, the K first fusion operators being operators obtained by fusing multiple operators corresponding to multiple operator nodes included in the first subgraph; determining the operator codes corresponding to each of the N subgraphs, the operator codes corresponding to each of the N subgraphs including the operator codes of the K first fusion operators corresponding to the first subgraph; and obtaining code for executing the computation process based on the operator codes corresponding to each of the N subgraphs. This method is applicable to various scenarios, has a wide range of applications, and strong generalization ability.

Description

Technical Field

[0001] Embodiments of this disclosure relate to an operator processing method, an operator processing apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Running neural networks on AI chips requires the support of numerous operators. Neural networks can be used in fields such as speech recognition, image recognition, and natural language processing. These operators can be predefined low-level operators or user-defined operators. Data interaction between operators is often achieved through memory access, resulting in relatively low code execution efficiency. A common approach is to combine several operators according to a certain pattern to form a fusion operator. The fusion operator can exchange data through registers, thereby improving operating efficiency. Summary of the Invention

[0003] At least one embodiment of this disclosure provides an operator processing method, comprising: obtaining a computation graph describing a computation process, wherein the computation graph includes M operator nodes and connection relationships between the M operator nodes, each of the M operator nodes corresponding to at least one operator; splitting the computation graph to obtain N subgraphs, wherein each of the N subgraphs includes at least one operator node from the M operator nodes, the N subgraphs including a first subgraph, the first subgraph corresponding to K first fusion operators, the K first fusion operators being operators obtained by fusing multiple operators corresponding to multiple operator nodes included in the first subgraph; determining operator codes corresponding to the N subgraphs respectively, wherein the operator codes corresponding to the N subgraphs respectively include the operator codes of the K first fusion operators corresponding to the first subgraph; and obtaining code for executing the computation process based on the operator codes corresponding to the N subgraphs respectively, wherein M, N, and K are all integers not less than 1.

[0004] For example, in an embodiment of the operator processing method provided in this disclosure, the first subgraph includes P operator nodes, where P is an integer greater than 1; determining the operator codes corresponding to the N subgraphs respectively includes: for the first subgraph, determining the codes of the operators corresponding to the P operator nodes respectively; and obtaining the operator codes of the K first fusion operators corresponding to the first subgraph based on the codes of the operators corresponding to the P operator nodes respectively.

[0005] For example, in an embodiment of the operator processing method provided in this disclosure, for the first subgraph, determining the code of the operator corresponding to each of the P operator nodes includes: for each of the P operator nodes, performing the following operations: if the operator corresponding to the operator node is a predefined operator, for the predefined operator, obtaining the configuration parameters and code module corresponding to the predefined operator, and obtaining the code of the predefined operator based on the configuration parameters and code module; if the operator corresponding to the operator node is a custom operator, for the custom operator, compiling the custom operator to obtain the code of the custom operator.

[0006] For example, in an embodiment of the present disclosure, the code of the K first fusion operators corresponding to the first subgraph is obtained based on the code of the operators corresponding to the P operator nodes respectively, including: combining the code of the operators corresponding to the P operator nodes respectively to obtain the operator code of the K first fusion operators.

[0007] For example, in an embodiment of the operator processing method provided in this disclosure, the computation graph is split into N subgraphs, including: if the target attributes of the operators corresponding to Q operator nodes are the same, then the Q operator nodes are divided into a subgraph, wherein the target attributes include at least one of type attributes, computation attributes and data transmission attributes, and Q is an integer greater than 1.

[0008] For example, in an embodiment of the operator processing method provided in this disclosure, the computation graph is split into N subgraphs, including: if the operators corresponding to the Q operator nodes are all operators of the same type, then the Q operator nodes are divided into a subgraph.

[0009] For example, in an embodiment of the operator processing method provided in this disclosure, the computation graph is split into N subgraphs, including: if the operators corresponding to the Q operator nodes are configured to transmit data through registers, then the Q operator nodes are divided into a subgraph.

[0010] For example, in an embodiment of the operator processing method provided in this disclosure, the computation graph is split into N subgraphs, including: if the operators corresponding to the Q operator nodes are configured to run on the same computation unit, then the Q operator nodes are divided into a subgraph.

[0011] For example, in an embodiment of the operator processing method provided in this disclosure, the computation graph is split into N subgraphs, including: if the type and execution order of the operators corresponding to Q operator nodes are consistent with the type and execution order of the operators included in the predetermined fusion operator, then the Q operator nodes are divided into a subgraph.

[0012] For example, in an embodiment of the operator processing method provided in this disclosure, the Q operator nodes are either consecutive or parallel to each other in the execution order determined according to the computation graph.

[0013] For example, in an embodiment of the operator processing method provided in this disclosure, at least one of the K first fusion operators includes multiple operators, which are connected in sequence and executed sequentially; for two adjacent operators among the multiple operators, the calculation result data of the preceding operator in the execution order is used as the input data of the following operator.

[0014] For example, in an embodiment of the operator processing method provided in this disclosure, the N subgraphs further include a second subgraph, the second subgraph corresponds to R second fusion operators, the R second fusion operators are operators obtained by fusing multiple operators corresponding to multiple operator nodes included in the second subgraph, where R is an integer not less than 1; the operator codes corresponding to the N subgraphs respectively also include the operator codes of the R second fusion operators corresponding to the second subgraphs; the method further includes: before determining the operator codes corresponding to the N subgraphs respectively, optimizing the K first fusion operators and the R second fusion operators to obtain configuration information for the K first fusion operators and the R second fusion operators respectively.

[0015] For example, in an embodiment of the operator processing method provided in this disclosure, optimization processing is performed on the K first fusion operators and the R second fusion operators to obtain configuration information for the K first fusion operators and the R second fusion operators. This includes: optimizing the resource configuration of the K first fusion operators and the R second fusion operators to obtain optimized resource configuration information for each first fusion operator and each second fusion operator, wherein the resources include computing resources and / or storage resources.

[0016] For example, in an embodiment of the operator processing method provided in this disclosure, optimization processing is performed on the K first fusion operators and the R second fusion operators to obtain configuration information for the K first fusion operators and the R second fusion operators, including: based on the calculation process of the computation graph, formulating a synchronization strategy for the K first fusion operators and the R second fusion operators to obtain synchronization configuration information for the K first fusion operators and the R second fusion operators.

[0017] For example, in an embodiment of the operator processing method provided in this disclosure, based on the computation process of the computation graph, a synchronization strategy is formulated for the K first fusion operators and the R second fusion operators to obtain synchronization configuration information for the K first fusion operators and the R second fusion operators. This includes: performing the following operation for at least two of the K first fusion operators and the R second fusion operators: if the two fusion operators have a data dependency relationship, and the scheduling unit identifier of the data receiver is different from the scheduling unit identifier of the data producer and the scheduling unit identifier between the data producer and the data receiver, then the two fusion operators are configured to perform data synchronization processing, wherein the data producer is one of the two fusion operators and the data producer is the other.

[0018] For example, in an embodiment of the operator processing method provided in this disclosure, the first fusion operator includes multiple operators corresponding to multiple operator nodes included in the first subgraph; the method further includes: for the first fusion operator, performing optimization processing among the multiple operators included in the first fusion operator to obtain configuration information for the multiple operators, wherein the optimization processing includes resource configuration optimization and / or formulating a synchronization strategy.

[0019] For example, in an embodiment of the present disclosure, the operator processing method, based on the operator codes corresponding to the N subgraphs respectively, obtains the code of the computation graph, including: combining the operator codes corresponding to the N subgraphs respectively to obtain the code of the computation graph.

[0020] At least one embodiment of this disclosure provides an operator processing apparatus, including an acquisition module, a splitting module, a determination module, and a code module. The acquisition module is configured to acquire a computation graph describing a computation process, wherein the computation graph includes M operator nodes and the connection relationships between the M operator nodes, and each of the M operator nodes corresponds to at least one operator. The splitting module is configured to split the computation graph to obtain N subgraphs, wherein each of the N subgraphs includes at least one operator node from the M operator nodes, and the N subgraphs include a first subgraph, the first subgraph corresponding to K first fusion operators, the K first fusion operators being operators obtained by fusing multiple operators corresponding to multiple operator nodes included in the first subgraph. The determination module is configured to determine the operator codes corresponding to the N subgraphs respectively, wherein the operator codes corresponding to the N subgraphs respectively include the operator codes of the K first fusion operators corresponding to the first subgraph. The code module is configured to obtain code for executing the computation process based on the operator codes corresponding to the N subgraphs respectively; wherein M, N, and K are all integers not less than 1.

[0021] At least one embodiment of this disclosure provides an electronic device, including a processor and a memory, wherein the memory stores one or more computer program modules, wherein the one or more computer program modules are configured to be executed by the processor to implement the operator processing method provided in any embodiment of this disclosure.

[0022] At least one embodiment of this disclosure provides a computer-readable storage medium storing non-transitory computer-readable instructions that, when executed by a computer, implement the operator processing method provided in any embodiment of this disclosure. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0024] Figure 1 A flowchart of code for generating a neural network is shown;

[0025] Figure 2 A flowchart of an operator processing method provided in at least one embodiment of this disclosure is shown;

[0026] Figure 3 A schematic diagram of a portion of a computation graph provided in at least one embodiment of the present disclosure is shown;

[0027] Figure 4 A schematic diagram of a processing flow provided by at least one embodiment of the present disclosure is shown;

[0028] Figure 5A A schematic diagram of a portion of a computation graph provided in at least one embodiment of this disclosure is shown;

[0029] Figure 5B It shows that Figure 5A The diagram shows a subgraph obtained by splitting a portion of the computational graph.

[0030] Figure 6 A schematic diagram of a fusion operator provided in at least one embodiment of the present disclosure is shown;

[0031] Figure 7 A schematic block diagram of an operator processing apparatus provided in at least one embodiment of the present disclosure is shown;

[0032] Figure 8 A schematic block diagram of an electronic device provided in at least one embodiment of the present disclosure is shown;

[0033] Figure 9 A schematic block diagram of another electronic device provided in at least one embodiment of the present disclosure is shown; and

[0034] Figure 10 A schematic diagram of a computer-readable storage medium provided in at least one embodiment of the present disclosure is shown. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0036] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “including,” “comprising,” or “containing,” and similar terms mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. The terms “connected,” “linked,” or similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “upper,” “lower,” “left,” and “right,” etc., are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0037] Figure 1 A flowchart of code for generating a neural network is shown. For example... Figure 1 As shown, the predefined underlying operator library 101 can contain multiple commonly used predefined operators (or underlying predefined operators); in addition, this process also uses one or more custom operators, which are user-defined operators not included in the predefined underlying operator library 101. For specific application scenarios, a corresponding predefined fusion operator library 102 can be pre-formed based on the predefined underlying operator library 101. The predefined fusion operator library 102 includes the fusion operators required for that specific application scenario (e.g., fusion operators A1 to Ag, where g is an integer greater than 1). When it is necessary to construct the neural network 105, the code for the computation graph 104 of the neural network is obtained based on the code of the relevant fusion operators in the predefined fusion operator library 102 and / or the operator code of the custom operator 103.

[0038] Different neural networks, different layers of the same neural network, and different data structures may require different fusion operators. Therefore, specific fusion operators need to be customized and optimized for a particular neural network, a specific network layer, and a specific data structure, and then combined into a computational graph to run the neural network under specific conditions. This process is as follows: Figure 1 The process is shown as a bottom-up process. During this process, because the fusion operators in these predefined fusion operator libraries are developed for specific situations, their applicability is narrow and their generalization ability is weak.

[0039] At least one embodiment of this disclosure provides an operator processing method, an operator processing apparatus, an electronic device, and a computer-readable storage medium. The operator processing method includes: acquiring a computation graph describing a computation process, wherein the computation graph contains M operator nodes and connection relationships between the M operator nodes, each of the M operator nodes corresponding to at least one operator; splitting the computation graph to obtain N subgraphs, wherein each of the N subgraphs contains at least one operator node from the M operator nodes, the N subgraphs including a first subgraph, the first subgraph corresponding to K first fusion operators, the K first fusion operators being operators obtained by fusing multiple operators corresponding to multiple operator nodes included in the first subgraph; determining operator codes corresponding to each of the N subgraphs, wherein the operator codes corresponding to each of the N subgraphs include the operator codes of the K first fusion operators corresponding to the first subgraph; and obtaining code for executing the computation process based on the operator codes corresponding to each of the N subgraphs, wherein M, N, and K are all integers not less than 1.

[0040] In this embodiment, the operator processing method generates fusion operators in a top-down manner. That is, from a global perspective, the computation graph is split into multiple subgraphs. The structure of the fusion operator to be generated is determined based on the subgraphs, and then the corresponding fusion operator is generated, followed by the generation of the computation graph code. This operator processing method can abstract the commonalities in the network structure during the generation of fusion operators, without including information specific to certain cases. It is applicable to various scenarios, has a wide range of applications, and strong generalization ability.

[0041] Figure 2 A flowchart of an operator processing method provided by at least one embodiment of the present disclosure is shown.

[0042] like Figure 2 As shown, the method may include steps S210 to S240.

[0043] Step S210: Obtain a computation graph to describe the computation process. The computation graph contains M operator nodes and the connection relationships between the M operator nodes. Each of the M operator nodes corresponds to at least one operator.

[0044] Step S220: Split the computation graph to obtain N subgraphs. Each of the N subgraphs contains at least one operator node from the M operator nodes. The N subgraphs include a first subgraph, which corresponds to K first fusion operators. The K first fusion operators are operators obtained by fusing multiple operators corresponding to multiple operator nodes included in the first subgraph.

[0045] Step S230: Determine the operator codes corresponding to the N subgraphs respectively. The operator codes corresponding to the N subgraphs include the operator codes of the K first fusion operators corresponding to the first subgraph.

[0046] Step S240: Based on the operator codes corresponding to the N subgraphs, obtain the code used to execute the calculation process.

[0047] For example, M, N, and K are all integers not less than 1.

[0048] For example, the above embodiments of this disclosure use a computation graph of a neural network model as an example for illustration, but this disclosure is not limited thereto. In practical applications, the computation graph can be the computation graph of any computation model.

[0049] For example, in step S210, the computation graph can be automatically generated based on the description information of the neural network. For example, a directed acyclic graph (i.e., a computation graph) can be automatically generated based on the network structure of the neural network. The directed acyclic graph includes multiple nodes and connections between nodes, and these connections represent the data dependencies and data flow between nodes.

[0050] For example, computational graphs use graphics to describe the computational process of a computational model. Figure 3 A schematic diagram of a portion of a computation graph provided in at least one embodiment of this disclosure is shown, such as... Figure 3 As shown, the computation graph includes multiple operator nodes, each corresponding to an operator. Here, an operator can be understood as a predefined operator from a predefined underlying operator library or a user-defined operator. For example, in... Figure 3 In the computation graph, operator nodes 301, 302, and 303 are included. Operator node 301 corresponds to the Conv operator (convolution operator), operator node 302 corresponds to the BatchNormalization operator (batch normalization operator), and operator node 303 corresponds to the node corresponding to the ReLU operator (linear rectified activation function). In some embodiments, each operator node may also correspond to multiple operators. Furthermore, the computation graph may also include other nodes, such as variable nodes and computation result nodes. Variable nodes are parameters used in the computation graph, such as... Figure 3Node 304 is a variable node, corresponding to the Weight parameter; nodes 305, 306 and 307 are calculation result nodes, corresponding to the calculation result data (tensor data) Tensor1, Tensor2 and Tensor3 respectively. The calculation result node can represent the calculation result data of the previous operator adjacent to it.

[0051] To more clearly describe the embodiments of this disclosure, only operator nodes in the computation graph will be described in the following embodiments, while other nodes will be ignored.

[0052] For example, the computation graph also includes connections between multiple operator nodes, which can be represented by arrowed lines. Two connected operators may have data transfer relationships (or data dependencies). For instance, if the computation result data of the Conv operator is used as input data for the Batch Normalization operator, the Conv operator can be connected to the Batch Normalization operator with the arrow of the connecting line pointing towards the Batch Normalization operator.

[0053] Figure 4 A schematic diagram of the processing flow provided in at least one embodiment of this disclosure is shown, such as... Figure 4 As shown, in step S220, the computation graph 401 can be divided into multiple subgraphs, for example, into subgraphs B1 to BN. Each subgraph, as part of the computation graph, contains a portion of the operator nodes of the computation graph and the connection relationships between these operator nodes. For example, different subgraphs may contain the same or different numbers of operator nodes, and there is no overlap between the operator nodes contained in different subgraphs. For example, at least one subgraph in the N subgraphs contains multiple (two or more) operator nodes; for example, the N subgraphs include a first subgraph containing multiple operator nodes. In some examples, each subgraph in the N subgraphs may contain multiple operator nodes; in other examples, some subgraphs in the N subgraphs may contain multiple operator nodes, and some subgraphs may contain only one operator node. For ease of description, a subgraph containing multiple operator nodes will be referred to as a multi-node subgraph, and a subgraph containing only one operator node will be referred to as a single-node subgraph.

[0054] For example, such as Figure 4 As shown, after obtaining multiple subgraphs B1 to BN, multiple fusion operators corresponding to these subgraphs B1 to BN can be determined, such as fusion operators C1 to Cs (s is an integer greater than 1, and s can be less than, equal to, or greater than N). Here, determining the fusion operator means determining the structural information of the fusion operator to be generated, such as determining the types of operators and connection relationships contained in the fusion operator to be generated, so that in step S230, the corresponding fusion operator can be generated based on this information of the fusion operator to be generated.

[0055] For example, in some examples, one or more fusion operators to be generated can be determined for each multi-node subgraph. Each fusion operator to be generated can be obtained by fusing all or some of the operator nodes contained in the corresponding subgraph. For example, if a multi-node subgraph corresponds to one fusion operator, the fusion operator can be obtained by fusing multiple operators corresponding to all operator nodes contained in the multi-node subgraph. If a multi-node subgraph corresponds to two or more fusion operators, each of the two or more fusion operators can be obtained by fusing operators corresponding to some of the operator nodes contained in the multi-node subgraph. In some of the following embodiments, the example of one fusion operator per multi-node subgraph is used for illustration.

[0056] Figure 5A A schematic diagram of a portion of a computation graph provided in at least one embodiment of this disclosure is shown. Figure 5B It shows that Figure 5A The diagram shows a subgraph obtained by splitting a portion of the computational graph.

[0057] like Figure 5A The computation graph shown includes a portion of operator nodes 501–509, such as… Figure 5B As shown, operator nodes 502, 503, and 504 can be divided into a multi-node subgraph B1 to determine the corresponding fusion operator to be generated, CBR (i.e., Conv+Batch Normalization+ReLU); operator nodes 505 and 506 can be divided into a multi-node subgraph B2 to determine the corresponding fusion operator to be generated, CB (i.e., Conv+Batch Normalization); and operator nodes 507, 508, and 509 can be divided into a multi-node subgraph B3 to determine the corresponding fusion operator to be generated, CBA (i.e., Conv+BatchNormalization+Add). For example, in some other examples, multiple fusion operators to be generated can also be determined based on a multi-node subgraph.

[0058] For example, the first subgraph corresponds to K first fusion operators, where at least one first fusion operator includes multiple operators. The following explanation uses K=1 as an example. The first fusion operator corresponding to the first subgraph can include multiple operators corresponding to the multiple operator nodes included in the first subgraph. These multiple operators are connected sequentially and executed in order. For any two adjacent operators among these multiple operators, the calculation result of the preceding operator is used as the input data for the following operator. For example, ... Figure 5BSubgraph B1 is shown as the first subgraph. The CBR operator is used as the first fusion operator. The Conv operator, Batch Normalization operator, and ReLU operator are connected sequentially and executed in order. The calculation result of the Conv operator is used as the input data of the Batch Normalization operator, and the calculation result of the Batch Normalization operator is used as the input data of the ReLU operator. For example, if the first subgraph corresponds to two or more first fusion operators, each first fusion operator may include some operators from the multiple operators corresponding to the first subgraph.

[0059] For example, in step S230, determining the operator codes corresponding to the N subgraphs includes determining the operator codes for multiple fusion operators corresponding to the N subgraphs. Figure 4 As shown, after obtaining multiple fusion operators C1 to Cs to be generated based on multiple subgraphs B1 to BN, the code of the fusion operators C1 to Cs to be generated can be determined according to the predefined underlying operator library 403 and / or the custom operator 404, and the fusion operators C1 to Cs can be generated.

[0060] Figure 6 A schematic diagram of a fusion operator provided in at least one embodiment of this disclosure is shown. For example... Figure 3 and Figure 6 As shown, before fusing the Conv, Batch Normalization, and ReLU operators, the result data (Tensor1 to Tensor3) calculated by each operator needs to be written to memory, and the next operator then reads the data from memory. Data interaction between operators is achieved through memory access, resulting in relatively low code execution efficiency. After fusing the Conv, Batch Normalization, and ReLU operators to obtain the fused operator CBR, the result Tensor3 can be directly calculated based on the fused operator CBR. The intermediate calculation results of the fused operator CBR can be buffered in the registers of the processor executing the fused operator (such as a graphics processing unit (GPU), data processing unit (DPU), or domain-specific architecture (DSA) chip (such as an AI accelerator), without going through memory located outside the processor, thus improving execution efficiency.

[0061] For example, in addition to the first subgraph, other subgraphs can be included. For instance, the N subgraphs can include a second subgraph in addition to the first subgraph. The second subgraph corresponds to R second fusion operators (R is an integer not less than 1). These R second fusion operators are obtained by fusing multiple operators corresponding to multiple operator nodes included in the second subgraph. The operator codes corresponding to the N subgraphs include the operator codes for the K first fusion operators corresponding to the first subgraph and the operator codes for the R second fusion operators corresponding to the second subgraph.

[0062] For example, after obtaining the operator codes corresponding to N subgraphs, the code for executing the computation process of the computation graph can be obtained based on the operator codes corresponding to the N subgraphs, and then the running code of models such as neural networks can be obtained.

[0063] According to the operator processing method of this disclosure, a top-down approach is adopted to generate fusion operators. That is, the computation graph is split into multiple subgraphs from a global perspective, the structure of the fusion operator to be generated is determined, the corresponding fusion operator is generated, and then the code for the computation graph is further generated. This operator processing method can abstract the commonalities in the network structure during the generation of fusion operators, without including information specific to certain cases. It is applicable to various scenarios, has a wide range of applications, and strong generalization ability.

[0064] The operator processing method of at least one embodiment of this disclosure can be applied to application scenarios involving the generation of fusion operator code in the software stack of artificial intelligence chips (e.g., image processors (GPUs), data processors (DPUs), or domain-specific architecture (DSA) chips (e.g., AI accelerators)). Fusion operators are the foundation for efficient computation in artificial intelligence chips, and this operator processing method can contribute to achieving efficient computation in artificial intelligence chips.

[0065] For example, in some examples, in step S220, if Q operator nodes correspond to operators with the same target attribute, then these Q operator nodes can be divided into a subgraph. The target attribute includes at least one of type attribute, computation attribute, and data transmission attribute, where Q is an integer greater than 1. For example, the Q operator nodes may be consecutive or parallel to each other in the execution order determined by the computation graph. That is, if the target attributes of the operators corresponding to Q consecutive or parallel operator nodes in the execution order are the same, then these Q operator nodes can be divided into a subgraph.

[0066] For example, taking type attributes as an example, if the Q operator nodes all correspond to operators of the same type, then the Q operator nodes are divided into a subgraph. For example, if the computation graph contains Q sequentially connected operator nodes, and these Q operator nodes all correspond to operators of the same type (e.g., Conv type), then the Q Conv type operator nodes can be divided into a subgraph; for example, Q consecutive Conv type operators can be merged into a single merged operator. Based on this method, duplicate operator nodes can be merged, improving execution efficiency.

[0067] For example, taking data transmission attributes as an example, if the operators corresponding to the Q operator nodes are configured to transmit data via registers, then the Q operator nodes are divided into a subgraph. For example, as... Figure 5B As shown, if the operators corresponding to operator nodes 507, 508, and 509 are configured to transfer data via registers—that is, the calculation result of the Conv operator corresponding to operator node 507 is stored in a register, the Batch Normalization operator corresponding to operator node 508 retrieves the calculation result of the Conv operator from the register, and the calculation result of the Batch Normalization operator is stored in a register, and the Add operator corresponding to operator node 509 retrieves the calculation result of the Conv operator from the register, while the calculation result of the Add operator can be stored in memory, for example—then operator nodes 507, 508, and 509 can be divided into a subgraph to merge the Conv, Batch Normalization, and Add operators into a single fused operator. For example, in other examples, if the operators corresponding to Q operator nodes are configured to transfer data via cache, the data transferred between the operators corresponding to these Q operator nodes also does not need to pass through memory; therefore, these Q operator nodes can also be divided into a subgraph. Based on this approach, it can be ensured that the intermediate calculation results of the fused operator are transferred via non-memory access methods such as registers.

[0068] For example, taking a computed attribute as an example, if the operators corresponding to the Q operator nodes are configured to run on the same computational unit, then the Q operator nodes are divided into a subgraph. For example, various computational units can be used to perform the computation process of the computation graph, such as tensor kernels (tcores) and vector kernels (vcores). Figure 5B As shown, if the operators corresponding to operator nodes 502, 503, and 504 all perform calculations using the tcore computation unit, and the operators corresponding to operator nodes 505 and 506 all perform calculations using the vcore computation unit, then operator nodes 502, 503, and 504 can be divided into one subgraph, and operator nodes 505 and 506 into another subgraph. Based on this approach, it can be ensured that the code for the fused operator runs on the same computation unit.

[0069] For example, in some examples, in step S220, if the types and execution order of the operators corresponding to the Q operator nodes are consistent with the types and execution order of the operators included in the predetermined fusion operator, then the Q operator nodes are divided into a subgraph. For example, some commonly used types of fusion operators can be predefined as predetermined fusion operators. When multiple consecutive operator nodes that can be combined into the predetermined fusion operator appear in the computation graph, these multiple operator nodes can be divided into a subgraph to fuse and obtain the predetermined fusion operator. The predetermined fusion operator may include, for example, a CBR type fusion operator, where the data flow path is Conv operator - Batch Normalization operator - ReLU operator. If the computation graph includes sequentially connected Conv operator nodes, Batch Normalization operator nodes, and ReLU operator nodes, then these three operator nodes can be divided into a subgraph to fuse and obtain a CBR type fusion operator.

[0070] For example, one of the above-mentioned subgraph partitioning methods can be used depending on the actual situation, or two or more of the above-mentioned subgraph partitioning methods can be combined. For example, by combining computational attributes and data transmission attributes, and configuring operators corresponding to Q consecutive operator nodes to be executed in the same computational unit and to transfer data through registers, the Q operator nodes can be partitioned into a subgraph.

[0071] For example, as mentioned above, N subgraphs can correspond to multiple fusion operators. If each subgraph corresponds to one fusion operator, then each fusion operator can be an operator obtained by fusing multiple operators corresponding to multiple operator nodes included in a multi-node subgraph. For example... Figure 4 As shown, before executing step 130 to determine the operator code corresponding to each of the N subgraphs, the optimization manager can be used to optimize the multiple fusion operators to be generated in order to obtain configuration information for the multiple fusion operators to be generated. The optimization process includes resource configuration optimization and / or formulating synchronization strategies.

[0072] For example, in other examples, optimization may be performed only on a subset of the multiple fusion operators.

[0073] For example, in other examples, the N subgraphs may include not only multi-node subgraphs but also single-node subgraphs, each of which may include an operator node, i.e., corresponding to at least one operator. For example... Figure 5BAs shown, for example, operator node 501 is divided into a subgraph, which is a single-node subgraph corresponding to the ReLU operator. The ReLU operator corresponding to operator node 501 does not need to be fused and can be used as a separate operator. In this case, the N subgraphs not only correspond to multiple fused operators but also to at least one unfused individual operator (or non-fused operator). During optimization, the multiple fused operators corresponding to the N subgraphs and at least one non-fused operator can be optimized together. For example, in some embodiments, non-fused operators can be expressed according to the data structure of fused operators. Therefore, non-fused operators with fused operator data structures can be regarded as fused operators. In some embodiments of optimization below, fused operators obtained by fusing several operators and non-fused operators with fused operator data structures are collectively referred to as fused operators. That is, in some embodiments of optimization below, fused operators can include fused operators obtained by fusing several operators, or non-fused operators expressed with fused operator data structures.

[0074] For example, to clarify the description, the following embodiments will use the optimization of the first and second fusion operators as an example. For instance, before determining the operator codes corresponding to the N subgraphs, the K first fusion operators and R second fusion operators can be optimized to obtain configuration information for the K first fusion operators and R second fusion operators respectively. The optimization of other fusion operators can refer to these first and second fusion operators.

[0075] For example, optimization processing may include optimizing resource configuration. For instance, resource configuration optimization may be performed on K first fusion operators and R second fusion operators to obtain optimized resource configuration information for each first fusion operator and each second fusion operator.

[0076] For example, resources can include computing resources, such as processor resources, for instance, a GPU. These computing resources include tensor computation cores, special function units (SFUs), etc. During the optimization process, the computing resources allocated to each first fusion operator and each second fusion operator can be adjusted to make the allocation of computing resources more reasonable and improve the utilization rate of computing resources.

[0077] For example, resources may include storage resources, such as memory space (e.g., external memory, internal registers, etc.). During optimization, the storage resources allocated to each first fusion operator and each second fusion operator can be adjusted to make the allocation of storage space more reasonable and improve the utilization of storage space.

[0078] For example, the optimization process may also include formulating a synchronization strategy, such as formulating a synchronization strategy for K first fusion operators and R second fusion operators based on the computation graph, so as to obtain synchronization configuration information for the K first fusion operators and R second fusion operators.

[0079] For example, in a single-branch network (i.e., the entire neural network has only one branch), the data dependencies are simple and the pattern is fixed, making data synchronization easy to achieve. However, for multi-branch networks, there is no fixed branching pattern, and different networks may have different multi-branch structures. To ensure data synchronization, related technologies determine the strong constraint execution order of operators based on data dependencies, ensuring a completely serial computation process between operators. That is, the next operator's computation is only started after the previous operator's computation is completely completed and the result is written. However, this strong constraint on the execution order makes the computation process completely serial, which severely reduces the computational efficiency of the neural network. Based on the above, ensuring data synchronization between fusion operators in a neural network is a necessary prerequisite for efficient execution of the network computation process. The operator processing method of this disclosure addresses various types of neural network structures by formulating synchronization strategies between fusion operators, ensuring accurate and efficient execution order even with complex data dependencies.

[0080] For example, based on the calculation process, it can be determined which fusion operators need to perform data synchronization processing and which do not. For two fusion operators that require data synchronization processing, the calculation of the second operator should only begin after the calculation of the first operator has been completed. This ensures the sequential execution of fusion operators with data dependencies, thereby achieving efficient computation of the neural network.

[0081] For example, for at least two of the K first fusion operators and R second fusion operators, the following operation is performed: If the two fusion operators have a data dependency relationship, and the scheduling unit identifier of the data receiver is different from the scheduling unit identifier of the data producer, as well as the scheduling unit identifier between the data producer and the data receiver, then the two fusion operators are configured to perform data synchronization processing, where the data producer is one of the two fusion operators and the data generator is the other. For example, the above operation can be performed for every two fusion operators.

[0082] For example, taking two fusion operators (e.g., a first fusion operator and a second fusion operator) as an example, one of the first and second fusion operators can be designated as the data producer and the other as the data receiver (e.g., the data consumer) based on the execution order. For instance, the one executed earlier in the order can be designated as the data producer, and the one executed later in the order can be designated as the data receiver. If the first and second fusion operators have a data dependency relationship, and the scheduling unit identifier of the data receiver is different from the scheduling unit identifier of the data producer, as well as the scheduling unit identifier between the data producer and the data receiver, then the first and second fusion operators are configured to perform data synchronization processing; otherwise, the first and second fusion operators are configured not to perform data synchronization processing.

[0083] For example, the data dependency between the first fusion operator and the second fusion operator includes the fact that the calculation process of one fusion operator (e.g., the second fusion operator) requires the direct or indirect use of the calculation result of the other fusion operator (e.g., the first fusion operator), that is, the output data of one fusion operator (e.g., the first fusion operator) will flow to the other fusion operator (e.g., the second fusion operator).

[0084] like Figure 5B As shown, the calculation process of the fusion operator CB corresponding to subgraph B2 directly uses the calculation result of the fusion operator CBR corresponding to subgraph B1. Therefore, there is a data dependency between the fusion operator CB and the fusion operator CBR. Furthermore, the fusion operator CBR is the data producer, and the fusion operator CB is the data receiver. Similarly, the calculation process of the fusion operator CBA corresponding to subgraph B3 indirectly uses the calculation result of the fusion operator CBR corresponding to subgraph B1. Therefore, there is also a data dependency between the fusion operator CBA and the fusion operator CBR. Furthermore, the fusion operator CBA is the data receiver, and the fusion operator CBR is the data producer. For another example, if the data required for the calculation process of a certain fusion operator (not shown in the figure) is unrelated to the calculation result of the fusion operator CBR corresponding to subgraph B1, and the calculation process of the fusion operator CBR corresponding to subgraph B1 does not need to use the calculation result of this certain fusion operator, then it can be considered that there is no data dependency between this certain fusion operator and the fusion operator CBR corresponding to subgraph B1.

[0085] For example, if the first fusion operator and the second fusion operator do not have a data dependency relationship, then there is no data interaction between them, and therefore no data synchronization processing is required.

[0086] For example, if there is a data dependency between the first fusion operator and the second fusion operator, it is also necessary to determine whether the scheduling unit identifiers of the first fusion operator and the second fusion operator are the same. If the scheduling unit identifiers of the first fusion operator and the second fusion operator are the same, then the first fusion operator and the second fusion operator are configured not to perform data synchronization processing.

[0087] For example, a warp corresponds to a group of threads, which executes parallel computation using a round-robin scheduling method. Generally, a program consists of multiple workgroups, each workgroup contains multiple thread groups, and each thread group contains multiple threads. Threads within the same workgroup can be grouped according to scheduling units, and then scheduled to hardware for execution group by group. This scheduling unit is called a thread group. A warp can be considered the most basic unit of execution; a warp contains 32 parallel threads that execute the same instructions using different data resources.

[0088] For example, whether a pair of fusion operators synchronize data depends not only on whether they store data dependencies, but also on the scheduling unit they belong to. As an example, different scheduling units are distinguished by their scheduling unit identifiers.

[0089] For example, the computation of fusion operators in a neural network depends on the scheduling of scheduling units. Before computation, each fusion operator can be assigned a scheduling unit identifier (warp ID). Different operators may be assigned the same scheduling unit identifier (i.e., located in the same scheduling unit) or different scheduling unit identifiers. Fusion operators with different scheduling unit identifiers belong to different scheduling units and can be executed in parallel in a manner such as round-robin scheduling. Whether fusion operators with data dependencies need to synchronize data depends on the allocation of scheduling unit identifiers between the two operators.

[0090] For example, if the first fusion operator and the second fusion operator have a data dependency relationship, and the first fusion operator and the second fusion operator are assigned to the same scheduling unit, that is, the first fusion operator and the second fusion operator have the same scheduling unit identifier, then there is no need to perform data synchronization processing between the first fusion operator and the second fusion operator. This is because the fusion operators on the same scheduling unit are executed sequentially, that is, data synchronization can be guaranteed in terms of execution order.

[0091] For example, if the first fusion operator and the second fusion operator have a data dependency relationship and the scheduling unit identifiers of the first fusion operator and the second fusion operator are different, then data synchronization processing is required for the first fusion operator and the second fusion operator. For example, if the second fusion operator is executed after the first fusion operator, and the scheduling unit identifiers of the second fusion operator and the first fusion operator are different, then data synchronization processing can be performed for the first fusion operator and the second fusion operator.

[0092] For example, if the first fusion operator and the second fusion operator have a data dependency, taking the first fusion operator as the data producer and the second fusion operator as the data receiver as an example, if there is at least one intermediate operator between the first and second fusion operators in the execution order, it is determined whether the scheduling unit identifier of the first fusion operator is different from the scheduling unit identifier of the second fusion operator and the scheduling unit identifier of the at least one intermediate operator. If they are all different, synchronization is required. If the scheduling unit identifier of the first fusion operator is the same as at least one of the second fusion operator and the intermediate operator, then no data synchronization is required. For example, if the second fusion operator is after the first fusion operator in the execution order, and the scheduling unit identifier of the second fusion operator is different from the scheduling unit identifiers of the several fusion operators preceding it, then data synchronization is required between the first and second fusion operators. Otherwise, synchronization is not required. For example, if there is a third fusion operator between the first fusion operator and the second fusion operator, and the scheduling unit identifiers of the first fusion operator and the second and third fusion operators are all different, then data synchronization processing is required. If the scheduling unit identifiers of the first fusion operator and the second fusion operator are the same and / or the scheduling unit identifiers of the first fusion operator and the third fusion operator are the same, then data synchronization processing is not required.

[0093] For example, in some embodiments, the scheduling unit identifier mentioned above can be replaced with a computation identifier, which can be understood as the identifier of the smallest computational granularity (e.g., a thread). For example, if the first fusion operator and the second fusion operator have a data dependency relationship and the first fusion operator and the second fusion operator have the same computation identifier, for example, if the first fusion operator and the second fusion operator run in the same thread, since the first fusion operator and the second fusion operator running in the same thread will be executed in sequence, that is, data synchronization can be guaranteed in terms of execution order, so there is no need to perform data synchronization on the first fusion operator and the second fusion operator. As another example, taking the first fusion operator as the data producer and the second fusion operator as the data receiver, there is a third fusion operator between the first fusion operator and the second fusion operator. If the computation identifiers of the first fusion operator are different from those of the second fusion operator and the third fusion operator, then the first fusion operator and the second fusion operator need to perform data synchronization processing. If the computation identifiers of the first fusion operator and the second fusion operator are the same and / or the computation identifiers of the first fusion operator and the third fusion operator are the same, then the first fusion operator and the second fusion operator do not need to perform data synchronization processing.

[0094] For example, if the input data of the data receiver comes from multiple data producers, i.e., there are multiple pairs of data dependencies, then the above judgment process needs to be performed separately for each pair of data dependencies. Therefore, it is possible that one data receiver needs to synchronize with multiple data producers, or multiple data receivers need to synchronize with one data producer. Regardless of whether a data receiver synchronizes with one data producer, one data receiver synchronizes with multiple data producers, or multiple data receivers synchronize with one data producer, synchronization relationships are always paired. Furthermore, multiple pairs of synchronization relationships on the same data producer or the same data receiver can use different synchronization barriers.

[0095] For example, the operator processing method in this embodiment of the present disclosure performs global optimization on the fusion operators to be generated before generating code. This global optimization can be understood as optimization from the perspective of the entire computation graph. Hardware resources are always limited; therefore, efficiently configuring the fusion operators from the perspective of the entire computation graph optimizes overall performance. Compared to the approach in related technologies that optimizes each fusion operator separately when generating a predefined fusion operator library, the optimization method in this embodiment of the present disclosure performs global optimization by combining multiple operators to be generated before generating code, considering the global picture rather than individual fusion operators, thus achieving global optimization.

[0096] For example, after optimizing the generated fusion operator as a whole, internal optimization can be performed on each fusion operator. Taking the first fusion operator as an example, the first fusion operator includes multiple operators corresponding to multiple operator nodes included in the first subgraph. For the first fusion operator, optimization processing is performed among the multiple operators included in the first fusion operator to obtain configuration information for multiple operators. The optimization processing includes resource configuration optimization and / or formulating synchronization strategies.

[0097] For example, the first fusion operator includes the Conv operator, the Batch Normalization operator, and the ReLU operator. Resource configuration optimization can be performed on these three operators, and / or synchronization strategies can be specified for them. The process of resource configuration optimization and specifying synchronization strategies can be referred to the above description and will not be repeated here. Performing resource configuration optimization and other processing on a single fusion operator can make full use of hardware resources (registers) to generate high-performance code, reduce inefficient data handling, and improve parallelism.

[0098] For example, in addition to the resource configuration optimization and synchronization strategy formulation mentioned above, optimization processing can also include other optimization processing, such as deriving the parameters between fusion operators. Parameters may include parameters that characterize whether the calculation result data of the fusion operator needs to be written to the cache.

[0099] For example, by pre-determining a global optimization strategy, the optimization manager can optimize all fusion operators that need to be generated, including configuring resources and establishing synchronization mechanisms. Because it takes a global perspective, considering both improving the performance of individual fusion operators and the influence of neighboring fusion operators, it achieves optimal global performance. The optimization manager can formulate optimization strategies and goals for fusion operators in any scenario, eliminating the need for manual optimization of each fusion operator and transforming the optimization work of an infinite number of fusion operators into a finite task of refining the optimization manager.

[0100] For example, after the optimization process is completed, step S230 can be executed to generate the operator codes corresponding to each subgraph. For instance, taking a first subgraph corresponding to a first fusion operator as an example, the first subgraph includes P operator nodes, where P is an integer greater than 1. In step S230, for the first subgraph, the operator codes corresponding to the P operator nodes can be determined; based on the operator codes corresponding to the P operator nodes, the operator code for the first fusion operator corresponding to the first subgraph is obtained. For example, the operator codes corresponding to the P operator nodes can be combined according to a corresponding combination method to obtain the operator code for the first fusion operator.

[0101] For example, in determining the code of the operators corresponding to the P operator nodes, the following operations can be performed for each of the P operator nodes: if the operator corresponding to the operator node is a predefined operator, obtain the configuration parameters and code module corresponding to the predefined operator, and obtain the code of the predefined operator based on the configuration parameters and code module; if the operator corresponding to the operator node is a custom operator, compile the custom operator to obtain the code of the custom operator.

[0102] For example, if the operator corresponding to a certain operator node is a predefined operator, the corresponding predefined operator is found in the predefined underlying operator library, and the code template of the predefined operator is extracted. The configuration information of the predefined operator is obtained, and the configuration information is filled into the code template to obtain the terminal code of the predefined operator. The configuration information may include the configuration information determined in the above optimization process, and may also include other configuration information. If the operator corresponding to a certain operator node is a custom operator, the operator code of the custom operator can be obtained using a compiler, assembler, etc. After obtaining the operator codes of multiple operators contained in each fusion operator, the operator codes are combined in a predetermined manner to obtain the operator code of the fusion operator.

[0103] For example, in step S240, the operator codes corresponding to the N subgraphs can be combined to obtain the code for executing the computation process of the computation graph. For example, if the N subgraphs correspond to fusion operators C1 to Cs, after obtaining the codes for fusion operators C1 to Cs, the codes for fusion operators C1 to Cs can be combined according to the corresponding combination method to obtain the code corresponding to the computation graph. The code for the computation process can also include other information besides the operator codes for the fusion operators. Therefore, during combination, codes for other information, such as codes for storage information and codes for parameter information, can also be combined to obtain the code for executing the computation process of the computation graph.

[0104] For example, when there are N subgraphs, including both multi-node and single-node subgraphs, in step S230, in addition to determining the operator codes of multiple fusion operators corresponding to the multi-node subgraphs, the operator codes of non-fusion operators corresponding to the single-node subgraphs can also be obtained based on a predefined underlying operator library and / or custom operators. Then, the operator codes of fusion operators, non-fusion operators, and other information can be combined to obtain the code used to execute the computation process of the computation graph.

[0105] The operator processing method of at least one embodiment of the present disclosure reduces the workload of developing and optimizing fusion operators and improves the generalization ability of fusion operators.

[0106] The operator processing method of at least one embodiment of the present disclosure abstracts the commonalities in the network structure, does not include information on specific cases, has a wide range of applications, and strong generalization ability.

[0107] The operator processing method of at least one embodiment of the present disclosure can generate corresponding high-performance fusion operator code by configuring specific parameters for specific situations, thereby reducing the workload of fusion operator development and maintenance.

[0108] The operator processing method of at least one embodiment of this disclosure, from a global perspective, configures resources and sets up synchronization mechanisms among fusion operators, which can more easily achieve optimal global performance.

[0109] Figure 7 A schematic block diagram of an operator processing apparatus 700 provided in at least one embodiment of the present disclosure is shown.

[0110] For example, such as Figure 7 As shown, the operator processing device 700 includes an acquisition module 710, a splitting module 720, a determination module 730, and a code module 740.

[0111] The acquisition module 710 is configured to acquire a computation graph describing the computation process, wherein the computation graph contains M operator nodes and the connection relationships between the M operator nodes, and each of the M operator nodes corresponds to at least one operator. The acquisition module 710 can, for example, execute... Figure 2 Step S210 is described.

[0112] The splitting module 720 is configured to split the computation graph into N subgraphs, each of which contains at least one operator node from the M operator nodes. The N subgraphs include a first subgraph, which corresponds to K first fusion operators. These K first fusion operators are obtained by fusing multiple operators corresponding to multiple operator nodes included in the first subgraph. For example, the splitting module 720 can execute... Figure 2 Step S220 is described.

[0113] The determining module 730 is configured to determine the operator codes corresponding to the N subgraphs, wherein the operator codes corresponding to the N subgraphs include the operator codes of the K first fusion operators corresponding to the first subgraph. The determining module 730 can, for example, execute... Figure 2 Step S230 is described.

[0114] Code module 740 is configured to obtain code for performing the calculation process based on the operator codes corresponding to the N subgraphs respectively. For example, code module 740 can execute... Figure 2 Step S240 is described.

[0115] For example, the acquisition module 710, the splitting module 720, the determining module 730, and the code module 740 can be hardware, software, firmware, or any feasible combination thereof. For example, the acquisition module 710, the splitting module 720, the determining module 730, and the code module 740 can be dedicated or general-purpose circuits, chips, or devices, or they can be a combination of a processor and memory. The embodiments of this disclosure do not limit the specific implementation of the above-mentioned units.

[0116] It should be noted that in the embodiments of this disclosure, each unit of the operator processing device 700 corresponds to each step of the aforementioned operator processing method. For the specific functions of the operator processing device 700, please refer to the relevant description of the operator processing method, which will not be repeated here. Figure 7 The components and structures of the operator processing device 700 shown are merely exemplary and not limiting. The operator processing device 700 may also include other components and structures as needed.

[0117] For example, in an embodiment of the operator processing apparatus provided in this disclosure, the first subgraph includes P operator nodes, where P is an integer greater than 1. The determining module 730 is further configured to: determine the operator codes corresponding to the P operator nodes for the first subgraph; and obtain the operator codes of the K first fusion operators corresponding to the first subgraph based on the operator codes corresponding to the P operator nodes.

[0118] For example, in an embodiment of the operator processing apparatus provided in this disclosure, the determining module 730 is further configured to perform the following operations for each of the P operator nodes: if the operator corresponding to the operator node is a predefined operator, obtain the configuration parameters and code module corresponding to the predefined operator, and obtain the code of the predefined operator based on the configuration parameters and code module; if the operator corresponding to the operator node is a custom operator, compile the custom operator to obtain the code of the custom operator.

[0119] For example, in an embodiment of the present disclosure, the determining module 730 is further configured to combine the codes of the operators corresponding to the P operator nodes respectively to obtain the operator codes of the K first fusion operators.

[0120] For example, in an embodiment of the present disclosure, the splitting module 720 is further configured to: if the target attributes of the operators corresponding to Q operator nodes are the same, then the Q operator nodes are divided into a subgraph, wherein the target attribute includes at least one of type attribute, calculation attribute and data transmission attribute, and Q is an integer greater than 1.

[0121] For example, in an embodiment of the present disclosure, the splitting module 720 is further configured to: if the operators corresponding to the Q operator nodes are all operators of the same type, then the Q operator nodes are divided into a subgraph.

[0122] For example, in an embodiment of the present disclosure, the splitting module 720 is further configured to: if the operators corresponding to the Q operator nodes are configured to transmit data through registers, then the Q operator nodes are divided into a subgraph.

[0123] For example, in an embodiment of the present disclosure, the splitting module 720 is further configured to: if the operators corresponding to the Q operator nodes are configured to run on the same computing unit, then the Q operator nodes are divided into a subgraph.

[0124] For example, in an embodiment of the present disclosure, the splitting module 720 is further configured to: if the type and execution order of the operators corresponding to the Q operator nodes are consistent with the type and execution order of the operators included in the predetermined fusion operator, then the Q operator nodes are divided into a subgraph.

[0125] For example, in an operator processing apparatus provided in one embodiment of this disclosure, the Q operator nodes are either consecutive or parallel to each other in the execution order determined according to the computation graph.

[0126] For example, in an embodiment of the present disclosure, at least one of the K first fusion operators includes a plurality of operators, which are connected in sequence and executed sequentially; for two adjacent operators among the plurality of operators, the calculation result data of the preceding operator in the execution order is used as the input data of the following operator.

[0127] For example, in an embodiment of the operator processing apparatus provided in this disclosure, the N subgraphs further include a second subgraph, which corresponds to R second fusion operators. The R second fusion operators are operators obtained by fusing multiple operators corresponding to multiple operator nodes included in the second subgraph, where R is an integer not less than 1. The operator codes corresponding to the N subgraphs respectively also include the operator codes of the R second fusion operators corresponding to the second subgraph. The operator processing apparatus further includes an optimization module, which is configured to: before determining the operator codes corresponding to the N subgraphs respectively, optimize the K first fusion operators and the R second fusion operators to obtain configuration information for the K first fusion operators and the R second fusion operators respectively.

[0128] For example, in an embodiment of the present disclosure, the operator processing apparatus is further configured to: optimize the resource configuration of the K first fusion operators and the R second fusion operators to obtain optimized resource configuration information for each first fusion operator and each second fusion operator, wherein the resources include computing resources and / or storage resources.

[0129] For example, in an embodiment of the present disclosure, the operator processing device is further configured to: formulate a synchronization strategy for the K first fusion operators and the R second fusion operators based on the calculation process of the computation graph, so as to obtain synchronization configuration information for the K first fusion operators and the R second fusion operators.

[0130] For example, in an embodiment of the operator processing apparatus provided in this disclosure, the optimization module is further configured to perform the following operation on at least two of the K first fusion operators and R second fusion operators: if the two fusion operators have a data dependency relationship, and the scheduling unit identifier of the data receiver is different from the scheduling unit identifier of the data producer and the scheduling unit identifier between the data producer and the data receiver, then the two fusion operators are configured to perform data synchronization processing, wherein the data producer is one of the two fusion operators and the data producer is the other.

[0131] For example, in an embodiment of the operator processing apparatus provided in this disclosure, the N subgraphs correspond to multiple fusion operators, and each fusion operator is an operator obtained by fusing multiple operators corresponding to multiple operator nodes included in a subgraph. The optimization module is further configured to: before determining the operator codes corresponding to the N subgraphs respectively, optimize the multiple fusion operators to obtain configuration information for the multiple fusion operators respectively, wherein the optimization process includes resource configuration optimization and / or formulating a synchronization strategy.

[0132] For example, in an embodiment of the operator processing apparatus provided in this disclosure, the first fusion operator includes multiple operators corresponding to multiple operator nodes included in the first subgraph. The optimization module is further configured to: perform optimization processing on the multiple operators included in the first fusion operator to obtain configuration information for the multiple operators, wherein the optimization processing includes resource configuration optimization and / or formulating a synchronization strategy.

[0133] For example, in an embodiment of the present disclosure, the code module 740 is further configured to combine the operator codes corresponding to the N subgraphs to obtain code for executing the calculation process.

[0134] At least one embodiment of this disclosure also provides an electronic device including a processor and a memory, the memory storing one or more computer program modules. The one or more computer program modules are stored in the memory and configured to be executed by the processor. The one or more computer program modules include instructions for implementing the operator processing method of any of the above embodiments, and are therefore used to implement the operator processing method of any of the above embodiments when executed by the processor. This electronic device can abstract the commonalities in the network structure during the generation of fusion operators, without including information specific to certain situations, making it applicable to various scenarios, with a wide range of applications and strong generalization ability.

[0135] Figure 8 This is a schematic block diagram of an electronic device provided for some embodiments of this disclosure. For example... Figure 8 As shown, the electronic device 800 includes a processor 810 and a memory 820. The memory 820 stores non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor 810 is used to execute the non-transitory computer-readable instructions, which, when executed by the processor 810, perform one or more steps in the operator processing method described above. The memory 820 and the processor 810 can be interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0136] For example, the processor 810 and memory 820 can be located on the server side (or in the cloud).

[0137] For example, processor 801 can control other components in electronic device 800 to perform desired functions. Processor 810 can be a central processing unit (CPU), a graphics processing unit (GPU), or other form of processing unit with data processing and / or program execution capabilities. For example, the central processing unit (CPU) can be an x86 or ARM architecture. Processor 810 can be a general-purpose processor or a special-purpose processor, and can control other components in electronic device 800 to perform desired functions.

[0138] For example, memory 820 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and processor 810 may run one or more computer program modules to implement various functions of electronic device 800. Various application programs and various data, as well as various data used and / or generated by the application programs, may also be stored in the computer-readable storage medium.

[0139] For example, in some embodiments, the electronic device 800 can be a mobile phone, tablet computer, electronic paper, television, monitor, laptop computer, digital photo frame, navigator, wearable electronic device, smart home device, etc.

[0140] For example, electronic device 800 may include a display panel, which can be used for image segmentation, etc. For example, the display panel can be a rectangular panel, a circular panel, an elliptical panel, or a polygonal panel. Furthermore, the display panel can be not only a flat panel, but also a curved panel, or even a spherical panel.

[0141] For example, electronic device 800 can have touch functionality, that is, electronic device 800 can be a touch device.

[0142] It should be noted that, in the embodiments of this disclosure, the specific functions and technical effects of the electronic device 800 can be referred to the description of the operator processing method above, and will not be repeated here.

[0143] Figure 9 This is a schematic block diagram of another electronic device provided in some embodiments of this disclosure. The electronic device 900 is, for example, suitable for implementing the operator processing method provided in the embodiments of this disclosure. The electronic device 900 may be a terminal device, etc. It should be noted that... Figure 9 The illustrated electronic device 900 is merely an example and does not impose any limitation on the functionality and scope of use of the embodiments of this disclosure.

[0144] like Figure 9As shown, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 910, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 920 or a program loaded from a storage device 980 into a random access memory (RAM) 930. The RAM 930 also stores various programs and data required for the operation of the electronic device 900. The processing unit 910, the ROM 920, and the RAM 930 are interconnected via a bus 940. An input / output (I / O) interface 950 is also connected to the bus 940.

[0145] Typically, the following devices can be connected to I / O interface 950: input devices 960 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 970 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 980 including, for example, magnetic tapes, hard disks, etc.; and communication devices 990. Communication device 990 allows electronic device 900 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 9 An electronic device 900 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 900 may alternatively implement or have more or fewer devices.

[0146] For example, according to embodiments of this disclosure, the above-described operator processing method can be implemented as a computer software program. For instance, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program including program code for performing the above-described operator processing method. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 990, or installed from a storage device 980, or installed from a ROM 920. When the computer program is executed by the processing device 910, it can implement the functions defined in the operator processing method provided in embodiments of this disclosure.

[0147] At least one embodiment of this disclosure also provides a computer-readable storage medium storing non-transitory computer-readable instructions that, when executed by a computer, can implement the aforementioned operator processing method. Using this computer-readable storage medium, commonalities in the network structure can be abstracted during the generation of fusion operators. It does not contain information specific to any particular situation, making it applicable to various scenarios, with a wide range of applications and strong generalization ability.

[0148] Figure 10 This is a schematic diagram of a storage medium provided for some embodiments of this disclosure. For example... Figure 10As shown, the storage medium 1000 stores non-transitory computer-readable instructions 1010. For example, when the non-transitory computer-readable instructions 1010 are executed by a computer, one or more steps in the operator processing method described above are performed.

[0149] For example, the storage medium 1000 can be used in the aforementioned electronic device 800. For example, the storage medium 1000 can be... Figure 8 The memory 820 in the illustrated electronic device 800. For example, a description of the storage medium 1000 can be found here. Figure 8 The corresponding description of the memory 820 in the illustrated electronic device 800 will not be repeated here.

[0150] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0151] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0152] The following points need to be explained:

[0153] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0154] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0155] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.

Claims

1. An operator processing method for a chip, comprising: obtaining, by the chip, a computation graph for describing a computation process, wherein the computation graph comprises M operator nodes and connection relationships between the M operator nodes, each of the M operator nodes corresponds to at least one operator; splitting, by the chip, the computation graph to obtain N subgraphs, wherein each of the N subgraphs comprises at least one of the M operator nodes, and the N subgraphs comprise a first subgraph; determining, by the chip, operator codes corresponding to the N subgraphs, respectively; and generating, by the chip, a code for executing the computation process based on the operator codes corresponding to the N subgraphs, respectively. The determining, by the chip, of the operator codes corresponding to the N subgraphs, respectively, comprises: generating K first fusion operators based on the first subgraph, wherein each of the K first fusion operators is an operator obtained by fusing a plurality of operators corresponding to a plurality of operator nodes included in the first subgraph, and an intermediate calculation result of each of the first fusion operators is configured to transmit data through a register of the chip; and determining operator codes corresponding to the K first fusion operators, wherein the operator codes corresponding to the N subgraphs, respectively, comprise the operator codes of the K first fusion operators corresponding to the first subgraph. M, N and K are all integers greater than or equal to 1.

2. The method of claim 1, wherein: the plurality of operator nodes included in the first subgraph are P operator nodes, wherein P is an integer greater than 1; The determining of the operator codes corresponding to the N subgraphs, respectively, comprises: determining, for the first subgraph, codes of operators corresponding to the P operator nodes, respectively; and obtaining, based on the codes of the operators corresponding to the P operator nodes, respectively, the operator codes of the K first fusion operators corresponding to the first subgraph.

3. The method of claim 2, wherein, The determining, for the first subgraph, of the codes of the operators corresponding to the P operator nodes, respectively, comprises: for each of the P operator nodes, performing the following operations: in a case where an operator corresponding to the operator node is a predefined operator, obtaining, for the predefined operator, a configuration parameter and a code module corresponding to the predefined operator, and obtaining a code of the predefined operator based on the configuration parameter and the code module; in a case where the operator corresponding to the operator node is a custom operator, compiling the custom operator to obtain a code of the custom operator.

4. The method of claim 2, wherein, The obtaining, based on the codes of the operators corresponding to the P operator nodes, respectively, of the operator codes of the K first fusion operators corresponding to the first subgraph comprises: combining the codes of the operators corresponding to the P operator nodes to obtain the operator codes of the K first fusion operators.

5. The method of claim 1, wherein, The splitting of the computation graph to obtain the N subgraphs comprises: if target attributes of Q operator nodes are the same, dividing the Q operator nodes into one subgraph, wherein the target attributes comprise at least one of a type attribute, a calculation attribute and a data transmission attribute, wherein Q is an integer greater than 1.

6. The method of claim 5, wherein, Splitting the computation graph to obtain N subgraphs, including: If the operators corresponding to the Q operator nodes are all operators of the same type, the Q operator nodes are divided into one subgraph.

7. The method of claim 5, wherein, Splitting the computation graph to obtain N subgraphs, including: If the operators corresponding to the Q operator nodes are configured to transmit data through registers, the Q operator nodes are divided into one subgraph.

8. The method of claim 5, wherein, Splitting the computation graph to obtain N subgraphs, including: If the operators corresponding to the Q operator nodes are configured to run on the same computing unit, the Q operator nodes are divided into one subgraph.

9. The method of claim 1, wherein, Splitting the computation graph to obtain N subgraphs, including: If the types and execution orders of the operators corresponding to the Q operator nodes are consistent with the types and execution orders of the operators contained in the predetermined fusion operator, the Q operator nodes are divided into one subgraph.

10. The method according to any one of claims 5 to 9, wherein, The Q operator nodes are continuous or parallel to each other in the execution order determined according to the computation graph.

11. The method of any one of claims 1 to 9, wherein, At least one of the K first fusion operators includes a plurality of operators that are connected in sequence and executed in sequence. For two adjacent operators in the plurality of operators, the calculation result data of the former operator in the execution order is used as the input data of the latter operator.

12. The method of any one of claims 1 to 9, wherein, The N subgraphs further include a second subgraph corresponding to R second fusion operators, the R second fusion operators being operators obtained by fusing a plurality of operators corresponding to a plurality of operator nodes included in the second subgraph, wherein R is an integer not less than 1. The operator codes corresponding to the N subgraphs further include operator codes of the R second fusion operators corresponding to the second subgraph. The method further includes: Before determining the operator codes corresponding to the N subgraphs, performing optimization processing on the K first fusion operators and the R second fusion operators to obtain configuration information for the K first fusion operators and the R second fusion operators.

13. The method of claim 12, wherein, The optimization processing on the K first fusion operators and the R second fusion operators to obtain configuration information for the K first fusion operators and the R second fusion operators includes: Resource configuration optimization is performed on the K first fusion operators and the R second fusion operators to obtain optimized resource configuration information for each of the first fusion operators and each of the second fusion operators, wherein the resources include computing resources and / or storage resources.

14. The method of claim 12, wherein, The optimization processing on the K first fusion operators and the R second fusion operators to obtain configuration information for the K first fusion operators and the R second fusion operators includes: Based on the calculation process of the computation graph, a synchronization strategy is formulated for the K first fusion operators and the R second fusion operators to obtain synchronization configuration information for the K first fusion operators and the R second fusion operators.

15. The method of claim 12, wherein, formulate a synchronization strategy for the K first fusion operators and the R second fusion operators based on a computing process of the computing graph, to obtain synchronization configuration information for the K first fusion operators and the R second fusion operators, including: at least for two fusion operators in the K first fusion operators and the R second fusion operators, performing the following operations: if the two fusion operators have a data dependency relationship, and the scheduling unit identifier of the data receiver and the scheduling unit identifier of the data producer and the scheduling unit identifier between the data producer and the data receiver are all different, configuring the two fusion operators to perform data synchronization processing, wherein the data producer is one of the two fusion operators, and the data producer is the other.

16. The method according to any one of claims 1 to 9, wherein, The first fusion operator includes a plurality of operators corresponding to a plurality of operator nodes included in the first subgraph. The method further includes: for the first fusion operator, performing optimization processing between the plurality of operators included in the first fusion operator to obtain configuration information for the plurality of operators, wherein the optimization processing includes resource configuration optimization and / or formulating a synchronization strategy.

17. The method according to any one of claims 1 to 9, wherein, based on the operator codes corresponding to the N subgraphs respectively, obtaining code for executing the computing process, including: combining the operator codes corresponding to the N subgraphs respectively to obtain code for executing the computing process.

18. An information processing apparatus for a chip, comprising: an acquisition module configured to acquire a computing graph for describing a computing process, wherein the computing graph includes M operator nodes and connection relationships between the M operator nodes, and each of the M operator nodes corresponds to at least one operator; a splitting module configured to split the computing graph to obtain N subgraphs, wherein each of the N subgraphs includes at least one of the M operator nodes, and the N subgraphs include a first subgraph; a determination module configured to determine operator codes corresponding to the N subgraphs respectively; and a code module configured to obtain code for executing the computing process based on the operator codes corresponding to the N subgraphs respectively, wherein the determination module is further configured to: based on the first subgraph, generate K first fusion operators, wherein the K first fusion operators are operators obtained by fusing a plurality of operators corresponding to a plurality of operator nodes included in the first subgraph, and an intermediate computing result of each of the first fusion operators is configured to transmit data through a register of the chip; and determine operator codes corresponding to the K first fusion operators, wherein the operator codes corresponding to the N subgraphs respectively include the operator codes of the K first fusion operators corresponding to the first subgraph, wherein M, N and K are integers not less than 1.

19. An electronic device, comprising: a processor; a memory storing one or more computer program modules; wherein the one or more computer program modules are configured to be executed by the processor to implement the operator processing method of any one of claims 1-17.

20. A computer-readable storage medium storing non-transitory computer-readable instructions that, when executed by a computer, implement the operator processing method of any one of claims 1-17.

Images