Operator processing method and related apparatus

Abstract

Provided in the present application is an operator processing method, comprising: acquiring topological information of a computational graph, wherein the topological information records a plurality of operators comprised in the computational graph, graph input data, graph output data, graph intermediate data, and a dependency relationship between the plurality of operators; and constructing an operator execution queue according to the topological information of the computational graph, wherein each element in the operator execution queue corresponds to one operator and input data and output data of the operator, the input data of the operator is at least one of the graph input data and the graph intermediate data, the output data of the operator is at least one of the graph intermediate data and the graph output data, and an execution sequence of the plurality of operators is determined according to the dependency relationship between the plurality of operators. The method supports a user in performing customized graph formation according to operators, and constructing an operator execution queue on the basis of topological information of a computational graph, so as to construct an execution graph in the form of the operator execution queue; and the method can be flexibly applied to all scenarios in which batch scheduling of operators is required, thereby meeting service requirements.

Description

Operator processing method and related device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 29, 2023, with application number 202310792403.3, entitled “A method for executing a computational graph”, and the Chinese patent application filed with the State Intellectual Property Office of China on September 27, 2023, with application number 202311281657.5, entitled “An operator processing method and related device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to an operator processing method, device, computer-readable storage medium, and computer program product. Background Art

[0003] An operator is a computing unit. For example, a binary operator is typically a computing unit in binary code. An operator typically includes host logic and device logic. The host includes a central processing unit (CPU) and host memory, and the device includes an external processing unit and device memory. The external processing unit can be a neural processing unit (NPU) or a graphics processing unit (GPU). Different types of processors can implement different instruction set architectures.

[0004] Operator execution typically involves interaction between the host and the device. Assuming the operator's input data is already in device memory, the main time-consuming steps include: the host deriving the output shape based on the input dimensions (shape), allocating device memory, calculating the operator's sharding parameters (tiling data), transferring the tilling data to the device, issuing the computation task to the device, and the device executing the computation task.

[0005] In many business scenarios, such as training neural networks or performing inference based on trained neural networks, multiple operators are often executed. When executing multiple operators, a compilation method, such as the Tensor Virtual Machine (TVM) or Accelerated Linear Algebra (XLA), can be used to compile the computational graph containing multiple operators into an intermediate representation (IR). The IR is used to describe the execution process, and then an execution graph is generated, which enables batch operator calls.

[0006] However, the above method requires a complex compilation optimization process and offline generation of execution graphs. It is a heavyweight operator scheduling technology and is not applicable to some scenarios that require lightweight batch operator calls, such as implementing the acceleration library application programming interface (API).

[0007] Summary of the Invention

[0008] This application provides an operator processing method that constructs an operator execution queue based on the topological information of a computation graph, thereby building an execution graph in the form of an operator execution queue. This method can be flexibly applied to all scenarios requiring batch operator scheduling, meeting business needs. This application also provides an operator processing device, computer-readable storage medium, and computer program product corresponding to the above-mentioned operator processing method.

[0009] In the first aspect, the present application provides an operator processing method. The method can be performed by an operator processing device. The operator processing device can be a software device, which can be an independent software package, or a cloud service, such as a cloud service provided through an interface (such as a composition interface), which supports users to customize group diagrams based on operators and represent the topology diagram in the form of a queue. The above-mentioned software device can be deployed in a computing device cluster (for example, a computing device that executes the operator or a third-party computing device cluster), and the computing device cluster executes the program code of the software device, thereby executing the operator processing method of the present application. The operator processing device can also be a hardware device, such as a computing device or a computing device cluster with an operator processing function. When the operator processing device is running, the operator processing method of the present application is executed.

[0010] Specifically, the operator processing device obtains the topology information of the computation graph, which records the multiple operators, graph input data, graph output data, graph intermediate data, and the dependencies between the multiple operators included in the computation graph. Then, based on the topology information of the computation graph, an operator execution queue is constructed. Each element in the operator execution queue corresponds to an operator in the computation graph and its input data and output data. The operator input data is at least one of the graph input data or the graph intermediate data, and the operator output data is at least one of the graph intermediate data or the graph output data. The execution order of the multiple operators in the operator execution queue is determined based on the dependencies between the multiple operators.

[0011] This method supports users to customize group graphs based on operators and construct operator execution queues based on the topological information of the computation graph, thereby building an execution graph in the form of an operator execution queue. It can be flexibly applied to all scenarios requiring batch operator scheduling, including lightweight batch operator scheduling scenarios such as acceleration library construction, meeting business needs.

[0012] In some possible implementations, the operator processing device can partition and number graph input data, graph output data, and graph intermediate data. Each element in the operator execution queue records an operator in the computation graph, as well as the numbers of the operator's input data and output data. This allows for binding tensor numbers to operators. The tensor numbers serve as node information in the operator execution queue and can be used as tensor indices during graph execution when the operator is subsequently executed.

[0013] In some possible implementations, the operator processing device can determine the size of the graph workspace (also referred to as the context environment in some cases) based on the dimensional information of the graph input data and the correspondence between the operator corresponding to each element in the operator execution queue and the operator's input data and output data. The operator processing device can then apply for memory as the graph workspace based on the size of the graph workspace. The operator processing device can apply for device memory of the corresponding size as the graph workspace, or the operator processing device can apply for device memory slightly larger than the above size as the graph workspace to prevent insufficient graph workspace.

[0014] This method deduce the size of the graph workspace based on the input dimension and the correspondence between operators and input and output data in the operator execution queue before executing the operator sequence, and applies for corresponding memory as the graph workspace based on the size. This method eliminates the need to apply for memory multiple times, reduces the number of memory applications, and improves overall operating efficiency.

[0015] In some possible implementations, for a graph intermediate data (such as a graph intermediate tensor), its life cycle ends immediately after all operators that rely on the graph intermediate data as input finish running. The dimensional information of the graph intermediate data is relatively large in typical application scenarios. Without considering memory transfer, reusing the device-side memory space as much as possible can improve memory utilization. Based on this, the operator processing device can obtain the dimensional information of the graph intermediate data and the dependency information of the graph intermediate data, and the dependency information is used to indicate the operator that depends on the graph intermediate data, and determine the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data according to the dimensional information of the graph intermediate data and the dependency information of the graph intermediate data.

[0016] The method determines the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data by combining the dimension information of the graph intermediate data and the dependency information of the graph intermediate data. It can realize memory reuse for different graph intermediate data and improve memory utilization.

[0017] In some possible implementations, the graph intermediate data includes first graph intermediate data and second graph intermediate data. When an operator that depends on the first graph intermediate data is executed before an operator that outputs the second graph intermediate data, an offset address of the first graph intermediate data is less than or equal to an offset address of the second graph intermediate data.

[0018] Since the life cycles of the first graph intermediate data and the second graph intermediate data do not overlap, the second graph intermediate data can reuse the memory space of the first graph intermediate data, thereby improving memory utilization.

[0019] In some possible implementations, when the dimensional information of the intermediate data of the second figure is less than or equal to the dimensional information of the intermediate data of the first figure, the intermediate data of the second figure can directly reuse the memory space of the intermediate data of the first figure. When the dimensional information of the intermediate data of the second figure is greater than the dimensional information of the intermediate data of the first figure, the offset address of the intermediate data of the third figure after the intermediate data of the first figure can be adjusted. For example, the offset address of the intermediate data of the third figure can be increased to reserve a certain amount of additional storage space for storing the intermediate data of the second figure that is larger than the intermediate data of the first figure, thereby reusing the memory space of the intermediate data of the first figure and improving memory utilization.

[0020] In some possible implementations, for the sharding parameters (tiling data) of an operator, it is usually necessary to load the sharding parameters from the host side to the specified device memory space before the operator is executed. The operator will no longer be used after execution. In order to reduce the number of loading times, this application supports loading all at once. Specifically, the operator processing device can also determine the offset addresses of the sharding parameters of multiple operators in the graph workspace and the sizes of the sharding parameters of multiple operators, and load the sharding parameters of the multiple operators into the graph workspace according to the offset addresses of the sharding parameters of multiple operators in the graph workspace and the sizes of the sharding parameters of multiple operators. In this way, the overhead of loading sharding parameters can be reduced and resource utilization (for example, communication resource utilization) can be improved.

[0021] In some possible implementations, the graph workspace is further provided with a temporary memory. The operator processing device may determine the offset address and size of the temporary memory in the graph workspace based on the amount of temporary data generated by the execution of the multiple operators. For example, the operator processing device may determine the maximum amount of temporary data generated by the execution of the multiple operators as the size of the temporary memory. In this way, the temporary memory can be reused.

[0022] In some possible implementations, the lifecycle of an operator's temporary memory (or temporary data) spans from the start of the operator's execution to its completion. Based on this, temporary memory for different operators can be reused. Specifically, the temporary memory for multiple operators can be configured to have the same offset address. This allows temporary memory for different operators to reuse memory space, improving memory utilization.

[0023] In some possible implementations, the operator processing device can asynchronously execute different computation graphs. Specifically, while the device side (such as the NPU on the device side) is still calculating, the operator processing device can prepare the computing resources for the next computation graph in advance, thereby reducing the waiting time on the device side and increasing the operator throughput.

[0024] In some possible implementations, a computation graph includes a first computation graph and a second computation graph. The first computation graph includes the same operators as the second computation graph, the dependencies between the operators in the first computation graph and the second computation graph are the same, and the dimension information of the data in the first computation graph and the second computation graph are the same. Accordingly, when executing the first computation graph, the operator processing device may adjust the address of the graph input data to the address of the graph input data of the second computation graph.

[0025] In this way, after the first computation graph is executed, the operator processing device can directly reuse the graph workspace of the first computation graph to execute the second computation graph. This method improves the operator throughput on the device side by asynchronously executing different computation graphs, thereby improving the overall operation speed.

[0026] In some possible implementations, at least one of the multiple operators is a compiled operator, such as a binary operator. A binary operator is a computational unit in binary code that can be executed on the device without compilation. By using at least one compiled operator to construct a computation graph, compilation time can be reduced, achieving lightweight compilation. When the computation graph uses all compiled operators, no compilation time can even be achieved, thus lightweighting operator scheduling.

[0027] In a second aspect, the present application provides an operator processing device, characterized in that the device includes:

[0028] An acquisition module is used to obtain topological information of a computational graph, wherein the topological information records the multiple operators, graph input data, graph output data, graph intermediate data, and dependency relationships of the multiple operators included in the computational graph;

[0029] A construction module is used to construct an operator execution queue based on the topological information of the computational graph, where each element in the operator execution queue corresponds to an operator in the computational graph and the input data and output data of the operator, the input data of the operator is at least one of the graph input data or the graph intermediate data, and the output data of the operator is at least one of the graph intermediate data or the graph output data. The execution order of multiple operators in the operator execution queue is determined according to the dependency relationship between the multiple operators.

[0030] In some possible implementations, the construction module is further configured to:

[0031] Partition numbering of the graph input data, the graph output data, and the graph intermediate data;

[0032] Each element in the operator execution queue records an operator in the computation graph, the number of the operator's input data, and the number of the operator's output data.

[0033] In some possible implementations, the apparatus further includes:

[0034] A memory planning module is used to determine the size of the graph workspace based on the dimensional information of the graph input data and the correspondence between the operator corresponding to each element in the operator execution queue and the input data and output data of the operator, and to apply for memory as the graph workspace based on the size of the graph workspace.

[0035] In some possible implementations, the memory planning module is further configured to:

[0036] Obtain the dimension information of the graph intermediate data and the dependency information of the graph intermediate data, where the dependency information is used to indicate the operator that depends on the graph intermediate data; determine the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data based on the dimension information of the graph intermediate data and the dependency information of the graph intermediate data.

[0037] In some possible implementations, the graph intermediate data includes first graph intermediate data and second graph intermediate data. When an operator that depends on the first graph intermediate data is executed before an operator that outputs the second graph intermediate data, the offset address of the first graph intermediate data is less than or equal to the offset address of the second graph intermediate data.

[0038] In some possible implementations, the memory planning module is further configured to:

[0039] Determining the offset addresses of the sharding parameters of the multiple operators in the graph workspace and the sizes of the sharding parameters of the multiple operators;

[0040] The sharding parameters of the multiple operators are loaded into the graph workspace according to the offset addresses of the sharding parameters of the multiple operators in the graph workspace and the sizes of the sharding parameters of the multiple operators.

[0041] In some possible implementations, the graph workspace is further provided with a temporary memory;

[0042] The memory planning module is also used to:

[0043] The offset address and size of the graph workspace in the temporary memory are determined according to the amount of temporary data generated by executing the multiple operators.

[0044] In some possible implementations, the temporary memories of the multiple operators have the same offset address.

[0045] In some possible implementations, the computation graph includes a first computation graph and a second computation graph, the operators included in the first computation graph are the same as the operators included in the second computation graph, the dependencies between the operators in the first computation graph are the same as the dependencies between the operators in the second computation graph, and the dimension information of the data in the first computation graph is the same as the dimension information of the data in the second computation graph;

[0046] The device further comprises:

[0047] An adjustment module is used to adjust the address of graph input data to the address of graph input data of the second computation graph when executing the first computation graph.

[0048] In a third aspect, the present application provides an operator processing device. The operator processing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is configured to execute instructions stored in the at least one memory, so that the operator processing device performs the operator processing method described in the first aspect or any implementation of the first aspect.

[0049] In a fourth aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and the instructions instruct a computing device or a computing device cluster to execute the operator processing method described in the above-mentioned first aspect or any implementation of the first aspect.

[0050] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when run on a computing device or a computing device cluster, enables the computing device or the computing device cluster to execute the operator processing method described in the first aspect or any one of the implementations of the first aspect.

[0051] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0053] FIG1 is a schematic diagram of a process of operator scheduling provided in an embodiment of the present application;

[0054] FIG2 is a hardware structure diagram of a server provided in an embodiment of the present application;

[0055] FIG3 is a flow chart of an operator processing method provided in an embodiment of the present application;

[0056] FIG4 is a schematic diagram of a patterning interface provided in an embodiment of the present application;

[0057] FIG5 is a schematic diagram of constructing an operator execution queue based on a computation graph according to an embodiment of the present application;

[0058] FIG6 is a schematic diagram of an operator processing device according to an embodiment of the present application applying for memory as a graph workspace;

[0059] FIG7 is a schematic diagram of a planning sharding parameter and an offset address and size temporarily stored in a graph workspace provided by an embodiment of the present application;

[0060] FIG8 is a schematic diagram of the offset address and size of intermediate data of a planning graph in a graph workspace provided by an embodiment of the present application;

[0061] FIG9 is a schematic diagram of updating a data pointer of graph intermediate data and a graph workspace pointer to execute a computation graph, provided by an embodiment of the present application;

[0062] FIG10 is a schematic diagram of a method for identifying graph intermediate data and updating a data pointer of the graph intermediate data provided by an embodiment of the present application;

[0063] FIG11 is a schematic diagram of a process for executing different computation graphs in parallel according to an embodiment of the present application;

[0064] FIG12 is a schematic structural diagram of an operator processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0066] First, some technical terms involved in the embodiments of this application are introduced.

[0067] An operator (OP) refers to an operation performed on data. Depending on the type of operation, operators can be divided into computing operators (such as the convolution operator Conv2D, the maximum pooling operator MaxPool), and communication operators (such as the collection operator AllGather, and the broadcast operator Broadcast). Among them, operators (such as heterogeneous operators) can include host logic and device logic. The host includes the CPU and host memory, and the device includes an external processing unit (such as an NPU or GPU) and device memory. The host logic may include logic for generating an execution graph (also called a run graph). The execution graph may be a sequence of operators (kernels) that can be executed directly on the device, and the device logic may include logic for running the execution graph.

[0068] Operator scheduling refers to the process of controlling operator execution, including data preparation, task dispatching, and obtaining calculation results. Operator scheduling designs the interaction between the host and device sides. As shown in Figure 1, considering that the operator's input data is already in the device memory, the main scheduling time consumption includes the host side deducing the output dimension information (also called the output shape) based on the input dimension (shape) information, requesting device-side memory, calculating the operator's sharding parameter Tilling Data, moving the Tilling Data to the device, dispatching the calculation task to the device, and executing the calculation task on the device side.

[0069] Batch operator scheduling is required in many business scenarios. For example, in the scenario of training neural networks, or in the scenario of reasoning based on trained neural networks, it is usually necessary to batch schedule multiple operators. The operator scheduling method based on artificial intelligence (AI) compilation supports batch operator scheduling. Specifically, the computation graph including multiple operators is compiled into an intermediate representation (IR) through the Tensor Virtual Machine (TVM) or Accelerated Linear Algebra (XLA). Then, an execution graph can be generated based on the IR, and batch operator calls are implemented through the execution graph.

[0070] However, the above method requires a complex compilation optimization process, among which the graph fusion process is complicated and tedious, and requires offline generation of the execution graph. It is a heavyweight operator scheduling technology and is not applicable to some scenarios that require lightweight batch operator calls, such as implementing the acceleration library application programming interface (API).

[0071] In view of this, the present application provides an operator processing method. The method can be performed by an operator processing device. The operator processing device can be a software device, which can be an independent software package, or a cloud service, such as a cloud service provided through an interface (such as a composition interface), which supports users to customize group diagrams based on operators and represent the topology diagram in the form of a queue. The above-mentioned software device can be deployed in a computing device cluster (for example, a computing device that executes the operator or a third-party computing device cluster), and the computing device cluster executes the program code of the software device, thereby executing the operator processing method of the present application. The operator processing device can also be a hardware device, such as a computing device or a computing device cluster with an operator processing function. When the operator processing device is running, the operator processing method of the present application is executed.

[0072] Specifically, the operator processing device obtains the topology information of the computation graph, which records the multiple operators, graph input data, graph output data, graph intermediate data, and the dependencies between the multiple operators included in the computation graph. Then, based on the topology information of the computation graph, an operator execution queue is constructed. Each element in the operator execution queue corresponds to an operator in the computation graph and its input data and output data. The operator input data is at least one of the graph input data or the graph intermediate data, and the operator output data is at least one of the graph intermediate data or the graph output data. The execution order of the multiple operators in the operator execution queue is determined based on the dependencies between the multiple operators.

[0073] This method supports users to customize group graphs based on operators and construct operator execution queues based on the topological information of the computation graph, thereby building an execution graph in the form of an operator execution queue. It can be flexibly applied to all scenarios requiring batch operator scheduling, including lightweight batch operator scheduling scenarios such as acceleration library construction, meeting business needs.

[0074] The operator processing method of the present application can be applied to scenarios where operators need to be called in batches to complete computing tasks. Typical scenarios include the construction of various acceleration library APIs or large model reasoning acceleration libraries. Among them, the acceleration API construction includes the construction of the Deep Neural Network (DNN) acceleration library API, and the large model reasoning acceleration library can include various large language models (LLM), especially conversational language models.

[0075] The operator processing device that executes the operator processing method can be implemented by a computing device or a computing device cluster. For example, a computing device or a computing device cluster can execute the operator processing method of the present application when building a large model inference acceleration library. Among them, the computing device can be a server or a terminal. The server can be a cloud server in a cloud environment, or a self-owned physical server, and the terminal can include but is not limited to a desktop computer, a laptop computer, and a smart phone. The computing device can adopt a single-machine single-card architecture, or a single-machine multi-card architecture, and the computing device cluster can be a multi-machine multi-card architecture. It should be noted that the "machine" in the above x machine x card represents the host, and the "card" represents an accelerator card and other devices. Among them, the accelerator card can include but is not limited to a GPU or an NPU. The following is an example of a computing device as a server.

[0076] 2 , a hardware structure diagram of a server is shown. The server 20 includes a host 22 and at least one device 24. The host 22 and the at least one device 24 are connected.

[0077] Host 22 includes a processor and memory. The processor may be a central processing unit (CPU), and the memory may be a dual in-line memory module (DIMM). Specifically, the DIMM may be a double data rate (DDR) type, for example, a DDR4 DIMM. In the example of FIG2 , host 22 includes four CPUs and four DDR4 DIMM groups. Each CPU is connected to one DDR4 DIMM group, and each DDR4 DIMM group includes eight DDR4 DIMMs. Multiple CPUs of host 22 can be connected to form a hydra mesh.

[0078] Optionally, the host 22 further includes an interface, such as one or more of a Serial Advanced Technology Attachment (SATA) interface, a next-generation non-volatile memory express (NVMe) interface, and a Gigabit Ethernet (GE) interface. The host 22 may further include a memory. The memory may include a SATA-supported memory or an NVMe-supported memory, such as a SATA-supported mechanical hard disk or an NVMe-supported solid-state drive (SSD).

[0079] The device 24 includes an accelerator card, such as an NPU or GPU. FIG2 illustrates an example in which the device side includes eight NPUs. In other possible implementations of the present application, the device side may also include more accelerator cards, such as a larger number of accelerator cards or more types of accelerator cards.

[0080] When a device includes multiple accelerator cards, they can concurrently execute multiple operators. These operators include communication operators, and the accelerator cards can exchange data directly through these communication operators to concurrently execute multiple operators. When a device includes a single accelerator card, the card can independently execute multiple operators.

[0081] In order to make the technical solution of the present application clearer and easier to understand, the operator processing method of the present application is introduced below with reference to the accompanying drawings.

[0082] Referring to the flowchart of an operator processing method shown in FIG3 , the method can be executed by an operator processing device. The operator processing device can be a software device, such as a software device deployed on the host side, or a software device deployed on a computing device cluster, which is used to construct multiple operators that need to be batch scheduled into an operator execution queue for execution on the device side. It should be noted that when the operator processing device is a software device deployed on a computing device cluster, the computing device that executes the operator can call the interface of the software device to obtain the operator execution queue, and execute multiple operators based on the operator execution queue on the device side. The operator processing device can also be a hardware device, such as a host on the computing device side, or a computing device cluster. The method includes:

[0083] S302: The operator processing device obtains topological information of the computation graph.

[0084] A computation graph is a topological graph that describes a computation process. A computation graph typically includes the operators involved in the computation process, the input data of the operators, and the output data of the operators. For the computation graph as a whole, the topological information of the computation graph records the multiple operators, graph input data, graph output data, graph intermediate data, and the dependencies between the multiple operators included in the computation graph. Graph input data can be the input data of several operators among the multiple operators included in the computation graph, and graph output data can be the output data of several operators among the multiple operators included in the computation graph. Graph intermediate data can be the output data of some (or one) operators and the input data of other (or another) operators among the multiple operators included in the computation graph. The input data, output data of an operator, or graph input data, graph output data, and graph intermediate data can be tensors. A tensor can be an array with any integer dimension, as well as various properties that describe this array, including but not limited to dimension (shape) information, data type, device type, data format, memory address, etc.

[0085] The computation graph can be user-defined. Specifically, the operator processing device can provide a composition interface, which can be a graphical user interface (GUI), a command user interface (CUI), or an API, through which users can customize the computation graph. For ease of description, the composition interface is used as an example GUI.

[0086] FIG4 shows a schematic diagram of a composition interface. This composition interface provides multiple operators for constructing computational graphs. Users can drag and drop multiple operators that require batch scheduling to the composition area. Furthermore, users can view the input and output data of multiple operators. Based on the input and output data of multiple operators, the dependencies between the multiple operators can be determined. Users can then connect multiple operators based on these dependencies to achieve custom computational graph construction. Accordingly, the operator processing device can obtain the topological information of the computational graph.

[0087] S302: The operator processing device constructs an operator execution queue according to the topology information of the computation graph.

[0088] The operator execution queue is a queue used to execute operators. Each element in the operator execution queue corresponds to an operator in the computation graph and its input and output data. The operator input data is at least one of the graph input data or the graph intermediate data, and the operator output data is at least one of the graph intermediate data or the graph output data. The execution order of multiple operators in the operator execution queue is determined by the dependencies between the multiple operators.

[0089] Specifically, the operator processing device can sort multiple operators based on their dependencies and then sequentially add them to an operator execution queue. Each operator can correspond to an element in the operator execution queue. The elements in the operator execution queue also record the operator's input and output data.

[0090] In some possible implementations, the operator execution queue can be a first-in-first-out (FIFO) queue. Accordingly, the operator processing device can add previously executed operators to the FIFO queue first, thereby ensuring that the execution order of the operators satisfies the dependencies between the operators.

[0091] The operator processing device also supports partitioning graph input data, graph output data, and graph intermediate data. Accordingly, each element in the operator execution queue records an operator in the computation graph, the number of the operator's input data, and the number of the operator's output data, thereby recording the operators in the computation graph and their input and output data.

[0092] Furthermore, the operators in the operator execution queue can include at least one compiled operator. For example, the operators in the operator execution queue can be binary operators. Binary operators are computing units in binary coded form that can be executed on the device without compilation. When all operators in the operator execution queue are binary operators, the operator sequence can also be called a kernel sequence. This can solve the problem of heavy AI compilation process, and operator scheduling can be achieved through light compilation or even no compilation.

[0093] For ease of understanding, an example is provided below for explanation.

[0094] As shown in Figure 5, the computation graph includes the following binary operators: OP1 to OP5. The computation graph also records the graph input tensor (InTensor), graph output tensor (OutTensor), and graph intermediate tensor (InterTensor). The graph input tensor is the input tensor of OP1, OP2, and OP5, the graph output tensor is the output tensor of OP5, and the graph intermediate tensor includes the output tensors of OP1 and OP2 (the output tensor of OP1 is also the input tensor of OP3, and the output tensor of OP2 is also the input tensor of OP3 and OP4). The graph intermediate tensor also includes the output tensor of OP3 (the output tensor of OP3 is also the input tensor of OP4) and the output tensor of OP4 (the output tensor of OP4 is also the input tensor of OP5).

[0095] Among them, the graph input tensor, graph output tensor, and graph intermediate tensor can be partitioned and numbered. For example, the above-mentioned graph input tensor can be numbered 1 to 3, the graph output tensor can be numbered 4, and the graph intermediate tensor can be numbered 5 to 8. Based on the dependency relationship of the operators in the topology information, the operator processing device determines that OP1 and OP2 are executed before OP3, OP3 is executed before OP4, and OP4 is executed before OP5. Therefore, OP1 to OP5 can be added to the operator execution queue in sequence, and the tensor numbers of the operator's input tensor and output tensor are bound to the operator as FIFO queue node information. The tensor numbers of the operator's input tensor and output tensor can be used as tensor indexes during the graph execution process.

[0096] It should be noted that in the above example, OP2 can also be added to the operator execution queue before OP1. In other words, OP2 can also be executed before OP1. In addition, the operator execution queue can also be other types of queues, which are not limited in this embodiment. When operators are executed concurrently on multiple devices, the operators in the operator execution queue can include communication operators, which are used to exchange data between different devices (for example, different NPUs).

[0097] Based on the above description, it can be seen that the operator processing method of this application supports user-defined group graphs. The operator execution queue can be constructed according to the topological information of the assembled calculation graph, thereby realizing the construction of the execution graph in the form of an operator execution queue. This can be flexibly applied to all scenarios that require batch operator scheduling, meeting business needs.

[0098] Furthermore, this application also supports pre-planning the memory layout of the computational graph's internal computing resources based on input data and operator execution queues. Memory reuse is performed based on its lifecycle, improving memory utilization and significantly reducing overall memory consumption. Furthermore, this method supports applying for large blocks of memory at once, reducing the number of memory requests and improving overall operational efficiency. The resource planning process is described in detail below.

[0099] The execution of an operator usually requires input data (such as input tensors InTensors), output data (such as OutTensors), slicing parameters (tiling data), and scratch memory. Among them, the slicing parameters are parameters used to slice the input data. The slicing parameters can include the total length of the parameter (totalLength), the number of splits (such as the number of times a single core loop processes data, recorded as tileNum) and the calculation scalar (such as LeakyRelu scalar). The scratch memory is used to temporarily store temporary data generated by the execution of the operator. For a graph user, they usually focus on: graph input tensors, graph output tensors, and graph workspace (GraphWorkspace). Therefore, the graph intermediate tensors, tiling data, and scratch memory need to be put into the graph workspace and passed in from outside the graph.

[0100] Among them, the graph workspace refers to the workspace (workspace) where the graph is executed. The workspace can also be called a context environment or a session. The execution of the computational graph needs to be in the context of a session. The session provides an environment for operator execution and tensor evaluation. A session may have some resources, such as variables or queues. When the session is no longer needed, you can call sess.close() to close the session (or use the context manager to automatically close it) to release the resources. If there are multiple computational graphs to be executed, you can create different sessions to load each computational graph, and each computational graph can be loaded in multiple sessions for calculation. Sessions are independent of each other.

[0101] Based on this, the operator processing device can determine the size of the graph workspace according to the dimension information of the graph input data and the correspondence between the operator corresponding to each element in the operator execution queue and the operator's input data and output data, and apply for memory as the graph workspace according to the size of the graph workspace.

[0102] Among them, the operator processing device can perform chain deduction based on the shape information of the graph input data (such as the graph input tensor) and the corresponding relationship between the operator's input data and output data, thereby deducing the shape information of the graph intermediate tensor, and then obtaining the memory size for storing the graph intermediate tensor. In addition, the operator processing device can also deduce the amount of temporary data generated during the execution process, and then estimate the size of the temporary memory. The operator processing device can obtain the size of the entire graph workspace based on the memory size for storing the graph intermediate tensor, the size of the temporary memory, and the memory size for storing the sharding parameters. It should be noted that the graph workspace can also be divided into an operator workspace (OpWorkspace) and a graph intermediate data area. Among them, the operator workspace includes a sharding parameter area and a temporary memory. The sharding parameter area is used to store the sharding parameters, and the temporary memory is used to store temporary data generated during the execution of the operator.

[0103] Taking Figure 6 as an example, the operator processing device can chain-derive the shape information of the graph intermediate tensor based on the shape information of the graph input tensor (such as tensor ID 1, 2, 3), and then obtain the size of the memory required to store the graph intermediate tensor. In addition, the operator processing device can also deduce the size of the memory required for the sharding parameters of each operator and the size of the temporary memory for temporarily storing temporary data generated during the execution of each operator. The operator processing device can predict the size of the graph workspace based on the size of the memory required to store the graph intermediate tensor, the size of the memory required to store the sharding parameters, and the size of the temporary memory, and apply for a memory block of the corresponding size from the device side as the graph workspace. It should be noted that the operator processing device can also apply for a memory block slightly larger than the above size as the graph workspace.

[0104] The following describes the memory planning for sharding parameters, temporary memory, and intermediate graph data.

[0105] For an operator's tiling parameters (tiling data), they usually need to be loaded from the host side into the designated device memory space before the operator is executed. They are no longer used after the operator is executed. To reduce the number of load times, this application supports loading all the tiling parameters at once. Accordingly, the lifecycle of the tiling parameters can be equivalent to the lifecycle of the graph. For an operator's scratch memory, its lifecycle can be the period from the start to the end of the corresponding operator execution. Based on this, the scratch memory of multiple operators can be reused.

[0106] In specific implementation, the operator processing device can determine the offset address (denoted as offset) of the sharding parameters of multiple operators in the graph workspace and the size (size) of the sharding parameters of multiple operators, and load the sharding parameters of multiple operators into the graph workspace according to the offset address of the sharding parameters of multiple operators in the graph workspace and the size of the sharding parameters of the multiple operators. With respect to the temporary memory, the operator processing device can determine the offset address and size of the temporary memory in the graph workspace according to the amount of temporary data generated by the execution of multiple operators. The size of the temporary memory can be the maximum amount of temporary data. When executing the operator, by passing the offset address and size of the temporary memory in the graph workspace into the graph workspace, the temporary data generated by executing the operator can be stored in the corresponding position or area. It should be noted that the temporary memories of multiple operators can have the same offset address, so that memory reuse can be achieved and memory overhead can be reduced.

[0107] Take Figure 7 as an example. The operator's tilling data occupies less memory space on the device side. In order to reduce the number of times the op tilling data is loaded, the tilling data of all operators in the operator execution queue can be loaded at one time. Among them, the tilling data of all operators can be placed at the starting address of the Graph Workspace. In other words, the offset of the tilling data can be equal to the offset of the Graph Workspace. The size of the tilling data area can be equal to the size of the tilling data of all operators in the operator execution queue. The scratch memory of each operator is no longer used after the execution of the operator is completed, so when the subsequent operator is executed, the scratch memory of the previous operator can be reused. Therefore, the offset of the scratch memory of all operators in the Graph Workspace is the same, but their sizes are different. Only the largest size needs to be taken.

[0108] The operator processing device may scan the operator execution queue and write the offset and size of each operator's tiling data and scratch memory into the operator's running information (denoted as runInfo) for use in subsequent execution of the operator.

[0109] For a graph's intermediate data (such as a tensor in a graph), its lifecycle ends immediately after all operators that rely on it as input complete their execution. In typical application scenarios, the dimensionality of graph intermediate data is relatively large. Without considering memory transfer, reusing device-side memory as much as possible can improve memory utilization.

[0110] Specifically, the operator processing device can obtain the dimension information of the graph intermediate data and the dependency information of the graph intermediate data. The dependency information is used to indicate the operator that depends on the graph intermediate data. Then the operator processing device can determine the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data based on the dimension information of the graph intermediate data and the dependency information of the graph intermediate data.

[0111] Among them, the operator processing device can determine the life cycle of each graph intermediate data based on the dependency information of the graph intermediate data, and then the operator processing device can determine the arrangement of the graph intermediate data in the memory based on the dimension information of the graph intermediate data and the life cycle of each graph intermediate data, thereby obtaining the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data. For example, the graph intermediate data includes first graph intermediate data and second graph intermediate data. When the operator that depends on the first graph intermediate data is executed before the operator that outputs the second graph intermediate data, the offset address of the first graph intermediate data is less than or equal to the offset address of the second graph intermediate data, that is, the second graph intermediate data can reuse the memory space of the first graph intermediate data.

[0112] Take Figure 8 as an example. After OP1 is executed, it outputs the intermediate tensor of the graph (i.e., tensor 5). Tensor 5 is dependent on the subsequent OP3, so the memory space of tensor 5 is not released. After OP2 is executed, it outputs the intermediate tensor of the graph (i.e., tensor 6). Tensor 6 is dependent on the subsequent OP3, so the memory space of tensor 6 is not released. At this time, the memory space of tensors 5 and 6 is not released. OP3 executes dependent tensors 5 and 6, and after execution, it outputs the intermediate tensor of the graph (i.e., tensor 7). At this time, tensor 5 is not dependent on subsequent operators, and the memory space of tensor 5 can be released. OP3 executes dependent tensors 6 and 7, and after execution, it outputs the intermediate tensor of the graph (i.e., tensor 8). The memory space used by tensor 8 is smaller than the memory space used by tensor 5, and tensor 8 can reuse the memory space of tensor 5. Based on this, the operator processing device can configure the offset of tensor 8 to be equal to the offset of tensor 5. In other possible implementations of the present embodiment, the operator processing device may also configure the offset of tensor 8 to be larger (e.g., slightly larger) than the offset of tensor 5, i.e., the offset of tensor 5 is smaller than the offset of tensor 8, to achieve memory reuse of tensors 5 and 8. After OP5 is executed, the memory space of all intermediate tensors in the graph can be released.

[0113] When the memory used by tensor 8 is larger than the memory used by tensor 5, for example, the memory used by tensor 8 is slightly larger than the memory used by tensor 5, it may result in the inability to reuse the memory space of tensor 5, and the memory space can only be expanded to the right. In order to avoid the above situation, the operator processing device can merge adjacent free blocks (that is, the memory blocks where the tensors are released), and try to select the smallest size block that meets the requirements each time, such as selecting the smallest size block that meets the requirements for reuse through the BestFit algorithm, so as to retain more and larger blocks as much as possible and improve the memory reuse rate. In addition, the operator processing device can also plan in advance based on global information, such as offsetting the starting addresses of tensors 6 and 7 to the right in advance, so that after the subsequent release of tensor 5, tensor 8 can be placed after the adjacent fragments are merged.

[0114] After completing the above memory planning, the operator execution queue can be executed (this process is also called graph execution). As shown in Figure 9, the execution of operators in the operator execution queue requires the preparation of the runInfo information of all operators first. The main process is to update the data pointer of the tensor in the middle of the graph and the workspace pointer of the operator. Since the operator processing device has stored the offset of each resource block during memory planning, based on this, when the operator processing device updates the pointer (data pointer, workspace pointer), it only needs to add the offset to the base address of the external workspace memory.

[0115] Further, referring to Figure 10, during the update process, the operator processing device can determine whether the input tensor and output tensor of each operator belong to the intermediate tensor of the graph based on the area of ​​the tensor in the tensor table of the graph. Since the data pointers of the graph input and output tensors do not need to be updated and are real addresses themselves, the above-mentioned update operation can be omitted. The operator processing device can perform the above-mentioned update operation when it is determined that the input tensor and output tensor of the operator belong to the intermediate tensor of the graph.

[0116] In some possible implementations, the operator processing device can support parallel execution of computational graphs. Specifically, the operator processing device can asynchronously execute graph computations, and while the device side (such as the NPU on the device side) is still computing, it can prepare computing resources for the next computational graph in advance. This will reduce the waiting time on the device side and increase operator throughput. Referring to FIG11 , in different graphs, or scenarios with different shapes of the same graph, the operator processing device can asynchronously execute resource planning for computational graph 2 (graph 2) while transferring tiling data of computational graph 1 (graph 1) from the host side to the device side, such as calculating the offset and size of the resource block.

[0117] Furthermore, for the same graph and the same shape scenario, the present application also supports the reuse of graph workspaces. Specifically, the calculation graph includes a first calculation graph (such as graph 1) and a second calculation graph (such as graph 2), the operators included in the first calculation graph are the same as the operators included in the second calculation graph, the dependencies of the operators in the first calculation graph are the same as the dependencies of the operators in the second calculation graph, and the dimension information of the data in the first calculation graph is the same as the dimension information of the data in the second calculation graph. When executing the first calculation graph, the operator processing device can adjust the address of the graph input data to the address of the graph input data of the second calculation graph. In this way, after the first calculation graph is executed, the operator processing device can directly reuse the graph workspace of the first calculation graph to execute the second calculation graph. This method improves the operator throughput on the device side and improves the overall running speed by asynchronously executing different calculation graphs.

[0118] Based on the aforementioned operator processing method, the present application provides an operator processing device, which will be introduced below from the perspective of functional modularization.

[0119] Referring to FIG12 , a schematic diagram of the structure of an operator processing device, the device 1200 includes:

[0120] An acquisition module 1202 is configured to acquire topology information of a computation graph, wherein the topology information records multiple operators, graph input data, graph output data, graph intermediate data, and dependencies between the multiple operators included in the computation graph;

[0121] Construction module 1204 is used to construct an operator execution queue based on the topological information of the computation graph, where each element in the operator execution queue corresponds to an operator in the computation graph and the input data and output data of the operator, the input data of the operator is at least one of the graph input data or the graph intermediate data, and the output data of the operator is at least one of the graph intermediate data or the graph output data. The execution order of multiple operators in the operator execution queue is determined according to the dependency relationship between the multiple operators.

[0122] The acquisition module 1202 is used to implement the relevant contents of S302 in the aforementioned method embodiment, and the construction module 1204 is used to implement the relevant contents of S304 in the aforementioned method embodiment.

[0123] The acquisition module 1202 and the construction module 1204 can be implemented by software or hardware.

[0124] When implemented by software, the acquisition module 1202 and the construction module 1204 can be applications running on a computing device. For example, the construction module 1204 can be a computing engine running on a computing device. The application can be provided to users in the form of a virtualization service. Virtualization services may include virtual machine (VM) services, bare metal server (BMS) services, and container services. Among them, the VM service can be a service that uses virtualization technology to virtualize a virtual machine (VM) resource pool on multiple physical hosts (such as computing devices) to provide users with VMs for use on demand. The BMS service is a service that virtualizes a BMS resource pool on multiple physical hosts to provide users with BMS for use on demand. The container service is a service that virtualizes a container resource pool on multiple physical hosts to provide users with containers for use on demand. VM is a simulated virtual computer, that is, a logical computer. BMS is a high-performance computing service that can be elastically scalable. Its computing performance is no different from that of a traditional physical machine and has the characteristics of secure physical isolation. Containers are a kernel virtualization technology that provides lightweight virtualization to isolate user spaces, processes, and resources. It should be understood that the VM service, BMS service, and container service mentioned above are merely examples. In actual applications, virtualization services can also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0125] When implemented through hardware, the acquisition module 1202 and the construction module 1204 may include at least one processor, such as a CPU. Alternatively, the acquisition module 1202 and the construction module 1204 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0126] In some possible implementations, the construction module 1204 is further configured to:

[0127] Partition numbering of the graph input data, the graph output data, and the graph intermediate data;

[0128] Each element in the operator execution queue records an operator in the computation graph, the number of the operator's input data, and the number of the operator's output data.

[0129] In some possible implementations, the apparatus 1200 further includes:

[0130] The memory planning module 1206 is used to determine the size of the graph workspace based on the dimensional information of the graph input data and the correspondence between the operator corresponding to each element in the operator execution queue and the input data and output data of the operator, and apply for memory as the graph workspace based on the size of the graph workspace.

[0131] Similar to the construction module 1204 , the memory planning module 1206 may be implemented by software or hardware.

[0132] When implemented in software, the memory planning module 1206 may be an application running on a computing device. For example, the memory planning module 1206 may be a computing engine running on the computing device. The application may be provided to users as a virtualized service. For example, the application may be provided to users via a VM service, a BMS service, or a container service.

[0133] When implemented by hardware, the memory planning module 1206 may include at least one processor, such as a CPU, etc. Alternatively, the memory planning module 1206 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD) (such as a CPLD or FPGA).

[0134] In some possible implementations, the memory planning module 1206 is further configured to:

[0135] Obtain the dimension information of the graph intermediate data and the dependency information of the graph intermediate data, where the dependency information is used to indicate the operator that depends on the graph intermediate data; determine the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data based on the dimension information of the graph intermediate data and the dependency information of the graph intermediate data.

[0136] In some possible implementations, the graph intermediate data includes first graph intermediate data and second graph intermediate data. When an operator that depends on the first graph intermediate data is executed before an operator that outputs the second graph intermediate data, the offset address of the first graph intermediate data is less than or equal to the offset address of the second graph intermediate data.

[0137] In some possible implementations, the memory planning module 1206 is further configured to:

[0138] Determining the offset addresses of the sharding parameters of the multiple operators in the graph workspace and the sizes of the sharding parameters of the multiple operators;

[0139] The sharding parameters of the multiple operators are loaded into the graph workspace according to the offset addresses of the sharding parameters of the multiple operators in the graph workspace and the sizes of the sharding parameters of the multiple operators.

[0140] In some possible implementations, the graph workspace is further provided with a temporary memory;

[0141] The memory planning module 1206 is further configured to:

[0142] The offset address and size of the graph workspace in the temporary memory are determined according to the amount of temporary data generated by executing the multiple operators.

[0143] In some possible implementations, the temporary memories of the multiple operators have the same offset address.

[0144] In some possible implementations, the computation graph includes a first computation graph and a second computation graph, the operators included in the first computation graph are the same as the operators included in the second computation graph, the dependencies between the operators in the first computation graph are the same as the dependencies between the operators in the second computation graph, and the dimension information of the data in the first computation graph is the same as the dimension information of the data in the second computation graph;

[0145] The apparatus 1200 further includes:

[0146] The adjustment module 1208 is configured to adjust the address of graph input data to the address of graph input data of the second computation graph when executing the first computation graph.

[0147] Similar to the above-mentioned acquisition module 1202 and construction module 1204 , the adjustment module 1208 can be implemented by software or hardware.

[0148] When implemented by software, the adjustment module 1208 may be an application running on a computing device. The application may be provided to the user as a virtualized service. For example, the application may be provided to the user via a VM service, a BMS service, or a container service.

[0149] When implemented by hardware, the adjustment module 1208 may include at least one processor, such as a CPU, etc. Alternatively, the adjustment module 1208 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD) (such as a CPLD or FPGA).

[0150] FIG12 introduces the operator processing device from the perspective of functional modularization. The following introduces the operator processing device from the perspective of hardware instantiation.

[0151] The operator processing device may include at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the operator processing device performs the operator processing method of the aforementioned embodiment.

[0152] The operator processing device may be a computing device, such as the server shown in FIG2 . Furthermore, the operator processing device may be a computing device cluster formed by multiple computing devices, such as a computing device cluster formed by multiple servers shown in FIG2 . The memories of the multiple computing devices in the computing device cluster may store the same instructions for the operator processing device 1200 to execute the operator processing method.

[0153] In some possible implementations, multiple computing devices in the computing device cluster may also be used to execute some of the instructions of the operator processing apparatus 1200 for executing the operator processing method. In other words, the combination of multiple computing devices may jointly execute the instructions of the operator processing apparatus 1200 for executing the operator processing method.

[0154] It should be noted that the memories in different computing devices in the computing device cluster may store different instructions for executing part of the functions of the operator processing apparatus 1200 .

[0155] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned application to the operator processing device 1200 for executing the operator processing method.

[0156] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored on any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the operator processing method described above.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An operator processing method, characterized in that: The method comprises: Obtaining topological information of a computational graph, wherein the topological information records a plurality of operators, graph input data, graph output data, graph intermediate data, and dependency relationships of the plurality of operators included in the computational graph; According to the topological information of the computational graph, an operator execution queue is constructed, wherein each element in the operator execution queue corresponds to an operator in the computational graph and the input data and output data of the operator, the input data of the operator is at least one of the graph input data or the graph intermediate data, the output data of the operator is at least one of the graph intermediate data or the graph output data, and the execution order of multiple operators in the operator execution queue is determined according to the dependency relationship between the multiple operators.

2. The method according to claim 1, characterized in that The method further comprises: Partition numbering the graph input data, the graph output data, and the graph intermediate data; Each element in the operator execution queue records an operator in the computation graph and the number of the input data of the operator and the number of the output data of the operator.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Determine the size of the graph workspace according to the dimension information of the graph input data and the correspondence between the operator corresponding to each element in the operator execution queue and the input data and output data of the operator; Apply for memory as the graph workspace according to the size of the graph workspace.

4. The method according to claim 3, characterized in that: The method further comprises: Acquire dimension information of the graph intermediate data and dependency information of the graph intermediate data, where the dependency information is used to indicate operators that depend on the graph intermediate data; According to the dimension information of the graph intermediate data and the dependency information of the graph intermediate data, the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data are determined.

5. The method according to claim 4, characterized in that The graph intermediate data includes first graph intermediate data and second graph intermediate data. When an operator dependent on the first graph intermediate data is executed before an operator outputting the second graph intermediate data, the offset address of the first graph intermediate data is less than or equal to the offset address of the second graph intermediate data.

6. The method according to any one of claims 3 to 5, characterized in that: The method further comprises: Determine the offset addresses of the slicing parameters of the multiple operators in the graph workspace and the sizes of the slicing parameters of the multiple operators; The sharding parameters of the multiple operators are loaded into the graph workspace according to the offset addresses of the sharding parameters of the multiple operators in the graph workspace and the sizes of the sharding parameters of the multiple operators.

7. The method according to any one of claims 3 to 6, characterized in that: The graph workspace is also provided with a temporary memory; The method further comprises: The offset address and size of the graph workspace in the temporary storage are determined according to the amount of temporary data generated by the execution of the multiple operators.

8. The method according to claim 7, characterized in that The temporary storage memories of the multiple operators have the same offset address.

9. The method according to any one of claims 1 to 8, characterized in that: The computation graph includes a first computation graph and a second computation graph, the operators included in the first computation graph are the same as the operators included in the second computation graph, the dependencies of the operators in the first computation graph are the same as the dependencies of the operators in the second computation graph, and the dimension information of the data in the first computation graph is the same as the dimension information of the data in the second computation graph; The method further comprises: When executing the first computation graph, the address of the graph input data is adjusted to the address of the graph input data of the second computation graph.

10. An operator processing device, characterized in that: The device comprises: An acquisition module is used to acquire topological information of a computational graph, wherein the topological information records a plurality of operators, graph input data, graph output data, graph intermediate data and dependency relationships of the plurality of operators included in the computational graph; A construction module is used to construct an operator execution queue according to the topological information of the computation graph, wherein each element in the operator execution queue corresponds to an operator in the computation graph and the input data and output data of the operator, wherein the input data of the operator is at least one of the graph input data or the graph intermediate data, and the output data of the operator is at least one of the graph intermediate data or the graph output data. The execution order of the multiple operators in the operator execution queue is determined according to the dependency relationship of the multiple operators.

11. The device according to claim 10, characterized in that The construction module is also used for: Partition numbering of the graph input data, the graph output data, and the graph intermediate data; Each element in the operator execution queue records an operator in the computation graph and the number of the input data of the operator and the number of the output data of the operator.

12. The device according to claim 10 or 11, characterized in that The device also includes: A memory planning module is used to determine the size of the graph workspace according to the dimensional information of the graph input data and the correspondence between the operator corresponding to each element in the operator execution queue and the input data and output data of the operator, and apply for memory as the graph workspace according to the size of the graph workspace.

13. The device according to claim 12, characterized in that The memory planning module is also used for: Obtain dimension information of the graph intermediate data and dependency information of the graph intermediate data, wherein the dependency information is used to indicate operators that depend on the graph intermediate data, and determine the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data according to the dimension information of the graph intermediate data and the dependency information of the graph intermediate data.

14. The device according to claim 13, characterized in that The graph intermediate data includes first graph intermediate data and second graph intermediate data. When an operator dependent on the first graph intermediate data is executed before an operator outputting the second graph intermediate data, the offset address of the first graph intermediate data is less than or equal to the offset address of the second graph intermediate data.

15. The device according to any one of claims 12 to 14, characterized in that The memory planning module is also used for: Determine the offset addresses of the slicing parameters of the multiple operators in the graph workspace and the sizes of the slicing parameters of the multiple operators; The sharding parameters of the multiple operators are loaded into the graph workspace according to the offset addresses of the sharding parameters of the multiple operators in the graph workspace and the sizes of the sharding parameters of the multiple operators.

16. The device according to any one of claims 12 to 15, characterized in that The graph workspace is also provided with a temporary memory; The memory planning module is also used for: The offset address and size of the graph workspace in the temporary storage are determined according to the amount of temporary data generated by the execution of the multiple operators.

17. The device according to claim 16, characterized in that The temporary storage memories of the multiple operators have the same offset address.

18. The device according to any one of claims 10 to 17, characterized in that The computation graph includes a first computation graph and a second computation graph, the operators included in the first computation graph are the same as the operators included in the second computation graph, the dependencies of the operators in the first computation graph are the same as the dependencies of the operators in the second computation graph, and the dimension information of the data in the first computation graph is the same as the dimension information of the data in the second computation graph; The device also includes: An adjustment module is used to adjust the address of graph input data to the address of graph input data of the second calculation graph when executing the first calculation graph.

19. An operator processing device, characterized in that: The device includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the operator processing device executes the operator processing method as described in any one of claims 1 to 9.

20. A computer program product, characterized in that Comprising computer-readable instructions; the computer-readable instructions are used to implement the operator processing method according to any one of claims 1 to 9.

Images