Operator processing method and related device

Abstract

The invention provides an operator processing method, which comprises the following steps of: acquiring topological information of a computational graph, recording a plurality of operators, graph input data, graph output data, graph intermediate data and a dependency relationship of the plurality of operators included in the computational graph by the topological information, and constructing an operator execution queue according to the topological information of the computational graph. Each element in the operator execution queue corresponds to one operator and input data and output data of the operator, the input data of the operator is at least one of graph input data or graph intermediate data, and the output data of the operator is at least one of graph intermediate data or graph output data; the execution sequence of the plurality of operators is determined according to the dependency relationship of the plurality of operators. According to the method, the user is supported to customize the group graph according to the operator, and the operator execution queue is constructed based on the topological information of the calculation graph, so that the execution graph is constructed in the form of the operator execution queue, the method can be flexibly applied to all scenes needing batch operator scheduling, and the service requirements are met.

Description

[0001] This application claims priority to a Chinese patent application filed with the State Intellectual Property Office of China on June 29, 2023, with application number 202310792403.3 and invention name “A method for executing a computational graph”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to an operator processing method, device, computer-readable storage medium, and computer program product. Background Art

[0003] An operator is a computing unit, for example, a binary operator is usually a computing unit in binary code form. An operator usually includes host logic and device logic. The host includes a central processing unit (CPU) and host memory, and the device includes an external processing unit and device memory. The external processing unit can be a neural processing unit (NPU) or a graphics processing unit (GPU). Different types of processors can implement different types of instruction set architectures.

[0004] The execution of operators usually involves the interaction between the host and the device. Considering that the input data of the operator is already in the device memory, the main time consumption includes: the host derives the output shape based on the input dimension (shape), applies for the memory on the device side, calculates the sharding parameter Tilling Data of the operator, moves the Tilling Data to the device side, sends the computing task to the device side, and the device side executes the computing task.

[0005] In many business scenarios, such as training neural networks or performing reasoning based on trained neural networks, multiple operators are usually executed. When executing multiple operators, the computational graph including multiple operators can be compiled into an intermediate representation (IR) through compilation methods, such as Tensor Virtual Machine (TVM) or Accelerated Linear Algebra (XLA), where IR is used to represent the description of the execution process, and then an execution graph is generated to implement batch operator calls through the execution graph.

[0006] However, the above method requires a complex compilation optimization process and offline generation of execution graphs. It is a heavyweight operator scheduling technology and is not applicable to some scenarios that require lightweight batch operator calls, such as implementing the acceleration library application programming interface (API). Summary of the invention

[0007] The present application provides an operator processing method, which constructs an operator execution queue based on the topological information of a computational graph, thereby realizing the construction of an execution graph in the form of an operator execution queue, which can be flexibly applied to all scenarios requiring batch operator scheduling, meeting business needs. The present application also provides an operator processing device, a computer-readable storage medium, and a computer program product corresponding to the above operator processing method.

[0008] In a first aspect, the present application provides an operator processing method. The method can be performed by an operator processing device. The operator processing device can be a software device, which can be an independent software package, or a cloud service, such as a cloud service provided through an interface (such as a composition interface), which supports users to customize group diagrams based on operators and represents the topology diagram in the form of a queue. The above-mentioned software device can be deployed in a computing device cluster (for example, a computing device that executes the operator or a third-party computing device cluster), and the computing device cluster executes the program code of the software device to execute the operator processing method of the present application. The operator processing device can also be a hardware device, such as a computing device or a computing device cluster with an operator processing function. When the operator processing device is running, the operator processing method of the present application is executed.

[0009] Specifically, the operator processing device obtains the topological information of the computation graph, and the topological information records the multiple operators, graph input data, graph output data, graph intermediate data and the dependency relationship of the multiple operators included in the computation graph, and then constructs the operator execution queue according to the topological information of the computation graph. Each element in the operator execution queue corresponds to an operator in the computation graph and the input data and output data of the operator, the input data of the operator is at least one of the graph input data or the graph intermediate data, the output data of the operator is at least one of the graph intermediate data or the graph output data, and the execution order of the multiple operators in the operator execution queue is determined according to the dependency relationship of the multiple operators.

[0010] This method supports users to customize group graphs according to operators and construct operator execution queues based on the topological information of the computational graph, thereby building execution graphs in the form of operator execution queues. It can be flexibly applied to all scenarios that require batch operator scheduling, including lightweight batch operator scheduling scenarios such as acceleration library construction, meeting business needs.

[0011] In some possible implementations, the operator processing device may partition the graph input data, graph output data, and graph intermediate data. Each element in the operator execution queue records an operator in the computation graph and the number of the operator's input data and the number of the operator's output data. In this way, the tensor number can be bound to the operator. The tensor number is the node information of the operator execution queue and can be used as a tensor index in the graph execution process when the operator is subsequently executed.

[0012] In some possible implementations, the operator processing device can determine the size of the graph workspace (also referred to as the context environment in some cases) based on the dimensional information of the graph input data and the correspondence between the operator corresponding to each element in the operator execution queue and the input data and output data of the operator. Then the operator processing device can apply for memory as the graph workspace based on the size of the graph workspace. Among them, the operator processing device can apply for device memory of the corresponding size as the graph workspace, or the operator processing device can apply for device memory slightly larger than the above size as the graph workspace to prevent insufficient graph workspace.

[0013] This method deduce the size of the graph workspace based on the input dimension and the correspondence between operators and input and output data in the operator execution queue before executing the operator sequence, and applies for corresponding memory as the graph workspace based on the size. There is no need to apply for memory multiple times, which reduces the number of memory applications and improves the overall operating efficiency.

[0014] In some possible implementations, for a graph intermediate data (such as a graph intermediate tensor), its life cycle ends immediately after all operators that rely on the graph intermediate data as input finish running. The dimensional information of the graph intermediate data is relatively large in typical application scenarios. Without considering memory transfer, reusing the device-side memory space as much as possible can improve memory utilization. Based on this, the operator processing device can obtain the dimensional information of the graph intermediate data and the dependency information of the graph intermediate data, and the dependency information is used to indicate the operator that depends on the graph intermediate data, and determine the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data according to the dimensional information of the graph intermediate data and the dependency information of the graph intermediate data.

[0015] The method determines the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data by combining the dimension information of the graph intermediate data and the dependency information of the graph intermediate data, and can realize memory reuse for different graph intermediate data, thereby improving memory utilization.

[0016] In some possible implementations, the graph intermediate data includes first graph intermediate data and second graph intermediate data. When an operator that depends on the first graph intermediate data is executed before an operator that outputs the second graph intermediate data, an offset address of the first graph intermediate data is less than or equal to an offset address of the second graph intermediate data.

[0017] Since the life cycles of the intermediate data of the first graph and the intermediate data of the second graph do not overlap, the intermediate data of the second graph can reuse the memory space of the intermediate data of the first graph, thereby improving memory utilization.

[0018] In some possible implementations, when the dimensional information of the intermediate data of the second figure is less than or equal to the dimensional information of the intermediate data of the first figure, the intermediate data of the second figure can directly reuse the memory space of the intermediate data of the first figure. When the dimensional information of the intermediate data of the second figure is greater than the dimensional information of the intermediate data of the first figure, the offset address of the intermediate data of the third figure after the intermediate data of the first figure can be adjusted, for example, the offset address of the intermediate data of the third figure can be increased to reserve a certain amount of additional storage space for storing the intermediate data of the second figure that is larger than the intermediate data of the first figure, thereby reusing the memory space of the intermediate data of the first figure and improving memory utilization.

[0019] In some possible implementations, for the slicing parameters (tiling data) of an operator, it is usually necessary to load the slicing parameters from the host side to the specified device memory space before the operator is executed. The slicing parameters are no longer used after the operator is executed. In order to reduce the number of loading times, this application supports loading all at once. Specifically, the operator processing device can also determine the offset addresses of the slicing parameters of multiple operators in the graph workspace and the sizes of the slicing parameters of multiple operators, and load the slicing parameters of the multiple operators into the graph workspace according to the offset addresses of the slicing parameters of multiple operators in the graph workspace and the sizes of the slicing parameters of multiple operators. In this way, the overhead of loading slicing parameters can be reduced and resource utilization (for example, communication resource utilization) can be improved.

[0020] In some possible implementations, the graph workspace is further provided with a temporary memory. The operator processing device may determine the offset address and size of the temporary memory in the graph workspace according to the amount of temporary data generated by the execution of the multiple operators. For example, the operator processing device may determine the maximum amount of temporary data generated by the execution of multiple operators as the size of the temporary memory. In this way, the reuse of the temporary memory can be achieved.

[0021] In some possible implementations, the life cycle of an operator's temporary memory (or temporary data) is from the start of the operator's execution to the end, based on which the temporary memory of different operators can be reused. Specifically, the temporary memory of multiple operators can be set to have the same offset address. In this way, the temporary memory of different operators can reuse memory space and improve memory utilization.

[0022] In some possible implementations, the operator processing device can asynchronously execute different computation graphs. Specifically, the operator processing device can prepare the computing resources of the next computation graph in advance while the device side (such as the NPU on the device side) is still computing, which can reduce the waiting time on the device side and increase the operator throughput.

[0023] In some possible implementations, the computation graph includes a first computation graph and a second computation graph, the operators included in the first computation graph are the same as the operators included in the second computation graph, the dependencies of the operators in the first computation graph are the same as the dependencies of the operators in the second computation graph, and the dimension information of the data in the first computation graph is the same as the dimension information of the data in the second computation graph. Accordingly, when executing the first computation graph, the operator processing device may adjust the address of the graph input data to the address of the graph input data of the second computation graph.

[0024] In this way, after the first calculation graph is executed, the operator processing device can directly reuse the graph workspace of the first calculation graph to execute the second calculation graph. This method improves the operator throughput on the device side and improves the overall operation speed by asynchronously executing different calculation graphs.

[0025] In some possible implementations, at least one of the multiple operators is a compiled operator, such as a binary operator. A binary operator is a computing unit in binary code form that can be executed on the device side without compilation. By using at least one compiled operator to form a computation graph, the compilation time can be reduced and lightweight compilation can be achieved. When the computation graph uses compiled operators, even no compilation time can be achieved, so that the operator scheduling can be lightweight.

[0026] In a second aspect, the present application provides an operator processing device, characterized in that the device comprises:

[0027] An acquisition module is used to acquire topological information of a computational graph, wherein the topological information records a plurality of operators, graph input data, graph output data, graph intermediate data and dependency relationships of the plurality of operators included in the computational graph;

[0028] A construction module is used to construct an operator execution queue according to the topological information of the computational graph, wherein each element in the operator execution queue corresponds to an operator in the computational graph and the input data and output data of the operator, the input data of the operator is at least one of the graph input data or the graph intermediate data, the output data of the operator is at least one of the graph intermediate data or the graph output data, and the execution order of multiple operators in the operator execution queue is determined according to the dependency relationship between the multiple operators.

[0029] In some possible implementations, the construction module is further used to:

[0030] Partition numbering the graph input data, the graph output data, and the graph intermediate data;

[0031] Each element in the operator execution queue records an operator in the computation graph and the number of the input data of the operator and the number of the output data of the operator.

[0032] In some possible implementations, the device further includes:

[0033] A memory planning module is used to determine the size of the graph workspace according to the dimensional information of the graph input data and the correspondence between the operator corresponding to each element in the operator execution queue and the input data and output data of the operator, and apply for memory as the graph workspace according to the size of the graph workspace.

[0034] In some possible implementations, the memory planning module is further used to:

[0035] Obtain dimension information of the graph intermediate data and dependency information of the graph intermediate data, wherein the dependency information is used to indicate operators that depend on the graph intermediate data, and determine the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data according to the dimension information of the graph intermediate data and the dependency information of the graph intermediate data.

[0036] In some possible implementations, the graph intermediate data includes first graph intermediate data and second graph intermediate data. When an operator that depends on the first graph intermediate data is executed before an operator that outputs the second graph intermediate data, the offset address of the first graph intermediate data is less than or equal to the offset address of the second graph intermediate data.

[0037] In some possible implementations, the memory planning module is further used to:

[0038] Determine the offset addresses of the slicing parameters of the multiple operators in the graph workspace and the sizes of the slicing parameters of the multiple operators;

[0039] The sharding parameters of the multiple operators are loaded into the graph workspace according to the offset addresses of the sharding parameters of the multiple operators in the graph workspace and the sizes of the sharding parameters of the multiple operators.

[0040] In some possible implementations, the graph workspace is further provided with a temporary memory;

[0041] The memory planning module is also used for:

[0042] The offset address and size of the graph workspace in the temporary storage are determined according to the amount of temporary data generated by the execution of the multiple operators.

[0043] In some possible implementations, the temporary memories of the multiple operators have the same offset address.

[0044] In some possible implementations, the computation graph includes a first computation graph and a second computation graph, the operators included in the first computation graph are the same as the operators included in the second computation graph, the dependencies of the operators in the first computation graph are the same as the dependencies of the operators in the second computation graph, and the dimension information of the data in the first computation graph is the same as the dimension information of the data in the second computation graph;

[0045] The device also includes:

[0046] An adjustment module is used to adjust the address of graph input data to the address of graph input data of the second calculation graph when executing the first calculation graph.

[0047] In a third aspect, the present application provides an operator processing device. The operator processing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is used to execute instructions stored in the at least one memory, so that the operator processing device performs the operator processing method described in the first aspect or any implementation of the first aspect.

[0048] In a fourth aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and the instructions instruct a computing device or a computing device cluster to execute the operator processing method described in the above-mentioned first aspect or any implementation of the first aspect.

[0049] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device or a computing device cluster, enables the computing device or the computing device cluster to execute the operator processing method described in the first aspect or any one of the implementations of the first aspect.

[0050] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0052] Figure 1 A schematic diagram of a process flow of operator scheduling provided in an embodiment of the present application;

[0053] Figure 2 A hardware structure diagram of a server provided in an embodiment of the present application;

[0054] Figure 3 A flowchart of an operator processing method provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of a composition interface provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of constructing an operator execution queue based on a computational graph provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of an operator processing device provided in an embodiment of the present application applying for memory as a graph workspace;

[0058] Figure 7 A schematic diagram of a planning slicing parameter and an offset address and size stored in a graph workspace in a temporary memory provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of the offset address and size of intermediate data of a planning diagram in a diagram workspace provided in an embodiment of the present application;

[0060] Fig. 9 A schematic diagram of updating a data pointer of graph intermediate data and a graph workspace pointer to execute a computation graph provided in an embodiment of the present application;

[0061] Fig.10 A schematic diagram of identifying graph intermediate data and updating a data pointer of the graph intermediate data provided in an embodiment of the present application;

[0062] Fig.11 A schematic diagram of a process for executing different computation graphs in parallel provided in an embodiment of the present application;

[0063] Fig.12 A schematic diagram of the structure of an operator processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0065] First, some technical terms involved in the embodiments of the present application are introduced.

[0066] Operator (OP) refers to the operation performed on data. Depending on the type of operation, operators can be divided into computing operators (such as convolution operator Conv2D, maximum pooling operator MaxPool), communication operators (such as collection operator AllGather, broadcast operator Broadcast). Among them, operators (such as heterogeneous operators) can include host logic and device logic. The host includes CPU and host memory, and the device includes external processing unit (such as NPU or GPU) and device memory. The host logic may include logic for generating an execution graph (also called a run graph). The execution graph may be a sequence of operators (kernels) that can be executed directly on the device, and the device logic may include logic for running the execution graph.

[0067] Operator scheduling refers to the process of controlling operator execution, including data preparation, task delivery, and obtaining calculation results. Operator scheduling designs the interaction between the host side and the device side. Figure 1 As shown in the figure, considering that the input data of the operator is already in the device memory, the main scheduling time includes the host side deducing the output dimension information (also called output Shape) based on the input dimension (Shape) information, applying for memory on the device side, calculating the sharding parameter Tilling Data of the operator, moving the Tilling Data to the device, sending the computing task to the device, and executing the computing task on the device side.

[0068] Batch operator scheduling is required in many business scenarios. For example, in the scenario of training neural networks, or in the scenario of reasoning based on trained neural networks, batch scheduling of multiple operators is usually required. The operator scheduling method based on artificial intelligence (AI) compilation supports batch operator scheduling. Specifically, the computational graph including multiple operators is compiled into an intermediate representation (IR) through the tensor virtual machine (TVM) or accelerated linear algebra (XLA), and then an execution graph can be generated based on the IR, and batch operator calls are implemented through the execution graph.

[0069] However, the above method requires a complex compilation optimization process, among which the graph fusion process is complicated and cumbersome, and requires offline generation of the execution graph. It is a heavyweight operator scheduling technology and is not applicable in some scenarios that require lightweight batch operator calls, such as implementing the acceleration library application programming interface (API) scenario.

[0070] In view of this, the present application provides an operator processing method. The method can be performed by an operator processing device. The operator processing device can be a software device, which can be an independent software package, or a cloud service, such as a cloud service provided through an interface (such as a composition interface), which supports users to customize group diagrams based on operators and represents the topology diagram in the form of a queue. The above-mentioned software device can be deployed in a computing device cluster (for example, a computing device that executes the operator or a third-party computing device cluster), and the computing device cluster executes the program code of the software device, thereby executing the operator processing method of the present application. The operator processing device can also be a hardware device, such as a computing device or a computing device cluster with an operator processing function. When the operator processing device is running, the operator processing method of the present application is executed.

[0071] Specifically, the operator processing device obtains the topological information of the computation graph, and the topological information records the multiple operators, graph input data, graph output data, graph intermediate data and the dependency relationship of the multiple operators included in the computation graph, and then constructs the operator execution queue according to the topological information of the computation graph. Each element in the operator execution queue corresponds to an operator in the computation graph and the input data and output data of the operator, the input data of the operator is at least one of the graph input data or the graph intermediate data, the output data of the operator is at least one of the graph intermediate data or the graph output data, and the execution order of the multiple operators in the operator execution queue is determined according to the dependency relationship of the multiple operators.

[0072] This method supports users to customize group graphs according to operators and construct operator execution queues based on the topological information of the computational graph, thereby building execution graphs in the form of operator execution queues. It can be flexibly applied to all scenarios that require batch operator scheduling, including lightweight batch operator scheduling scenarios such as acceleration library construction, meeting business needs.

[0073] The operator processing method of the present application can be applied to scenarios where operators need to be called in batches to complete computing tasks. Typical scenarios include various acceleration library API constructions or large model reasoning acceleration library constructions. Among them, the acceleration API construction includes the deep neural network (DNN) acceleration library API construction, and the large model reasoning acceleration library can include various large language models (LLM), especially conversational language models.

[0074] The operator processing device that executes the operator processing method can be implemented by a computing device or a computing device cluster. For example, a computing device or a computing device cluster can execute the operator processing method of the present application when building a large model reasoning acceleration library. Among them, the computing device can be a server or a terminal. The server can be a cloud server in a cloud environment, or a self-owned physical server, and the terminal can include but is not limited to a desktop, a laptop, and a smart phone. The computing device can adopt a single-machine single-card architecture, or a single-machine multi-card architecture, and the computing device cluster can be a multi-machine multi-card architecture. It should be noted that the "machine" in the above x machine x card represents the host, and the "card" represents an accelerator card and other devices. Among them, the accelerator card may include but is not limited to a GPU or an NPU. The following is an example of a computing device as a server.

[0075] See also Figure 2 The hardware structure diagram of a server shown in FIG. 20 includes a host 22 and at least one device 24. The host 22 and the at least one device 24 are connected.

[0076] The host 22 includes a processor and a memory. The processor may be a central processing unit (CPU), and the memory may be a dual in-line memory module (DIMM). The DIMM may be a double data rate (DDR) type, for example, the memory may be a DDR4 DIMM. Figure 2 In the example, the host 22 includes 4 CPUs and 4 DDR4 DIMM groups, each CPU is connected to 1 DDR4 DIMM group, and each DDR4 DIMM group includes 8 DDR4 DIMMs. Multiple CPUs of the host 22 can be connected to form a hydra mesh.

[0077] Optionally, the host 22 further includes an interface, such as one or more of a Serial Advanced Technology Attachment (SATA) interface, a new generation non-volatile memory express (NVMe) interface, and a Gigabit Ethernet (GE) interface. The host 22 may also include a memory. The memory may include a SATA-supported memory or an NVMe-supported memory, such as a mechanical hard disk that supports SATA, or a solid state drive (SSD) that supports NVMe.

[0078] Device 24 includes an accelerator card, such as an NPU or a GPU. Figure 2 The device side includes 8 NPUs for illustration. In other possible implementations of the embodiment of the present application, the device side may also include more accelerator cards, such as a greater number of accelerator cards or more types of accelerator cards.

[0079] When the device side includes multiple accelerator cards, the multiple accelerator cards can execute multiple operators concurrently. Among them, the multiple operators include communication operators, and the multiple accelerator cards can directly exchange data through the communication operators to execute multiple operators concurrently. When the device side includes one accelerator card, the accelerator card can independently execute multiple operators.

[0080] In order to make the technical solution of the present application clearer and easier to understand, the operator processing method of the present application is introduced below with reference to the accompanying drawings.

[0081] See also Figure 3 A flowchart of an operator processing method is shown, and the method can be executed by an operator processing device. The operator processing device can be a software device, such as a software device deployed on the host side, or a software device deployed on a computing device cluster, which is used to construct multiple operators that need to be batch scheduled into an operator execution queue for execution on the device side. It should be noted that when the operator processing device is a software device deployed on a computing device cluster, the computing device that executes the operator can call the interface of the software device to obtain the operator execution queue, and execute multiple operators on the device side based on the operator execution queue. The operator processing device can also be a hardware device, such as a host on the computing device side, or a computing device cluster. The method includes:

[0082] S302: The operator processing device obtains topological information of the computation graph.

[0083] A computational graph is a topological graph that describes a computational process. A computational graph generally includes operators involved in the computational process, input data of operators, and output data of operators. For the computational graph as a whole, the topological information of the computational graph records the dependencies of multiple operators, graph input data, graph output data, graph intermediate data, and multiple operators included in the computational graph. Among them, graph input data may be input data of several operators among the multiple operators included in the computational graph, and graph output data may be output data of several operators among the multiple operators included in the computational graph. Graph intermediate data may be output data of some (or one) operators among the multiple operators included in the computational graph and input data of other (or another) operators. Among them, the input data, output data of the operator or graph input data, graph output data, and graph intermediate data may be tensors. A tensor may be an array with any integer dimension, as well as various properties describing this array, including but not limited to dimension (shape) information, data type, device type, data format, memory address, etc.

[0084] The computation graph may be a user-defined construction. Specifically, the operator processing device may provide a composition interface, which may be a graphical user interface (GUI), or a command user interface (CUI), or an API, and the user may customize the computation graph through the composition interface. For ease of description, the composition interface is used as an example of a GUI.

[0085] See also Figure 4 A schematic diagram of a composition interface is shown, which provides multiple operators for constructing a computational graph. The user can add multiple operators that need to be batch-scheduled to the composition area by dragging and dropping. Furthermore, the user can view the input data and output data of multiple operators, and the dependency relationship of multiple operators can be determined based on the input data and output data of multiple operators. The user can connect multiple operators based on the dependency relationship to realize custom construction of the computational graph. Accordingly, the operator processing device can obtain the topological information of the computational graph.

[0086] S302: The operator processing device constructs an operator execution queue according to the topological information of the computation graph.

[0087] The operator execution queue is a queue for executing operators. Each element in the operator execution queue corresponds to an operator in the computation graph and the input data and output data of the operator. The input data of the operator is at least one of the graph input data or the graph intermediate data, and the output data of the operator is at least one of the graph intermediate data or the graph output data. The execution order of multiple operators in the operator execution queue is determined according to the dependency relationship of the multiple operators.

[0088] Specifically, the operator processing device may sort the multiple operators according to the dependency relationship of the multiple operators, and then add the multiple operators to the operator execution queue in order. Each operator may correspond to an element in the operator execution queue. The elements in the operator execution queue also record the input data and output data of the operator.

[0089] In some possible implementations, the operator execution queue may be a first-in-first-out (FIFO) queue. Accordingly, the operator processing device may add the previously executed operator to the FIFO queue first, thereby ensuring that the execution order of the operators can satisfy the dependencies between the operators.

[0090] The operator processing device also supports partitioning the graph input data, the graph output data, and the graph intermediate data. Accordingly, each element in the operator execution queue records an operator in the computation graph and the number of the operator's input data and the number of the operator's output data, thereby recording the operators in the computation graph and the operator's input data and output data.

[0091] Furthermore, the operators in the operator execution queue may include at least one compiled operator. For example, the operators in the operator execution queue may be binary operators. Binary operators are computing units in binary coded form that can be executed on the device side without compilation. When all operators in the operator execution queue are binary operators, the operator sequence may also be referred to as a kernel sequence. In this way, the problem of the AI ​​compilation process being too heavy can be solved, and operator scheduling can be achieved through light compilation or even compilation-free.

[0092] To facilitate understanding, an example is provided below for explanation.

[0093] like Figure 5 As shown, the computation graph includes the following binary operators: OP1 to OP5. The computation graph also records the graph input tensor (InTensor), graph output tensor (OutTensor), and graph intermediate tensor (InterTensor), where the graph input tensor is the input tensor of OP1, OP2, and OP5, the graph output tensor is the output tensor of OP5, the graph intermediate tensor includes the output tensors of OP1 and OP2 (the output tensor of OP1 is also the input tensor of OP3, and the output tensor of OP2 is also the input tensor of OP3 and OP4), and the graph intermediate tensor also includes the output tensor of OP3 (the output tensor of OP3 is also the input tensor of OP4) and the output tensor of OP4 (the output tensor of OP4 is also the input tensor of OP5).

[0094] Among them, the graph input tensor, graph output tensor, and graph intermediate tensor can be partitioned and numbered. For example, the above-mentioned graph input tensor can be numbered from 1 to 3, the graph output tensor can be numbered from 4, and the graph intermediate tensor can be numbered from 5 to 8. Based on the dependency relationship of the operators in the topology information, the operator processing device determines that OP1 and OP2 are executed before OP3, OP3 is executed before OP4, and OP4 is executed before OP5. Therefore, OP1 to OP5 can be added to the operator execution queue in sequence, and the tensor numbers of the input tensor and output tensor of the operator are bound to the operator as FIFO queue node information. The tensor numbers of the input tensor and output tensor of the operator can be used as tensor indexes during the graph execution process.

[0095] It should be noted that, in the above example, OP2 may also join the operator execution queue before OP1. In other words, OP2 may also be executed before OP1. In addition, the operator execution queue may also be other types of queues, which are not limited in this embodiment. When operators are executed concurrently on multiple devices, the operators in the operator execution queue may include communication operators, which are used to exchange data between different devices (e.g., different NPUs).

[0096] Based on the above description, it can be seen that the operator processing method of the present application supports user-defined group graphs. The operator execution queue can be constructed according to the topological information of the assembled calculation graph, thereby realizing the construction of the execution graph in the form of an operator execution queue. This can be flexibly applied to all scenarios that require batch operator scheduling, meeting business needs.

[0097] Furthermore, this application also supports planning the memory layout of the internal computing resources of the computational graph in advance according to the input data and the operator execution queue, and reuses the memory according to its life cycle, which improves the memory utilization and greatly reduces the overall memory consumption. In addition, this method supports applying for large blocks of memory at one time, reducing the number of memory applications and improving the overall operating efficiency. The resource planning process is described in detail below.

[0098] The execution of an operator usually requires input data (such as input tensors InTensors), output data (such as OutTensors), slicing parameters (tilling data), and scratch memory. Among them, the slicing parameters are parameters used to slice the input data. The slicing parameters can include the total length of the parameter (totalLength), the number of slices (such as the number of times a single core loop processes data, recorded as tileNum), and the calculation scalar (such as LeakyRelu scalar). The scratch memory is used to temporarily store temporary data generated by the execution of the operator. For a user of a graph, they usually pay attention to: graph input tensors, graph output tensors, and graph workspace (GraphWorkspace). Therefore, the graph intermediate tensors, tillingdata, and scratch memory need to be put into the graph workspace and passed in from outside the graph.

[0099] Among them, the graph workspace refers to the workspace (workspace) where the graph is executed. The workspace can also be called a context environment or a session. The execution of the computational graph needs to be in the context of a session. The Session provides an environment for operator execution and tensor evaluation. A Session may have some resources, such as variables or queues. When the session is no longer needed, you can call sess.close() to close the session (or use the context manager to automatically close it) to release the resources. If there are multiple computational graphs to be executed, you can create different sessions to load each computational graph, and each computational graph can be loaded in multiple sessions for calculation. Sessions are independent of each other.

[0100] Based on this, the operator processing device can determine the size of the graph workspace according to the dimensional information of the graph input data and the correspondence between the operator corresponding to each element in the operator execution queue and the input data and output data of the operator, and apply for memory as the graph workspace according to the size of the graph workspace.

[0101] Among them, the operator processing device can perform chain deduction based on the shape information of the graph input data (such as the graph input tensor) and the corresponding relationship between the operator input data and output data, so as to derive the shape information of the graph intermediate tensor, and then obtain the memory size for storing the graph intermediate tensor. In addition, the operator processing device can also derive the amount of temporary data generated during the execution process, and then estimate the size of the temporary memory. The operator processing device can obtain the size of the entire graph workspace based on the memory size of the graph intermediate tensor, the size of the temporary memory, and the memory size of the slicing parameters. It should be noted that the graph workspace can also be divided into an operator workspace (OpWorkspace) and a graph intermediate data area. Among them, the operator workspace includes a slicing parameter area and a temporary memory, the slicing parameter area is used to store the slicing parameters, and the temporary memory is used to store temporary data generated during the execution of the operator.

[0102] by Figure 6 As an example, the operator processing device can chain-derive the shape information of the graph intermediate tensor based on the shape information of the graph input tensor (such as tensor ID 1, 2, 3), and then obtain the size of the memory required to store the graph intermediate tensor. In addition, the operator processing device can also deduce the size of the memory required for the slicing parameters of each operator and the size of the temporary memory for temporarily storing temporary data generated during the execution of each operator. The operator processing device can predict the size of the graph workspace based on the size of the memory required to store the graph intermediate tensor, the size of the memory required to store the slicing parameters, and the size of the temporary memory, and apply for a memory block of the corresponding size from the device side as the graph workspace. It should be noted that the operator processing device can also apply for a memory block slightly larger than the above size as the graph workspace.

[0103] The following describes the sharding parameters, temporary memory, and memory planning for graph intermediate data.

[0104] For the sharding parameters (tiling data) of an operator, it is usually necessary to load the sharding parameters from the host side to the specified device memory space before the operator is executed. The sharding parameters are no longer used after the operator is executed. In order to reduce the number of loading times, this application supports loading all at once. Accordingly, the life cycle of the sharding parameters can be equivalent to the life cycle of the graph. For the scratch memory of an operator, its life cycle can be the period from the start to the end of the corresponding operator execution. Based on this, the scratch memory of multiple operators can be reused.

[0105] In specific implementation, the operator processing device can determine the offset address (denoted as offset) of the slicing parameters of multiple operators in the graph workspace and the size (size) of the slicing parameters of multiple operators, and load the slicing parameters of multiple operators into the graph workspace according to the offset address of the slicing parameters of multiple operators in the graph workspace and the size of the slicing parameters of the multiple operators. For the temporary memory, the operator processing device can determine the offset address and size of the temporary memory in the graph workspace according to the amount of temporary data generated by the execution of multiple operators. Among them, the size of the temporary memory can be the maximum value of the amount of temporary data. When executing the operator, by passing the offset address and size of the temporary memory in the graph workspace into the graph workspace, the temporary data generated by the execution of the operator can be stored in the corresponding position or area. It should be noted that the temporary memory of multiple operators can have the same offset address, so that memory reuse can be achieved and memory overhead can be reduced.

[0106] by Figure 7 Example description. The tilling data of the operator occupies less memory space on the device side. In order to reduce the number of times the tilling data is loaded, the tilling data of all operators in the operator execution queue can be loaded at one time. Among them, the tilling data of all operators can be placed at the starting address of the Graph Workspace. In other words, the offset of the tilling data can be equal to the offset of the GraphWorkspace. The size of the tilling data area can be equal to the size of the tilling data of all operators in the operator execution queue. The scratch memory of each operator is no longer used after the execution of the operator is completed, so when the subsequent operator is executed, the scratchmemory of the previous operator can be reused. Therefore, the offset of the scratch memory of all operators in the Graph Workspace is the same, but their sizes are different, so you only need to take the largest size.

[0107] The operator processing device may scan the operator execution queue, and write the offset and size of the tilling data and scratch memory of each operator into the operation information (referred to as runInfo) of the operator for use in subsequent execution of the operator.

[0108] For a graph intermediate data (such as a graph intermediate tensor), its life cycle ends immediately after all operators that rely on the graph intermediate data as input finish running. The dimensional information of graph intermediate data is relatively large in typical application scenarios. Without considering memory transfer, reusing device-side memory space as much as possible can improve memory utilization.

[0109] Specifically, the operator processing device can obtain the dimension information of the graph intermediate data and the dependency information of the graph intermediate data, and the dependency information is used to indicate the operator that depends on the graph intermediate data. Then the operator processing device can determine the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data based on the dimension information of the graph intermediate data and the dependency information of the graph intermediate data.

[0110] Among them, the operator processing device can determine the life cycle of each graph intermediate data according to the dependency information of the graph intermediate data, and then the operator processing device can determine the arrangement of the graph intermediate data in the memory based on the dimension information of the graph intermediate data and the life cycle of each graph intermediate data, thereby obtaining the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data. For example, the graph intermediate data includes the first graph intermediate data and the second graph intermediate data. When the operator that depends on the first graph intermediate data is executed before the operator that outputs the second graph intermediate data, the offset address of the first graph intermediate data is less than or equal to the offset address of the second graph intermediate data, that is, the second graph intermediate data can reuse the memory space of the first graph intermediate data.

[0111] by Figure 8 Example description. After OP1 is executed, the intermediate tensor of the output graph (i.e., tensor 5) is output. Tensor 5 is dependent on the subsequent OP3, so the memory space of tensor 5 is not released. After OP2 is executed, the intermediate tensor of the output graph (i.e., tensor 6) is output. Tensor 6 is dependent on the subsequent OP3, so the memory space of tensor 6 is not released. At this time, the memory space of tensor 5 and tensor 6 is not released. OP3 executes dependent tensors 5 and 6, and after execution, it outputs the intermediate tensor of the graph (i.e., tensor 7). At this time, tensor 5 is not dependent on subsequent operators, and the memory space of tensor 5 can be released. OP3 executes dependent tensors 6 and 7, and after execution, it outputs the intermediate tensor of the output graph (i.e., tensor 8). The memory space used by tensor 8 is smaller than the memory space used by tensor 5, and tensor 8 can reuse the memory space of tensor 5. Based on this, the operator processing device can configure the offset of tensor 8 to be equal to the offset of tensor 5. In other possible implementations of the embodiment of the present application, the operator processing device may also configure the offset of tensor 8 to be greater than (for example, slightly greater than) the offset of tensor 5, that is, the offset of tensor 5 is smaller than the offset of tensor 8, so as to achieve memory reuse of tensor 5 and tensor 8. After OP5 is executed, the memory space of all the intermediate tensors of the graph can be released.

[0112] When the memory used by tensor 8 is larger than the memory used by tensor 5, for example, the memory used by tensor 8 is slightly larger than the memory used by tensor 5, the memory space of tensor 5 cannot be reused, and the memory space can only be expanded to the right. In order to avoid the above situation, the operator processing device can merge adjacent free blocks (i.e., the memory blocks where the tensors are released), and try to select the smallest size block that meets the requirements each time, such as selecting the smallest size block that meets the requirements for reuse through the BestFit algorithm, so as to retain more and larger blocks as much as possible and improve the memory reuse rate. In addition, the operator processing device can also plan in advance according to global information, such as shifting the starting addresses of tensors 6 and 7 to the right in advance, so that after the subsequent release of tensor 5, tensor 8 can be placed after the adjacent fragments are merged.

[0113] After completing the above memory planning, you can execute the operator execution queue (this process is also called graph execution). Fig. 9 As shown, the execution of operators in the operator execution queue requires the preparation of runInfo of all operators. The main process is to update the data pointer of the tensor in the middle of the graph and the workspace pointer of the operator. Since the operator processing device has stored the offset of each resource block during memory planning, based on this, when the operator processing device updates the pointer (data pointer, workspace pointer), it only needs to add the offset to the base address of the workspace memory passed in from the outside.

[0114] Further, see Fig.10 During the update process, the operator processing device can determine whether the input tensor and output tensor of each operator belong to the intermediate tensor of the graph based on the area of ​​the tensor in the tensor table of the graph. Since the data pointers of the graph input and output tensors do not need to be updated and are real addresses themselves, the above-mentioned update operation can be omitted. The operator processing device can perform the above-mentioned update operation when it is determined that the input tensor and output tensor of the operator belong to the intermediate tensor of the graph.

[0115] In some possible implementations, the operator processing device can support parallel execution of computation graphs. In particular, the operator processing device can asynchronously execute graph calculations, and prepare computing resources for the next computation graph in advance while the device side (such as the NPU on the device side) is still calculating, which will reduce the waiting time on the device side and increase the operator throughput. Fig.11 In different graphs or different shape scenarios of the same graph, the operator processing device can carry out calculations on the host side. Figure 1 (graph 1) When tiling data is sent to the device, the calculation is performed asynchronously Figure 2 Resource planning (graph 2), for example, is to calculate the offset and size of the resource block.

[0116] Furthermore, for the same graph and the same shape scenario, the present application also supports the reuse of graph workspaces. Specifically, the calculation graph includes a first calculation graph (such as graph 1) and a second calculation graph (such as graph 2), the operators included in the first calculation graph are the same as the operators included in the second calculation graph, the dependencies of the operators in the first calculation graph are the same as the dependencies of the operators in the second calculation graph, and the dimension information of the data in the first calculation graph is the same as the dimension information of the data in the second calculation graph. When executing the first calculation graph, the operator processing device can adjust the address of the graph input data to the address of the graph input data of the second calculation graph. In this way, after the first calculation graph is executed, the operator processing device can directly reuse the graph workspace of the first calculation graph to execute the second calculation graph. This method improves the operator throughput on the device side and improves the overall running speed by asynchronously executing different calculation graphs.

[0117] Based on the above-mentioned operator processing method, the present application provides an operator processing device. The operator processing device will be introduced from the perspective of functional modularization.

[0118] See also Fig.12 The schematic diagram of the structure of an operator processing device shown in FIG. 1 includes:

[0119] An acquisition module 1202 is used to acquire topological information of a computation graph, wherein the topological information records a plurality of operators, graph input data, graph output data, graph intermediate data and dependency relationships of the plurality of operators included in the computation graph;

[0120] Construction module 1204 is used to construct an operator execution queue according to the topological information of the computation graph, wherein each element in the operator execution queue corresponds to an operator in the computation graph and the input data and output data of the operator, the input data of the operator is at least one of the graph input data or the graph intermediate data, the output data of the operator is at least one of the graph intermediate data or the graph output data, and the execution order of multiple operators in the operator execution queue is determined according to the dependency relationship between the multiple operators.

[0121] The acquisition module 1202 is used to implement the content related to S302 in the aforementioned method embodiment, and the construction module 1204 is used to implement the content related to S304 in the aforementioned method embodiment.

[0122] The acquisition module 1202 and the construction module 1204 may be implemented by software or hardware.

[0123] When implemented by software, the acquisition module 1202 and the construction module 1204 may be applications running on a computing device. For example, the construction module 1204 may be a computing engine running on a computing device. The application may be provided to users in the form of virtualized services. Virtualized services may include virtual machine (VM) services, bare metal server (BMS) services, and container services. Among them, the VM service may be a service that virtualizes a virtual machine (VM) resource pool on multiple physical hosts (such as computing devices) through virtualization technology to provide users with VMs on demand for use. The BMS service is a service that virtualizes a BMS resource pool on multiple physical hosts to provide users with BMS on demand for use. The container service is a service that virtualizes a container resource pool on multiple physical hosts to provide users with containers on demand for use. VM is a simulated virtual computer, that is, a logical computer. BMS is a high-performance computing service that can be elastically scalable, and its computing performance is no different from that of a traditional physical machine, and it has the characteristics of secure physical isolation. Container is a kernel virtualization technology that can provide lightweight virtualization to achieve the purpose of isolating user space, processes and resources. It should be understood that the VM service, BMS service and container service in the above virtualization services are only used as specific examples. In actual applications, the virtualization service can also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0124] When implemented by hardware, the acquisition module 1202 and the construction module 1204 may include at least one processor, such as a CPU, etc. Alternatively, the acquisition module 1202 and the construction module 1204 may also be implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0125] In some possible implementations, the construction module 1204 is further configured to:

[0126] Partition numbering the graph input data, the graph output data, and the graph intermediate data;

[0127] Each element in the operator execution queue records an operator in the computation graph and the number of the input data of the operator and the number of the output data of the operator.

[0128] In some possible implementations, the apparatus 1200 further includes:

[0129] The memory planning module 1206 is used to determine the size of the graph workspace according to the dimension information of the graph input data and the correspondence between the operator corresponding to each element in the operator execution queue and the input data and output data of the operator, and apply for memory as the graph workspace according to the size of the graph workspace.

[0130] Similar to the construction module 1204, the memory planning module 1206 may be implemented by software or by hardware.

[0131] When implemented by software, the memory planning module 1206 may be an application running on a computing device. For example, the memory planning module 1206 may be a computing engine running on a computing device. The application may be provided to the user in the form of a virtualized service. For example, the application may be provided to the user via a VM service, a BMS service, or a container service.

[0132] When implemented by hardware, the memory planning module 1206 may include at least one processor, such as a CPU, etc. Alternatively, the memory planning module 1206 may also be implemented by a dedicated integrated circuit ASIC, or a programmable logic device PLD (such as CPLD, FPGA), etc.

[0133] In some possible implementations, the memory planning module 1206 is further configured to:

[0134] Obtain dimension information of the graph intermediate data and dependency information of the graph intermediate data, wherein the dependency information is used to indicate operators that depend on the graph intermediate data, and determine the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data according to the dimension information of the graph intermediate data and the dependency information of the graph intermediate data.

[0135] In some possible implementations, the graph intermediate data includes first graph intermediate data and second graph intermediate data. When an operator that depends on the first graph intermediate data is executed before an operator that outputs the second graph intermediate data, the offset address of the first graph intermediate data is less than or equal to the offset address of the second graph intermediate data.

[0136] In some possible implementations, the memory planning module 1206 is further configured to:

[0137] Determine the offset addresses of the slicing parameters of the multiple operators in the graph workspace and the sizes of the slicing parameters of the multiple operators;

[0138] The sharding parameters of the multiple operators are loaded into the graph workspace according to the offset addresses of the sharding parameters of the multiple operators in the graph workspace and the sizes of the sharding parameters of the multiple operators.

[0139] In some possible implementations, the graph workspace is further provided with a temporary memory;

[0140] The memory planning module 1206 is also used for:

[0141] The offset address and size of the graph workspace in the temporary storage are determined according to the amount of temporary data generated by the execution of the multiple operators.

[0142] In some possible implementations, the temporary memories of the multiple operators have the same offset address.

[0143] In some possible implementations, the computation graph includes a first computation graph and a second computation graph, the operators included in the first computation graph are the same as the operators included in the second computation graph, the dependencies of the operators in the first computation graph are the same as the dependencies of the operators in the second computation graph, and the dimension information of the data in the first computation graph is the same as the dimension information of the data in the second computation graph;

[0144] The device 1200 further includes:

[0145] The adjustment module 1208 is used to adjust the address of the graph input data to the address of the graph input data of the second computation graph when executing the first computation graph.

[0146] Similar to the acquisition module 1202 and the construction module 1204 described above, the adjustment module 1208 may be implemented by software or by hardware.

[0147] When implemented by software, the adjustment module 1208 may be an application running on a computing device. The application may be provided to the user in the form of a virtualized service. For example, the application may be provided to the user via a VM service, a BMS service, or a container service.

[0148] When implemented by hardware, the adjustment module 1208 may include at least one processor, such as a CPU, etc. Alternatively, the adjustment module 1208 may also be implemented by an application specific integrated circuit ASIC, or a programmable logic device PLD (such as CPLD, FPGA), etc.

[0149] Fig.12The operator processing device is introduced from the perspective of functional modularization. Next, the operator processing device is introduced from the perspective of hardware instantiation.

[0150] The operator processing device may include at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the operator processing device executes the operator processing method of the aforementioned embodiment.

[0151] The operator processing device may be a computing device, for example Figure 2 Furthermore, the operator processing device may also be a computing device cluster formed by multiple computing devices, for example, multiple computing devices such as Figure 2 The servers shown form a computing device cluster. The memories of multiple computing devices in the computing device cluster may store the same operator processing apparatus 1200 for executing instructions of the operator processing method.

[0152] In some possible implementations, multiple computing devices in the computing device cluster may also be used to execute some instructions of the operator processing device 1200 for executing the operator processing method. In other words, the combination of multiple computing devices may jointly execute the instructions of the operator processing device 1200 for executing the operator processing method.

[0153] It should be noted that the memories in different computing devices in the computing device cluster may store different instructions for executing part of the functions of the operator processing apparatus 1200 .

[0154] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned application to the operator processing device 1200 for executing the operator processing method.

[0155] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the above-mentioned operator processing method.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. An operator processing method, characterized in that: The method comprises: Obtaining topological information of a computational graph, wherein the topological information records a plurality of operators, graph input data, graph output data, graph intermediate data, and dependency relationships of the plurality of operators included in the computational graph; According to the topological information of the computational graph, an operator execution queue is constructed, wherein each element in the operator execution queue corresponds to an operator in the computational graph and the input data and output data of the operator, the input data of the operator is at least one of the graph input data or the graph intermediate data, the output data of the operator is at least one of the graph intermediate data or the graph output data, and the execution order of multiple operators in the operator execution queue is determined according to the dependency relationship between the multiple operators.

2. The method according to claim 1, characterized in that The method further comprises: Partition numbering the graph input data, the graph output data, and the graph intermediate data; Each element in the operator execution queue records an operator in the computation graph and the number of the input data of the operator and the number of the output data of the operator.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Determine the size of the graph workspace according to the dimension information of the graph input data and the correspondence between the operator corresponding to each element in the operator execution queue and the input data and output data of the operator; Apply for memory as the graph workspace according to the size of the graph workspace.

4. The method according to claim 3, characterized in that The method further comprises: Acquire dimension information of the graph intermediate data and dependency information of the graph intermediate data, where the dependency information is used to indicate operators that depend on the graph intermediate data; According to the dimension information of the graph intermediate data and the dependency information of the graph intermediate data, the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data are determined.

5. The method according to claim 4, characterized in that The graph intermediate data includes first graph intermediate data and second graph intermediate data. When an operator dependent on the first graph intermediate data is executed before an operator outputting the second graph intermediate data, the offset address of the first graph intermediate data is less than or equal to the offset address of the second graph intermediate data.

6. The method according to any one of claims 3 to 5, characterized in that: The method further comprises: Determine the offset addresses of the slicing parameters of the multiple operators in the graph workspace and the sizes of the slicing parameters of the multiple operators; The sharding parameters of the multiple operators are loaded into the graph workspace according to the offset addresses of the sharding parameters of the multiple operators in the graph workspace and the sizes of the sharding parameters of the multiple operators.

7. The method according to any one of claims 3 to 6, characterized in that: The graph workspace is also provided with a temporary memory; The method further comprises: The offset address and size of the graph workspace in the temporary storage are determined according to the amount of temporary data generated by the execution of the multiple operators.

8. The method according to claim 7, characterized in that The temporary storage memories of the multiple operators have the same offset address.

9. The method according to any one of claims 1 to 8, characterized in that: The computation graph includes a first computation graph and a second computation graph, the operators included in the first computation graph are the same as the operators included in the second computation graph, the dependencies of the operators in the first computation graph are the same as the dependencies of the operators in the second computation graph, and the dimension information of the data in the first computation graph is the same as the dimension information of the data in the second computation graph; The method further comprises: When executing the first computation graph, the address of the graph input data is adjusted to the address of the graph input data of the second computation graph.

10. An operator processing device, characterized in that: The device comprises: An acquisition module is used to acquire topological information of a computational graph, wherein the topological information records a plurality of operators, graph input data, graph output data, graph intermediate data and dependency relationships of the plurality of operators included in the computational graph; A construction module is used to construct an operator execution queue according to the topological information of the computational graph, wherein each element in the operator execution queue corresponds to an operator in the computational graph and the input data and output data of the operator, the input data of the operator is at least one of the graph input data or the graph intermediate data, the output data of the operator is at least one of the graph intermediate data or the graph output data, and the execution order of multiple operators in the operator execution queue is determined according to the dependency relationship between the multiple operators.

11. The device according to claim 10, characterized in that The construction module is also used for: Partition numbering the graph input data, the graph output data, and the graph intermediate data; Each element in the operator execution queue records an operator in the computation graph and the number of the input data of the operator and the number of the output data of the operator.

12. The device according to claim 10 or 11, characterized in that The device also includes: A memory planning module is used to determine the size of the graph workspace according to the dimensional information of the graph input data and the correspondence between the operator corresponding to each element in the operator execution queue and the input data and output data of the operator, and apply for memory as the graph workspace according to the size of the graph workspace.

13. The device according to claim 12, characterized in that The memory planning module is also used for: Obtain dimension information of the graph intermediate data and dependency information of the graph intermediate data, wherein the dependency information is used to indicate operators that depend on the graph intermediate data, and determine the offset address of the graph intermediate data in the graph workspace and the size of the graph intermediate data according to the dimension information of the graph intermediate data and the dependency information of the graph intermediate data.

14. The device according to claim 13, characterized in that The graph intermediate data includes first graph intermediate data and second graph intermediate data. When an operator dependent on the first graph intermediate data is executed before an operator outputting the second graph intermediate data, the offset address of the first graph intermediate data is less than or equal to the offset address of the second graph intermediate data.

15. The device according to any one of claims 12 to 14, characterized in that The memory planning module is also used for: Determine the offset addresses of the slicing parameters of the multiple operators in the graph workspace and the sizes of the slicing parameters of the multiple operators; The sharding parameters of the multiple operators are loaded into the graph workspace according to the offset addresses of the sharding parameters of the multiple operators in the graph workspace and the sizes of the sharding parameters of the multiple operators.

16. The device according to any one of claims 12 to 15, characterized in that The graph workspace is also provided with a temporary memory; The memory planning module is also used for: The offset address and size of the graph workspace in the temporary storage are determined according to the amount of temporary data generated by the execution of the multiple operators.

17. The device according to claim 16, characterized in that The temporary storage memories of the multiple operators have the same offset address.

18. The device according to any one of claims 10 to 17, characterized in that The computation graph includes a first computation graph and a second computation graph, the operators included in the first computation graph are the same as the operators included in the second computation graph, the dependencies of the operators in the first computation graph are the same as the dependencies of the operators in the second computation graph, and the dimension information of the data in the first computation graph is the same as the dimension information of the data in the second computation graph; The device also includes: An adjustment module is used to adjust the address of graph input data to the address of graph input data of the second calculation graph when executing the first calculation graph.

19. An operator processing device, characterized in that: The device includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the operator processing device executes the operator processing method as described in any one of claims 1 to 9.

20. A computer program product, characterized in that Comprising computer-readable instructions; the computer-readable instructions are used to implement the operator processing method according to any one of claims 1 to 9.

Images