Method for optimizing instruction execution time by using computational graph, and computational graph execution method and device

Abstract

Embodiments of the invention provide a method for optimizing an instruction execution moment by using a computational graph, and a computational graph execution method and apparatus. The computational graph includes nodes representing instructions and edges representing dependencies between the instructions. When the instructions in the computational graph are optimized, node time consumption and edge time consumption during execution of the computational graph can be determined, and a plurality of to-be-planned constraints containing execution moments of a plurality of instructions are constructed based on a dependency relationship among the instructions in the computational graph, the node time consumption, the edge time consumption and instruction overlapping execution capability of hardware; and determining the latest execution moment of the instructions in the plurality of to-be-planned constraints, and solving the plurality of to-be-planned constraints by taking minimization of the latest execution moment as a target to obtain optimized execution moments of the plurality of instructions, so that an executor can execute the plurality of instructions according to the optimized instruction execution moments when executing the computational graph. The execution sequence of a plurality of instructions in a computational graph belongs to privacy data, and privacy protection needs to be carried out.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer technology, and more particularly to a method for optimizing instruction execution timing using a computation graph, and a computation graph execution method and apparatus. Background Art

[0002] The computational graph is a core component of the deep learning framework, which provides powerful support for building, training, and deploying complex neural network models. The computational graph graphically represents the structure of the network model, including the various layers in the model and the connections between them. The computational graph defines the flow path of data in the network. Each node receives input data, performs calculations, and then passes the output to the next node. The data processing process in the computational graph includes privacy protection processing for private data. Using computational graphs can also help optimize the use of computing resources and improve the efficiency of model training and reasoning.

[0003] At present, it is hoped that there will be an improved solution that can use the computational graph to maximize the efficiency of utilizing the computing resources in the computer. Summary of the invention

[0004] One or more embodiments of this specification describe a method for optimizing instruction execution time using a computation graph, a computation graph execution method and device, so as to utilize the computation graph to maximize the utilization efficiency of computing resources in a computer. The specific technical solution is as follows.

[0005] In a first aspect, an embodiment provides a method for optimizing instruction execution time using a computation graph, wherein the computation graph includes nodes representing instructions and edges representing dependencies between instructions; the method includes:

[0006] Determine the node time and edge time of the computation graph during execution;

[0007] Based on the dependency relationship between instructions in the computation graph, the node time consumption and the edge time consumption, and the instruction overlapping execution capability of the hardware, construct a plurality of constraints to be planned including the execution time of a plurality of instructions;

[0008] Determining the latest execution time of the instructions in the plurality of constraints to be planned;

[0009] With the goal of minimizing the latest execution time, the plurality of constraints to be planned are solved to obtain optimized execution times of the plurality of instructions.

[0010] In one embodiment, the step of determining the node time consumption and edge time consumption of the computation graph during execution includes:

[0011] Determine the instruction time consumed when executing the instructions in the computation graph;

[0012] Based on the timing of several instructions, the corresponding node timing and edge timing are determined.

[0013] In one embodiment, the step of determining the instruction time consumed when executing instructions in the computation graph includes:

[0014] Obtain a data table, wherein the data table is used to store instruction parameters and corresponding instruction time consumption;

[0015] Determining instruction parameters of instructions in the computation graph;

[0016] Matching the instruction parameters of the instructions in the computation graph with the instruction parameters in the data table respectively;

[0017] When the match is successful, the instruction time corresponding to the successfully matched instruction parameter in the data table is determined as the instruction time of the instruction in the calculation graph;

[0018] When the match fails, the corresponding instruction time consumption is estimated based on the instruction parameters of the instruction in the computation graph and the hardware parameters when the instruction is executed.

[0019] In one embodiment, the instruction includes a computing instruction and / or a communication instruction, and the node corresponding to the communication instruction includes a subnode representing a communication start instruction and a subnode representing a communication end instruction and an edge between the two subnodes; the step of determining the corresponding node time consumption and edge time consumption based on the time consumption of several instructions includes:

[0020] When a node represents a computing instruction, the instruction time of the computing instruction is used as the node time, and the preset small time is used as the edge time connected to the node;

[0021] When a node represents a communication instruction, the instruction time of the communication instruction is used as the edge time of the edge included in the node, and the preset small time is used as the node time of the child node included in the node.

[0022] In one embodiment, the step of constructing a plurality of constraints to be planned including the execution times of a plurality of instructions comprises:

[0023] When there is a dependency relationship between two instructions in the computation graph, constructing a constraint to be planned for sequential execution based on node time consumption and edge time consumption associated with the two instructions;

[0024] When there is no dependency relationship between two instructions in the computation graph, constraints to be planned for overlapping execution are constructed based on node timing and edge timing associated with the two instructions and overlapping execution capabilities of the instructions.

[0025] In one embodiment, the instructions include computing instructions and communication instructions; the instruction overlapping execution capability includes allowing the computing instructions and the communication instructions to be executed overlappingly;

[0026] The step of constructing constraints to be planned for overlapping execution includes:

[0027] For two nodes of the same type without a dependency relationship in the computation graph, the constraints to be planned for overlapping execution constructed include: a start execution time of one node is later than an end execution time of another node.

[0028] In one implementation, the two nodes are a first node and a second node; and the constraints to be planned for overlapping execution of the two nodes specifically include:

[0029] Through the logical variables to be constrained, control the following constraints to be satisfied:

[0030] The start execution time of the first node is later than the end execution time of the second node;

[0031] The start execution time of the second node is later than the end execution time of the first node.

[0032] In one embodiment, the computation graph is used to be executed in a distributed system, the communication instructions are used to perform communication operations between multiple executors included in the distributed system, and the computation instructions are used to perform computation operations within the executors.

[0033] In one implementation, the execution time of the instruction includes the start execution time and the end execution time of the instruction, and the latest execution time is the latest end execution time of the instruction.

[0034] In a second aspect, an embodiment provides a method for executing a computation graph in a distributed system, wherein the computation graph includes nodes representing instructions and edges representing dependencies between instructions; the method includes:

[0035] Obtain optimized execution times of several instructions in the computation graph; the optimized execution times of the several instructions are determined using the method described in the first aspect;

[0036] According to the optimized execution time, the calculation graph is jointly executed by multiple executors included in the distributed system.

[0037] In a third aspect, an embodiment provides a device for optimizing instruction execution time using a computation graph, wherein the computation graph includes nodes representing instructions and edges representing dependencies between instructions; the device includes:

[0038] A time-consuming determination module, configured to determine the node time-consuming and edge time-consuming of the computation graph during execution;

[0039] A constraint construction module is configured to construct a plurality of constraints to be planned including execution times of a plurality of instructions based on dependencies between instructions in the computation graph, the node time consumption and the edge time consumption, and the instruction overlapping execution capability of the hardware;

[0040] A time determination module, configured to determine the latest execution time of the instructions in the plurality of constraints to be planned;

[0041] The constraint solving module is configured to solve the plurality of constraints to be planned with the goal of minimizing the latest execution time, and obtain optimized execution times of the plurality of instructions.

[0042] In a fourth aspect, an embodiment provides a computing graph execution device in a distributed system, wherein the computing graph includes nodes representing instructions and edges representing dependencies between instructions; the device includes:

[0043] A sorting acquisition submodule, configured to acquire optimized execution times of a plurality of instructions in the computation graph; the optimized execution times of the plurality of instructions are determined using the method described in the first aspect;

[0044] The computation graph execution submodule is configured to jointly execute the computation graph through multiple executors included in the distributed system according to the optimized execution time.

[0045] In a fifth aspect, an embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute any one of the methods in the first to second aspects.

[0046] In a sixth aspect, an embodiment provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any one of the first aspect to the second aspect is implemented.

[0047] In the method and apparatus provided in the embodiments of this specification, the constraints to be planned are constructed by calculating the dependencies between instructions in the graph, the node time consumption and the edge time consumption, and the instruction overlapping execution capability of the hardware, and the constraints to be planned are solved with the goal of minimizing the latest execution time in the constraints to be planned. The computation graph is executed according to the optimized execution time of the solved instructions, which can minimize the overall execution time of the computation graph, thereby maximizing the efficiency of instruction execution and improving the utilization efficiency of computing resources in the computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A schematic diagram of a structure of a calculation graph provided for an embodiment;

[0050] Figure 2 A flowchart of a method for optimizing instruction execution time by using a computation graph provided by an embodiment;

[0051] Figure 3 A schematic diagram of a distributed system implementation scenario provided by an embodiment;

[0052] Figure 4 A schematic diagram of a flow chart of a method for executing a computational graph in a distributed system provided in an embodiment;

[0053] Figure 5 A schematic block diagram of a device for optimizing instruction execution time using a computation graph provided in an embodiment;

[0054] Figure 6 A schematic diagram of a flow chart of a computational graph execution device in a distributed system provided in an embodiment. DETAILED DESCRIPTION

[0055] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0056] Figure 1 A schematic diagram of a structure of a computation graph provided in an embodiment. It includes multiple nodes represented by circles and edges between nodes represented by arrow lines. The numbers in the circles represent the node numbers, and the directions of the arrows represent the input and output relationships of the data, and also represent the dependency relationships between the nodes. Figure 1 This is only an example and does not limit the structure of the computational graph. In actual applications, the computational graph may contain more nodes and more complex connection relationships.

[0057] Among them, nodes represent instructions, and edges represent dependencies between instructions. This dependency refers to the data transmission relationship between nodes. In the computational graph, edges are used to express the connection relationship and dependency between instructions. They are usually unidirectional. If the input of an instruction needs to rely on the output of other instructions, then other instructions will be connected to this instruction through an edge. For example, node 5 is used to represent a node, and edge 54 is used to represent the edge from node 5 to node 4. This means that there is a dependency relationship between node 5 and node 4, that is, node 5 outputs data to node 4.

[0058] An instruction is the smallest execution unit used to express variables, functions, and operations in a computational graph. An instruction is also an operation that can be performed by computer hardware. For example, instructions may include matrix multiplication instructions, matrix addition instructions, function calculation instructions, start transfer instructions, end transfer instructions, read data instructions, and store data instructions. The hardware that executes instructions may include a CPU and / or a GPU. A GPU (Graphics Processing Unit) is a hardware chip specially designed to efficiently process large amounts of graphics and image data. With the development of technology, GPUs have gradually demonstrated powerful parallel computing capabilities and are widely used in the field of general computing, especially for computationally intensive, highly parallelized tasks such as artificial intelligence.

[0059] Instructions can be divided into communication instructions and computing instructions. Computing instructions are mostly used to perform calculations inside the device, such as matrix multiplication instructions. Communication instructions are used to perform data transmission operations between devices of different numbers and channels, such as global protocol instructions.

[0060] A computational graph is a data structure used to represent mathematical computation processes, and is particularly suitable for representing complex computational processes involving a large number of variables, functions, and operations. In machine learning and deep learning, computational graphs are widely used to describe the structure of models, organize computational tasks, and perform efficient numerical calculations. Using nodes to represent instructions and edges to represent dependencies between instructions, a computational graph containing multiple instructions and dependencies between instructions can be constructed.

[0061] In machine learning and deep learning scenarios, the computational graph may be a computational graph corresponding to a neural network model, which reflects the connection relationship between each layer and the layers, and the instructions in the computational graph are data processing related operations contained in the layers of the neural network model. Executing the instructions in the computational graph can realize the execution of the neural network model, which includes forward calculations and backward calculations and model prediction processes. The computational graph in the embodiments of this specification is not limited to the computational graph of the neural network model. As long as there are multiple instructions and the dependencies between them, the computational graph can be constructed based on this dependency.

[0062] When executing a computation graph, the hardware of the execution device can usually execute multiple instructions in sequence according to the dependencies between the instructions in the computation graph. However, the computation graph also contains instructions that have no direct dependencies, such as Figure 1There is no dependency between nodes 5 and 6, and there is no dependency between nodes 2, 3, and 4, and so on. For instructions without dependencies, the execution of these instructions by the hardware will affect the overall execution efficiency of the computation graph. In other words, the overlapping execution of instructions without dependencies can be achieved by using the hardware's instruction overlapping execution capability, so as to make full use of the hardware's computing resources and achieve the purpose of optimizing computing resources.

[0063] Which instructions can be executed in an overlapping manner and the overall execution order of all instructions usually depend on the order in which the developer writes the program for the computation graph. This execution order of instructions cannot achieve a relatively optimized state.

[0064] In order to shorten the overall execution time of the computation graph and improve the utilization efficiency of hardware computing resources, an embodiment of this specification provides a method for optimizing the execution time of instructions using a computation graph. The method can be executed by a computing device, and the computing device can be the same as or different from the execution device. The computing device is used to execute the method for optimizing the execution time of instructions using a computation graph to obtain the optimized execution time of the instruction. The execution device is used to execute the computation graph based on the optimized execution time of the instruction to implement the corresponding computing task, such as implementing the execution of a neural network model.

[0065] Combine the following Figure 2 This embodiment will be described in detail.

[0066] Figure 2 A flowchart of a method for optimizing instruction execution time using a computation graph is provided in an embodiment. The computation graph G includes nodes representing instructions and edges representing dependencies between instructions. The method is executed by a computing device and includes the following steps.

[0067] Step S210, determining the node time and edge time of the computation graph G during execution. This refers to determining the node time of each node and the edge time of each edge in the computation graph. This step can determine the corresponding node time and edge time based on the instruction time, and the instruction time is the time consumed when the instruction is executed. The instruction refers to the length of time consumed.

[0068] In one implementation, the corresponding node time and the associated edge time may be determined based on the instruction time and the instruction type. In this implementation, step S210 may be specifically performed by following steps 1 and 2.

[0069] Step 1: Determine the instruction time consumed when executing instructions in the computation graph G.

[0070] Step 2: Determine the corresponding node time and edge time based on the time consumption of a number of instructions. The number includes one or more.

[0071] In step 1, several instructions in the computation graph G may be executed according to the existing instruction execution time. During the execution of the computation graph G, the instruction time consumption of several instructions in the computation graph G is measured.

[0072] In one implementation, in order to improve processing efficiency, the instruction time consumption in step 1 may also be obtained according to the following steps 1_1 to 1_5.

[0073] Step 1_1, obtain the data table inst_profiler, which is used to store instruction parameters and corresponding instruction time consumption. The data table inst_profiler can be stored in the database DB in the form of a hash function. Among them, the instruction parameters can be hashed to obtain an index string. When storing, it can be stored in the DB according to the corresponding relationship between the index string and the instruction time consumption.

[0074] In this step, the computation graph G can be tested according to the existing instruction execution time, that is, several instructions in the computation graph G are executed and the instruction time consumption of several instructions in the computation graph G is measured. Next, the instruction parameters of each instruction in the computation graph G are collected, and the corresponding relationship between the instruction parameters and the instruction time consumption is constructed, and the corresponding relationship is stored in the data table inst_profiler.

[0075] When testing the computation graph G, virtual samples or real samples may be input into the computation graph G, and the computation graph G may be run to measure the instruction time consumption of several instructions in the computation graph G.

[0076] An instruction may include an instruction name and instruction parameters, and the instruction parameters include the data type, data shape, and execution instruction type of the instruction operation. For example, for a calculation instruction, the data type includes precision or half-precision; the data shape includes the dimension of the matrix, the dimension of the vector, etc. The matrix is ​​768*1024 or 1024*4096; the execution instruction type includes matrix addition and matrix multiplication. The instruction can be uniquely identified based on the instruction parameters.

[0077] Step 1_2, determine the instruction parameters of the instructions in the computation graph G. In this step, the instruction parameters can be collected from the computation graph G in an existing manner.

[0078] The instruction parameters of the instructions in the calculation graph G in this step may be different from the instruction parameters in the data table. The calculation graph G in this step is a calculation graph corresponding to a specific task, and the specific task may be different from the task performed by the calculation graph when the data table inst_profiler is constructed. The structures of the two calculation graphs may be the same, but the instruction parameters may have changed. For example, when constructing the data table inst_profiler, the task targeted by the calculation graph G may be the processing of a 768*1024-dimensional image, while the current specific task is to use the calculation graph G to process a 1024*4096-dimensional image. For another example, when constructing the data table inst_profiler, the task targeted by the calculation graph G may be the processing of 2*2-dimensional user data, while the current specific task is to use the calculation graph G to process 2*8-dimensional user data.

[0079] Step 1_3, match the instruction parameters of the instructions in the computation graph G with the instruction parameters in the data table inst_profiler. Specifically, the instruction parameters in the computation graph G can be hashed to obtain a hash value, and the hash value is matched with the index string in the data table inst_profiler. When the match is successful, execute step 1_4. When the match fails, execute step 1_5.

[0080] Step 1_4, determine the instruction time corresponding to the successfully matched instruction parameter in the data table inst_profiler as the instruction time of the instruction in the computation graph G. When the instruction parameter matches successfully, it means that the instruction parameter of a certain instruction in the computation graph G is exactly the same as the instruction parameter of a certain instruction in the data table inst_profiler. At this time, it can be determined that the two instructions are exactly the same, and the instruction time of the instruction can be reused.

[0081] Step 1_5, based on the instruction parameters of the instructions in the calculation graph G and the hardware parameters when executing the instructions, estimate the corresponding instruction time. Specifically, the computational amount of the instruction can be determined based on the instruction parameters, and the instruction time can be determined using the computational amount and the hardware parameters. For example, for a calculation instruction, the instruction time can be estimated according to the following formula: instruction computational amount / hardware computing power corresponding to the instruction. The hardware computing power can be determined by hardware parameters. An online estimator can also be used to estimate the instruction time based on instruction parameters and hardware parameters.

[0082] In one implementation, for step 1, steps 1_1 to 1_4 may not be performed, and step 1_5 may be directly used to determine the instruction time consumption of each instruction.

[0083] In step 2, the corresponding node time and edge time are determined based on several instruction times. The instruction time of a certain instruction can be used as the node time of the node corresponding to the instruction, and the preset micro time can be used as the edge time. The preset micro time is a preset value, which is much smaller than the instruction time and can be ignored during calculation. However, the preset micro time cannot be set to 0. For example, when the instruction time is between 1000 and 2000 μs, the preset micro time can be set to a value such as 1 μs, 2 μs or 3 μs.

[0084] Step S220, based on the dependencies between instructions in the computation graph, node timing and edge timing, and the instruction overlapping execution capability of the hardware, construct a plurality of constraints to be planned including the execution times of a plurality of instructions.

[0085] Among them, the constraint to be planned is a constraint, and a constraint refers to a relationship, which can be an equality relationship or a greater than or less than relationship. The execution time of an instruction includes the start execution time and the end execution time of the instruction. In the constructed constraints to be planned, the relationship between the execution time of the instruction, the node time and the edge time is constructed. In other words, several constraints to be planned are constraints between the execution time, the node time and the edge time of the instruction. It should be noted here that for a node, the node contains the start execution time, the end execution time and the node time. The start execution time of the node is the start execution time of the instruction, and the end execution time of the node is the end execution time of the instruction. For an edge, the edge only contains the edge time.

[0086] The instruction overlap execution capability of hardware refers to the ability of computer hardware to execute multiple instructions at the same time. It can be expressed by the instructions that the hardware allows to be overlapped and the number of instructions, which is also a pre-set parameter. The instruction overlap execution capability includes the requirements for the number of instructions to be overlapped and the requirements for the types of instructions to be overlapped. Overlapping as many instructions as possible is undoubtedly an efficient use of the hardware computing power.

[0087] The computation graph contains the dependencies between instructions. This dependency is used to plan the execution time of instructions with dependencies. The hardware's instruction overlap execution capability is used to plan the execution time of instructions without dependencies.

[0088] When constructing several constraints to be planned including the execution time of several instructions, the following aspects are included:

[0089] When there is a dependency between two instructions in the computation graph, a constraint to be planned for sequential execution is constructed based on the node and edge times associated with the two instructions.

[0090] When there is no dependency between two instructions in the computation graph, the constraints to be planned for overlapping execution are constructed based on the node timing and edge timing associated with the two instructions and the overlapping execution capabilities of the instructions.

[0091] That is to say, the constraints to be planned include sequentially executed constraints and overlappingly executed constraints. The former is used to represent the sequential execution relationship between instructions, and the latter represents the overlapping execution relationship between instructions.

[0092] When the execution time of a node includes the start execution time and the end execution time, a constraint relationship between the start execution time, the end execution time and the node time consumption of a node can also be constructed, that is, the end execution time of a node is equal to the sum of its start execution time and the node time consumption. This constraint relationship can be expressed by the following equation:

[0093] E A = S A +C A (1)

[0094] Among them, E A represents the end execution time of node A, S A represents the start time of node A, C A is the node time of node A. Similarly, this equality constraint can be constructed for each node.

[0095] for Figure 1 For any two nodes A and B with dependency relationship, where B depends on A, a sequentially executed constraint can be constructed between nodes A and B, that is, the start execution time of node B is greater than or equal to (or not less than) the start execution time of node A and the edge duration of edge AB, which can be expressed by the following inequality:

[0096] S B ≥E A +C AB (2)

[0097] Among them, S B is the start execution time of node B, E A is the start time of node A, C AB is the edge time of edge AB. Similarly, such inequality constraints can be constructed between each set of nodes with constraint relationships.

[0098] When constructing the constraints to be planned for overlapping execution, it is necessary to construct them according to the overlapping execution capability of the hardware. For example, when the overlapping execution capability of the hardware allows two instructions to be executed overlappingly, for two first nodes i and second nodes j that do not have a dependency relationship, such constraints to be planned for overlapping execution can be constructed.

[0099] The start execution time of the second node j is earlier than the end execution time of the first node i; or,

[0100] The start execution time of the first node i is earlier than the end execution time of the second node j.

[0101] And, through a logical variable b to be constrained, the above two constraints can be controlled to be executed alternatively. The plan-to-be-scheduled with overlapping execution can be expressed by the following formula:

[0102] S i -E j <M(1 - b)

[0103] S j -E i < Mb (3)

[0104] Wherein, S i and E i are the start execution time and end execution time of the first node i, and S j and E j are the start execution time and end execution time of the second node j. M is a sufficiently large number, which can also be preset or randomly generated. i and j take values among all the nodes without dependencies in the computational graph G. b takes values between 0 and 1, that is, b can only take 0 or 1, and it is a parameter to be solved. For the two expressions in formula (3), when b takes 1 or 0, only one of these two expressions holds, and the other holds naturally, so the two expressions are simplified to one expression. For example, when b takes 1, the following expression S j -E i <0 takes effect (when the value of E i is larger than the value of S j , that is, S j is earlier than E i ), and the above expression S i -E j <M holds naturally. For any two nodes without dependencies in the computational graph G, the above plan-to-be-scheduled constraints can be constructed.

[0105] By constructing constraints for all the nodes in the computational graph G in the above manner, multiple plan-to-be-scheduled constraints can be obtained.

[0106] Step S230, determine the latest execution time makespan of the instructions in several plan-to-be-scheduled constraints.

[0107] Wherein, the latest execution time makespan can reflect the end execution time of the computational graph G.

[0108] When the execution time of an instruction includes the start execution time and the end execution time of the instruction, the latest execution time makespan can be the latest end execution time of the instruction. The latest end execution time among several instructions is the end execution time of the computation graph G. The optimized execution time obtained by this planning is more reasonable. Among them, makespan = max(E0, E1, ..., E i ), i takes values ​​between 0 and N-1, and N is the number of all instructions in the computation graph G.

[0109] Of course, in practical applications, the latest start execution time of the instruction can also be used as the latest execution time makespan. The value obtained by adding the latest start execution time of several instructions to the node time consumption of the node is the end execution time of the calculation graph G. The node time consumption of the node is a negligible value relative to the total time consumption of the calculation graph G. Therefore, the latest start execution time of several instructions can also reflect the end execution time of the calculation graph G.

[0110] Step S240 , with the goal of minimizing the latest execution time makespan, solves a number of constraints to be planned to obtain optimized execution times of a number of instructions.

[0111] Several constraints to be planned are linear constraints, and minimizing the latest execution time makespan is the linear programming goal. Therefore, the process of solving several constraints to be planned is the process of solving linear programming. Linear programming (LP) is a mathematical optimization method used to solve the maximum or minimum value of a linear objective function under a given set of linear constraints. Linear programming has a wide range of applications in economics, management science, engineering design, production scheduling, logistics planning and other fields, and is particularly suitable for solving problems such as resource allocation, cost minimization, and profit maximization. The solution process can adopt the existing linear programming solution method, which will not be repeated here.

[0112] Solving several constraints to be planned, we get the start execution times corresponding to several nodes, that is, the start execution times corresponding to several instructions. Using these start execution times to execute the computation graph G is a better execution solution.

[0113] When solving the problem, steps S230 and S240 may be solved together. The solution process may be solved by multiple rounds of iterations. In each round of solution, the latest execution time makespan may change and gradually approach a minimum value.

[0114] In the above embodiment, by calculating the dependency between instructions in the graph and the overlapping execution capability of the hardware, it is possible to construct constraints to be planned for sequential execution and overlapping execution of the execution times of several instructions, and by solving these constraints to be planned, it is possible to obtain the optimized execution time of each instruction while minimizing the latest execution time. The embodiment uses linear programming to solve the minimum value of the latest execution time of the calculation graph in the way of operations optimization, which is a means of optimization from a global perspective.

[0115] Figure 3 A schematic diagram of a distributed system implementation scenario provided in an embodiment, including multiple executors such as executor 1, executor 2 and executor 3. Multiple executors jointly execute a computation graph G, wherein only a portion of the computation graph G is shown.

[0116] Distributed systems can be implemented through software systems built on the network, consisting of multiple independent computers or servers (nodes). These computers or servers, as different executors, can also be called computing nodes in the distributed system. Distributed systems communicate and collaborate through the network to jointly complete one or more tasks. Distributed systems are designed to improve the reliability, scalability, performance and fault tolerance of the system. Nodes can be physical machines, virtual machines, containers, etc., and are interconnected through local area networks, wide area networks or the Internet.

[0117] When multiple executors in a distributed system jointly perform calculations of a neural network model, a distributed computing graph can be used for the calculations.

[0118] Among them, the distributed computing graph refers to a structure that deploys a computing graph to multiple computing nodes for parallel or distributed computing. The multiple computing nodes or executors here can be multiple servers, multiple GPUs or CPU cores, etc. Compared with the computing graph, nodes and edges for data communication on multiple devices and different hardware of the same device are added. Collective communication is carried out between multiple executors.

[0119] Collective communication refers to a communication method in which multiple processes or nodes interact as a logical group (collection) in a distributed system.

[0120] In distributed computational graph applications, nodes and edges for data communication between multiple executors are added to the computational graph, which enables multiple executors to use the same computational graph and execute it synchronously to achieve joint computation of neural network models.

[0121] The above is only an example of an application scenario. Based on this application scenario, another embodiment of this specification can be obtained. In this embodiment, the instructions in the calculation graph G have different instruction types. For example, the instructions in the calculation graph G include first-class instructions and second-class instructions.

[0122] based on Figure 2 In an embodiment, the data table inst_profiler mentioned in step 1_1 can be used to store instruction parameters, instruction types and corresponding instruction time consumption. For communication instructions, the instruction parameters of the communication instructions may include data types, data shapes and communication group sizes, etc. Among them, the data type includes precision or half precision; the data shape includes the dimension of the matrix, the dimension of the vector, etc. The communication group size refers to the number of communication devices and / or hardware when executing the communication instruction. The instruction can be uniquely identified based on the instruction parameters.

[0123] For communication instructions, when there are no matching instruction parameters in the data table inst_profiler, the corresponding instruction time consumption is estimated based on the instruction parameters and the hardware parameters when executing the instruction in step 1_5. Specifically, the communication volume of the instruction can be determined by the instruction parameters, and the instruction time consumption can be determined using the communication volume and the communication bandwidth of the hardware. For example, the communication instruction time consumption can be estimated according to the following formula: the communication volume of the instruction / (the communication bandwidth of the corresponding communication group under the current communication volume). The communication bandwidth of the communication group under the current communication volume can be obtained from the hardware system information.

[0124] based on Figure 2 In an embodiment, when executing step 2, i.e., determining the corresponding node time and edge time based on the time consumption of several instructions, the corresponding node time and edge time can be determined based on the time consumption of several instructions and the type of instruction. For example, when the instruction belongs to the first category of instructions, its instruction time is used as the corresponding node time, and the preset small time consumption is used as the edge time consumption of the edge connected to the instruction. When the instruction belongs to the second category of instructions, the instruction is usually split into two child nodes and the edge between the two child nodes, the instruction time of the instruction is used as the edge time consumption of the edge, and the preset small time consumption is used as the node time consumption of the two child nodes.

[0125] In a distributed system scenario, a computation graph is used to execute in a distributed system. The first type of instruction may be a computation instruction, which is used to execute computation operations within a corresponding executor. The second type of instruction may be a communication instruction, which is used to execute communication operations between multiple executors included in a distributed system.

[0126] The node corresponding to the communication instruction includes a sub-node representing the communication start instruction and a sub-node representing the communication end instruction, as well as an edge between the two sub-nodes. Figure 3The communication instruction in includes a child node C representing a communication start instruction, a child node A representing a communication end instruction, and an edge between child nodes A and C. The edge executes the communication end instruction from the communication start instruction.

[0127] Specifically, in step 2, when executing the step of determining corresponding node time consumption and edge time consumption based on a number of instruction times and instruction types, it can be performed in the following manner.

[0128] When a node represents a computing instruction, the instruction time of the computing instruction is used as the node time, and the preset small time is used as the edge time connected to the node;

[0129] When a node represents a communication instruction, the instruction time of the communication instruction is used as the edge time of the edge included in the node, and the preset small time is used as the node time of the child node included in the node.

[0130] by Figure 3 Taking the computation graph in as an example, node B is a computation node. The instruction time of the instruction represented by the node can be used as the node time of node B, and the edge time of the edge connected to node B can be set to the preset small time. The communication instruction includes subnode A and subnode C and the edge between them. Subnode A and subnode C are communication nodes, and edge CA is a communication edge. The instruction time of the communication instruction is used as the edge time of edge CA, and the preset small time is used as the node time of subnode A and subnode C. Subnode A can be referred to as node A, and subnode C can be referred to as node C.

[0131] For based on Figure 2 In step S220 of the embodiment, when the instruction includes a computing instruction and a communication instruction, and the hardware's instruction overlapping execution capability is related to the overlap of different types of instructions, for example, only computing instructions and communication instructions are allowed to overlap at the same time, for two nodes of the same type that do not have a dependency relationship in the computation graph G, the constraints to be planned for overlapping execution constructed include:

[0132] The start execution time of one node is later than the end execution time of another node.

[0133] That is to say, for two nodes in the computation graph G that do not have a dependency relationship and belong to computation instructions, it is required that the start execution time of one node is later than the end execution time of the other node, that is, the two computation instruction nodes do not overlap. At the same time, for two nodes in the computation graph G that do not have a dependency relationship and belong to communication instructions, it is also required that the start execution time of one node is later than the end execution time of the other node, that is, the two communication instruction nodes do not overlap. In this way, when two nodes belonging to the same instruction type and without a dependency relationship cannot overlap, what remains is to allow different instruction types to overlap at the same time, that is, to allow computation instructions and communication instructions to overlap.

[0134] For example, two nodes of the same instruction type that do not have a dependency relationship in the computation graph G are the first node i and the second node j. The constraints to be planned for the overlapping execution of these two nodes specifically include:

[0135] The start execution time of the first node i is later than the end execution time of the second node j;

[0136] The start execution time of the second node j is later than the end execution time of the first node i.

[0137] Moreover, the logical variable b to be constrained can be used to control whether one of the above two constraints is true or not. The overlapping execution of the pending plan can be expressed by the following formula:

[0138] S i -E j ≥-M(1-b)

[0139] S j -E i ≥ -Mb (4)

[0140] Among them, S i and E i is the start and end execution time of the first node i, S j and E j is the start and end execution time of the second node j. M is a sufficiently large number, which can be preset or randomly generated. i and j take values ​​in all nodes with no dependencies and the same instruction type contained in the computation graph G. b takes values ​​between 0 and 1, that is, b can only take 0 or 1, and is the parameter to be solved. For the two equations in equation (4), when b takes 1 or 0, only one of the two equations is valid, and the other is naturally valid, so the two equations are simplified to one equation. For example, when b takes 1, the equation S above j -E i ≥0 works (E i The value is higher than S j The value of is small, that is, E i Earlier than S j ), the formula S below i -E j ≥-M naturally holds true.

[0141] For the two nodes corresponding to all the computing instructions without dependencies in the computing graph G, the above constraints to be planned can be constructed using formula (4). For the two nodes corresponding to all the communication instructions without dependencies in the computing graph G, the above constraints to be planned can be constructed using formula (4).

[0142] Below Figure 3Taking some nodes in as an example, the constraints to be planned constructed for communication instructions without dependencies and the basis for setting the communication instructions to include child nodes are explained.

[0143] for Figure 3 Nodes A, B and C in FIG. 1 are nodes A, B and C. Node C and node A are respectively a start communication instruction and an end communication instruction in a communication instruction. Node B is a computing node. The meanings of various parameters related to nodes A, B and C are shown in Table 1.

[0144]

[0145] For according to Figure 3 At the middle node A, the following equation exists: A =S A +C A . Because C A It is set to the preset small time consumption which can be ignored, so it can be approximated that E A =S A For node C, there is also E C =S C That is to say, for a communication node, its execution start time is equal to its execution end time, and the time consumption of communication instructions is mainly concentrated on the edge between two communication nodes.

[0146] Since the start and end execution times of the nodes at both ends of the communication edge can be approximated to be the same time, the execution time of the communication node is the execution time of the communication edge. Therefore, the constraint that the same type of communication nodes cannot be executed overlappingly in the instruction overlapping execution capability is equivalent to the same type of communication edges cannot be executed overlappingly. Setting the communication instruction to consist of two nodes and the edge between them does not require special settings for the communication instruction, which can simplify the form of the constraints to be planned and simplify the solution process.

[0147] In addition, for nodes A and B that have a dependency relationship, the following relationship can be constructed: B ≥E A +C AB That is, the start execution time of node B should be later than the end execution time of node A. AB can be ignored, so there exists S B ≥E A relationship.

[0148] In this embodiment, Figure 2 The same steps as in the embodiment will not be repeated here. Figure 2 The description of the embodiments can be carried out.

[0149] In this embodiment, the constraints to be planned constructed in the above manner are solved to obtain the optimized execution time of the instruction that meets the overlapping execution capability requirements of the instruction. In this way, through the above linear programming, the embodiment realizes the use of operations research methods to allocate and calculate the entire distributed computing graph, which is transparent to the upper-level applications and parallel methods, thereby achieving a more optimized communication-computing overlap ratio, reducing the communication cost of the distributed system as much as possible, and achieving a more optimized efficiency of the distributed system.

[0150] Similarly, when the first category of instructions and the second category of instructions are other types of instructions, the constraints to be planned may be modified accordingly.

[0151] Figure 4 A flowchart of a method for executing a computational graph in a distributed system provided in an embodiment. The computational graph includes nodes representing instructions and edges representing dependencies between instructions. Instructions include communication instructions and computational instructions. The method is executed by an execution device. The distributed system includes multiple executors such as executor 1, executor 2, ..., executor i. The execution device can be a device of any one of the multiple executors. The method includes the following steps.

[0152] Step S410, obtaining the optimized execution time of several instructions in the computation graph G, wherein the optimized execution time of several instructions is adopted Figure 2 The method of the embodiment is determined.

[0153] Step S420: According to the above-mentioned optimized execution time, the calculation graph G is jointly executed by multiple executors included in the distributed system.

[0154] In this embodiment, multiple executors in the distributed system execute communication instructions at the same time and computing instructions at the same time according to the optimized execution times of multiple instructions, thereby realizing the joint execution of distributed computing graphs by multiple parties and obtaining more optimized system efficiency.

[0155] In this specification, the word "first" in the terms such as the first node, and the corresponding "second" (if any) in the text, etc., are merely for the convenience of distinction and description and do not have any limiting meaning.

[0156] In this specification, computing devices and execution devices can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities.

[0157] The foregoing describes certain embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0158] Figure 5 A schematic block diagram of an apparatus for optimizing instruction execution timing using a computation graph is provided in an embodiment. The computation graph includes nodes representing instructions and edges representing dependencies between instructions. Figure 2 The method embodiment shown corresponds to the embodiment. The device 500 includes: a time-consuming determination module 510, a constraint construction module 520, a moment determination module 530 and a constraint solving module 540. Among them, the time-consuming determination module 510 is configured to determine the node time and edge time of the calculation graph during execution. The constraint construction module 520 is configured to construct a plurality of constraints to be planned including the execution time of a plurality of instructions based on the dependency relationship between the instructions in the calculation graph, the node time and edge time, and the instruction overlapping execution capability of the hardware. The moment determination module 530 is configured to determine the latest execution time of the instructions in the plurality of constraints to be planned. The constraint solving module 540 is configured to solve the plurality of constraints to be planned with the goal of minimizing the latest execution time to obtain the optimized execution time of the plurality of instructions.

[0159] In one embodiment, the time-consuming determination module 510 includes: an instruction time-consuming determination submodule 511 and a node time-consuming determination submodule 512. The instruction time-consuming determination submodule 511 is configured to determine the instruction time when the instruction in the computation graph is executed. The node time-consuming determination submodule 512 is configured to determine the corresponding node time and edge time based on a number of instruction times.

[0160] In one embodiment, the instruction time determination submodule 511 includes: an acquisition unit, a determination unit, a matching unit, an assignment unit and an estimation unit (not shown in the figure). Among them, the acquisition unit is configured to acquire a data table, and the data table is used to store instruction parameters and corresponding instruction time. The determination unit is configured to determine the instruction parameters of the instructions in the calculation graph. The matching unit is configured to match the instruction parameters of the instructions in the calculation graph with the instruction parameters in the data table respectively. The assignment unit is configured to, when the match is successful, determine the instruction time corresponding to the successfully matched instruction parameters in the data table as the instruction time of the instruction in the calculation graph. The estimation unit is configured to, when the match fails, estimate the corresponding instruction time based on the instruction parameters of the instructions in the calculation graph and the hardware parameters when executing the instructions.

[0161] In one embodiment, the instructions include computing instructions and / or communication instructions, and the node corresponding to the communication instruction includes a sub-node representing a communication start instruction and a sub-node representing a communication end instruction, and an edge between the two sub-nodes; the node time determination sub-module 512 is specifically configured as: when the node represents a computing instruction, the instruction time of the computing instruction is used as the node time, and the preset small time is used as the edge time connected to the node; when the node represents a communication instruction, the instruction time of the communication instruction is used as the edge time of the edge contained in the node, and the preset small time is used as the node time of the sub-node contained in the node.

[0162] In one embodiment, the constraint construction module 520 includes: a sequential constraint submodule 521 and an overlapping constraint submodule 522. The sequential constraint submodule 521 is configured to construct a constraint to be planned for sequential execution based on the node time and edge time associated with the two instructions when there is a dependency relationship between the two instructions in the computation graph. The overlapping constraint submodule 522 is configured to construct a constraint to be planned for overlapping execution based on the node time and edge time associated with the two instructions and the overlapping execution capability of the instructions when there is no dependency relationship between the two instructions in the computation graph.

[0163] In one implementation, the instructions include computing instructions and communication instructions; the instruction overlapping execution capability includes allowing computing instructions and communication instructions to be executed overlappingly; the overlapping constraint submodule 522 is specifically configured as follows:

[0164] For two nodes of the same type without a dependency relationship in the computation graph, the constraints to be planned for overlapping execution constructed include: a start execution time of one node is later than an end execution time of another node.

[0165] In one implementation, the two nodes are a first node and a second node; and the constraints to be planned for overlapping execution of the two nodes specifically include:

[0166] Through the logical variables to be constrained, control the following constraints to be satisfied:

[0167] The start execution time of the first node is later than the end execution time of the second node;

[0168] The start execution time of the second node is later than the end execution time of the first node.

[0169] In one embodiment, the computation graph is used to be executed in a distributed system, the communication instructions are used to perform communication operations between multiple executors included in the distributed system, and the computation instructions are used to perform computation operations within the executors.

[0170] In one implementation, the execution time of the instruction includes the start execution time and the end execution time of the instruction, and the latest execution time is the latest end execution time of the instruction.

[0171] Figure 6 A flowchart of a computing graph execution device in a distributed system provided in an embodiment. The computing graph includes nodes representing instructions and edges representing dependencies between instructions. Figure 4 The method embodiment shown corresponds to the embodiment shown. The device 600 includes:

[0172] The sorting acquisition submodule 610 is configured to acquire the optimized execution time of several instructions in the computation graph; the optimized execution time of the several instructions is adopted Figure 2 The method described in the embodiment is determined;

[0173] The computation graph execution submodule 620 is configured to jointly execute the computation graph through multiple executors included in the distributed system according to the optimized execution time.

[0174] The above-mentioned device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0175] The present specification also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute Figures 1 to 4 Any of the methods described above.

[0176] The embodiment of the present specification also provides a computing device, including a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, Figures 1 to 4 Any of the methods described above.

[0177] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0178] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0179] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for optimizing instruction execution time using a computation graph, wherein the computation graph includes nodes representing instructions and edges representing dependencies between instructions; the method includes: Determine the node time and edge time of the computation graph during execution; Based on the dependencies between the instructions in the computation graph, the node time consumption and the edge time consumption, and the instruction overlapping execution capability of the hardware, construct a plurality of constraints to be planned including the execution times of the plurality of instructions; Determine the latest execution time of the instructions in the plurality of constraints to be planned; With the goal of minimizing the latest execution time, the plurality of constraints to be planned are solved to obtain optimized execution times of the plurality of instructions.

2. According to the method of claim 1, the step of determining the node time and edge time of the computation graph during execution comprises: Determine the instruction time consumed when executing the instructions in the computation graph; Based on the times of several instructions, the corresponding node times and edge times are determined.

3. According to the method of claim 2, the step of determining the instruction time consumed when executing the instructions in the computation graph comprises: Obtain a data table, wherein the data table is used to store instruction parameters and corresponding instruction time consumption; Determining instruction parameters of instructions in the computation graph; Matching the instruction parameters of the instructions in the computation graph with the instruction parameters in the data table respectively; When the match is successful, the instruction time corresponding to the successfully matched instruction parameter in the data table is determined as the instruction time of the instruction in the calculation graph; When the match fails, the corresponding instruction time consumption is estimated based on the instruction parameters of the instruction in the computation graph and the hardware parameters when the instruction is executed.

4. The method according to claim 2, wherein the instruction comprises a computing instruction and / or a communication instruction, and the node corresponding to the communication instruction comprises a sub-node representing a communication start instruction and a sub-node representing a communication end instruction and an edge between the two sub-nodes; The step of determining corresponding node time consumption and edge time consumption based on the time consumption of a plurality of instructions includes: When a node represents a computing instruction, the instruction time of the computing instruction is used as the node time, and the preset small time is used as the edge time connected to the node; When a node represents a communication instruction, the instruction time of the communication instruction is used as the edge time of the edge included in the node, and the preset small time is used as the node time of the child node included in the node.

5. The method according to claim 1, wherein the step of constructing a plurality of constraints to be planned including the execution times of a plurality of instructions comprises: When there is a dependency relationship between two instructions in the computation graph, constructing a constraint to be planned for sequential execution based on node time consumption and edge time consumption associated with the two instructions; When there is no dependency relationship between two instructions in the computation graph, constraints to be planned for overlapping execution are constructed based on node timing and edge timing associated with the two instructions and overlapping execution capabilities of the instructions.

6. The method according to claim 5, wherein the instructions include computing instructions and communication instructions; the instruction overlapping execution capability includes allowing computing instructions and communication instructions to be executed overlappingly; The step of constructing constraints to be planned for overlapping execution includes: For two nodes of the same type without a dependency relationship in the computation graph, the constraints to be planned for overlapping execution constructed include: a start execution time of one node is later than an end execution time of another node.

7. The method according to claim 6, wherein the two nodes are a first node and a second node; and the constraints to be planned for overlapping execution of the two nodes specifically include: Through the logical variables to be constrained, control the following constraints to be satisfied: The start execution time of the first node is later than the end execution time of the second node; The start execution time of the second node is later than the end execution time of the first node.

8. According to the method according to claim 6, the computation graph is used to be executed in a distributed system, the communication instructions are used to perform communication operations between multiple executors included in the distributed system, and the computing instructions are used to perform computing operations within the executors.

9. The method according to claim 1, wherein the execution time of the instruction includes the start execution time and the end execution time of the instruction, and the latest execution time is the latest end execution time of the instruction.

10. A method for executing a computation graph in a distributed system, the computation graph comprising nodes representing instructions and edges representing dependencies between instructions; the method comprising: Obtaining optimized execution times of several instructions in the computation graph; the optimized execution times of the several instructions are determined using the method of claim 1; According to the optimized execution time, the calculation graph is jointly executed by multiple executors included in the distributed system.

11. A device for optimizing instruction execution time using a computation graph, wherein the computation graph includes nodes representing instructions and edges representing dependencies between instructions; the device includes: A time-consuming determination module, configured to determine the node time-consuming and edge time-consuming of the computation graph during execution; A constraint construction module is configured to construct a plurality of constraints to be planned including execution times of a plurality of instructions based on dependencies between instructions in the computation graph, the node time consumption and the edge time consumption, and the instruction overlapping execution capability of the hardware; A time determination module, configured to determine the latest execution time of the instructions in the plurality of constraints to be planned; The constraint solving module is configured to solve the plurality of constraints to be planned with the goal of minimizing the latest execution time, and obtain optimized execution times of the plurality of instructions.

12. A computing graph execution device in a distributed system, the computing graph comprising nodes representing instructions and edges representing dependencies between instructions; the device comprising: A sorting acquisition submodule, configured to acquire optimized execution times of a plurality of instructions in a computation graph; the optimized execution times of the plurality of instructions are determined using the method of claim 1; The computation graph execution submodule is configured to jointly execute the computation graph through multiple executors included in the distributed system according to the optimized execution time.

13. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 10.

14. A computing device, comprising a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 10 is implemented.

Images