Parallel execution method and device for computational graph

Abstract

A method and device for parallel execution of a computational graph are provided. The method comprises: constructing a computational graph corresponding to a computational task, wherein the computational graph comprises computing nodes, data nodes, and edges connecting computing nodes and data nodes; determining a parallel execution mode of the computational graph according to the computational task, wherein the parallel execution mode is one of the following modes or a combination of at least two of the following modes: parallel execution within a computing node, parallel execution between computing nodes, and parallel execution between computational graphs; and executing the computational graph in multiple threads according to the determined parallel execution mode, wherein parallel execution within a computing node means that at least one individual computing node in the computational graph is executed in parallel through different threads; parallel execution between computing nodes means that different computing nodes in the computational graph are executed in parallel through different threads; and parallel execution between computational graphs means that the computational graph and at least one copy thereof are executed in parallel through different threads.

Description

[0001] This application is a divisional application of a patent application with application date of December 28, 2018, application number 201811618988.2, and titled “Parallel execution method and device for computational graph”. Technical Field

[0002] The present invention relates to the field of parallel data processing, and more specifically, to a method and device for parallel execution of a computational graph. Background Art

[0003] A computational graph is a general representation method for computational processes, which is widely used on various data processing platforms. Its representation method is easy to understand and highly flexible, and complex logic can be implemented by combining simple operations. As an example, computational graphs can be applied to neural networks, which are machine learning models that use one or more model layers to generate outputs, such as one or more classifications, for received inputs. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input of the next layer in the network (i.e., the next hidden layer or output layer of the network). Each layer of the network generates output from the received input based on the current value of the corresponding parameter set of that layer.

[0004] The existing definition of computational graphs includes nodes and directed edges. Taking the application of neural networks as an example, each node of the computational graph represents the corresponding operation performed by the neural network, which is an operation that determines the output of the neural network based on the input of the neural network. Each directed edge of the computational graph connects the previous node to the next node, indicating that the output of the previous node is used as the input of the next node.

[0005] Figure 2A and Figure 2B The following are schematic diagrams showing situations where there are limitations in the parallel execution of conventional computation graphs.

[0006] When a computation graph is executed, the execution of the existing computation graph has two levels of parallelism, one is the parallelism between nodes, and the other is the parallelism within the node.

[0007] The degree of parallelism between nodes depends on the dependencies between nodes in the computation graph. If there are many dependencies, the degree of parallelism is limited. Figure 2A , Figure 2A There are cases where two nodes share outputs (e.g., share output locations) in the computation graph shown in Figure 2A In the calculation graph with node-shared output shown in the figure, the outputs of node A and node B are both inputs of node C, but if their outputs are the same data, a write conflict will occur. Therefore, when executing this calculation graph, node A and node B cannot be calculated in parallel. Therefore, Figure 2AThe nodes in the computational graph shown in the figure cannot be run in parallel at all and can only be executed serially. At this time, the thread communication overhead caused by the parallelism between nodes will slow down the overall execution speed.

[0008] Intra-node parallelism requires a synchronization operation on the parallel threads of the node before and after the node starts. The overhead of the synchronization operation increases with the degree of parallelism. Therefore, intra-node parallelism depends on the relationship between the computation of the node itself and the overhead of the synchronization operation. It is only advantageous when the computation of the node itself is much greater than the overhead of the synchronization operation. For lighter computing nodes, intra-node parallelism will be slower, so the degree of parallelism of intra-node parallelism will also be limited. For example, Figure 2B The computational graph shown in the figure has a large number of lightweight nodes, including node A, node B...node N, but the calculation of all nodes depends on the previous node in sequence. In this case, both intra-node parallelism and inter-node parallelism cannot play a role.

[0009] In summary, the parallelism between nodes and the parallelism within nodes used in the parallel execution of computation graphs in the prior art are both limited. Summary of the invention

[0010] The purpose of the present invention is to provide a device for a method for parallel execution of a computational graph to solve the problem that the existing parallel execution of computational graphs has limitations.

[0011] One aspect of the present invention provides a method for parallel execution of a computational graph, characterized in that it includes: constructing a computational graph corresponding to a computational task, wherein the computational graph includes operation nodes, data nodes, and edges connecting the operation nodes and the data nodes; determining a parallel execution mode of the computational graph according to the computational task, wherein the parallel execution mode is one of the following modes or a combination of at least two of the following modes: parallel execution within an operation node, parallel execution between operation nodes, and parallel execution between computational graphs; and executing the computational graph in multiple threads according to the determined parallel execution mode, wherein parallel execution within an operation node refers to at least one individual operation node in the computational graph being executed in parallel through different threads; parallel execution between operation nodes refers to different operation nodes in the computational graph being executed in parallel through different threads; and parallel execution between computational graphs refers to the computational graph and at least one copy thereof being executed in parallel through different threads.

[0012] Optionally, the step of determining the parallel execution mode of the computational graph based on the computational task may include: determining the parallel execution mode of the computational graph based on the time cost of parallel execution of the computational task within a computing node, parallel execution between computing nodes, and / or parallel execution between computational graphs.

[0013] Optionally, the time cost may be determined according to the synchronous operation of the computing task.

[0014] Optionally, the step of determining the parallel execution mode of the computational graph according to the type of the computational task includes: when the computational task involves asynchronous computation, determining the parallel execution mode between computational graphs as the parallel execution mode of the computational graph; and / or, when the computational task involves synchronous computation and the computational graph includes synchronous computational nodes, determining the combination of parallel execution between computational nodes and parallel execution within computational nodes as the parallel execution mode of the computational graph; and / or, when the computational task involves synchronous computation and the computational graph does not include synchronous computational nodes, determining the combination of parallel execution between computational nodes and parallel execution between computational graphs as the parallel execution mode of the computational graph.

[0015] Optionally, the asynchronous computing may refer to asynchronous training of a neural network; and / or, the synchronous computing may refer to synchronous training of a neural network; and / or, the synchronous computing node may refer to a batch normalization node.

[0016] Optionally, the storage location of the operation data of the data node declared in the calculation graph is the relative index of the operation data in the memory space, and the step of executing the calculation graph in multiple threads according to the determined parallel execution method may include: allocating corresponding memory space to each thread, and executing the calculation graph in parallel in the multiple threads according to the starting addresses of the memory spaces of each of the multiple threads and the relative index.

[0017] Optionally, the step of executing the computational graph in parallel in the multiple threads according to the starting addresses of the memory spaces of the multiple threads and the relative index may include: using each thread to determine the memory address of the operation data of the at least one data node included in the corresponding computational graph according to the starting address of the corresponding memory space and the relative index; and using the multiple threads to execute their respective computational graphs according to their respective determined memory addresses.

[0018] Another aspect of the present invention provides a device for executing a computation graph in parallel, characterized in that it includes: a computation graph construction unit, which constructs a computation graph corresponding to a computation task, wherein the computation graph includes operation nodes, data nodes, and edges connecting the operation nodes and the data nodes; a parallel mode determination unit, which determines the parallel execution mode of the computation graph according to the computation task, wherein the parallel execution mode is one of the following modes or a combination of at least two of the following modes: parallel execution within a computation node, parallel execution between computation nodes, and parallel execution between computation graphs; and an execution unit, which executes the computation graph in multiple threads according to the determined parallel execution mode, wherein parallel execution within a computation node refers to at least one individual computation node in the computation graph being executed in parallel through different threads; parallel execution between computation nodes refers to different computation nodes in the computation graph being executed in parallel through different threads; and parallel execution between computation graphs refers to the computation graph and at least one copy thereof being executed in parallel through different threads.

[0019] Optionally, the parallel mode determination unit may determine the parallel execution mode of the computation graph according to the time cost of parallel execution of the computation task within the computing node, parallel execution between computing nodes, and / or parallel execution between computation graphs.

[0020] Optionally, the parallel mode determination unit may determine the time cost according to the synchronous operation of the computing task.

[0021] Optionally, the parallel mode determination unit may determine the parallel execution mode between computation graphs as the parallel execution mode of the computation graph when the computation task involves asynchronous computation; and / or may determine the combination of parallel execution between operation nodes and parallel execution within operation nodes as the parallel execution mode of the computation graph when the computation task involves synchronous computation and the computation graph includes synchronous operation nodes; and / or may determine the combination of parallel execution between operation nodes and parallel execution between computation graphs as the parallel execution mode of the computation graph when the computation task involves synchronous computation and the computation graph does not include synchronous operation nodes.

[0022] Optionally, the asynchronous computing may refer to asynchronous training of a neural network; and / or, the synchronous computing may refer to synchronous training of a neural network; and / or, the synchronous computing node may refer to a batch normalization node.

[0023] Optionally, the computation graph construction unit declares in the computation graph that the storage location of the operation data of the data node is the relative index of the operation data in the memory space, and allocates corresponding memory space to each thread; and the execution unit executes the computation graph in parallel in the multiple threads according to the starting addresses of the memory spaces of the multiple threads and the relative index.

[0024] Optionally, the execution unit can utilize each thread to determine the memory address of the operation data of the at least one data node included in the corresponding computation graph according to the starting address of the corresponding memory space and the relative index; and utilize the multiple threads to execute their respective computation graphs according to their respective determined memory addresses.

[0025] Another aspect of the present invention provides a system comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to execute the parallel execution method of the computational graph as described above.

[0026] Another aspect of the present invention provides a computer-readable storage medium storing instructions, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the parallel execution method of the computational graph as described above.

[0027] According to one or more aspects of the present invention, the computational graph used in the parallel execution method of the computational graph solves the problem that shared outputs cannot be executed in parallel by adding data nodes; the computational graph is executed by multiple threads, so that the parallelism between the computational graphs is increased on the basis of the original parallelism between nodes and within nodes; and further, it is also possible to consider determining the parallel mode of the computational graph according to the computing task, thereby reducing the overall overhead of the computational graph (for example, computing resources and / or time cost).

[0028] Additional aspects and / or advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description in conjunction with the accompanying drawings, in which:

[0030] Figure 1 is a block diagram illustrating a device for executing a computation graph in parallel according to an embodiment of the present invention;

[0031] Figure 2A and Figure 2B are schematic diagrams respectively showing situations where there are limitations in the parallel execution of existing computation graphs;

[0032] Figure 3 is a flow chart illustrating a method for parallel execution of a computation graph according to an embodiment of the present invention;

[0033] Figure 4 is a schematic diagram illustrating a computation graph including data nodes according to an embodiment of the present invention;

[0034] Figure 5 is a schematic diagram showing a computation graph including data nodes and thread replicas according to an embodiment of the present invention;

[0035] Figure 6 , Figure 7 and Figure 8 is a diagram showing the relationship between the computation graph and the memory space of the present invention;

[0036] Fig. 9 is a flowchart illustrating a method for performing inter-computational graph parallelism according to an embodiment of the present invention;

[0037] Fig.10 It is a flowchart showing the steps of executing at least one operation included in the computation graph according to the starting addresses and relative indexes of the memory spaces of the respective first threads according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0039] Figure 1 is a block diagram showing a device for executing a computation graph in parallel according to an embodiment of the present invention. Figure 1 As shown, the device for executing a computation graph in parallel according to an embodiment of the present invention includes a computation graph construction unit 100, a parallel mode determination unit 200, and an execution unit 300.

[0040] Specifically, the computation graph construction unit 100 constructs a computation graph corresponding to a computation task, wherein the computation graph includes operation nodes, data nodes, and edges connecting the operation nodes and the data nodes.

[0041] As an example, at least one operation included in the computation graph may also be referred to as a computing operation. As an example, the at least one operation involves at least one computing operation in a machine learning algorithm. The corresponding operation data may include input data and output data of the operation.

[0042] When a computation graph is declared, the memory space of the computation graph's operation data is represented by relative indexes, rather than direct indexes (i.e., actual indexes). That is, the storage location of the operation data of a declared data node in the computation graph is the relative index of the operation data in the memory space.

[0043] In the computer field, as an example, the actual index of the memory space can be represented by 8 bytes of data, and the memory space of each operation data of the calculation graph according to an embodiment of the present invention is represented by a relative index. The maximum value of the relative index can be set to the total number of operation data of at least one operation included in the calculation graph (i.e., the number of operation data). Usually, this total number is small and can be represented by 4 or 2 bytes of data, thereby reducing the memory size occupied by the calculation graph.

[0044] In addition, the computation graph construction unit 100 is also used to allocate corresponding memory space to each thread.

[0045] As an example, when the memory of the operation data of the computation graph is uniformly managed by the memory management module, the computation graph construction unit 100 can use the memory management module to allocate corresponding memory space to each thread.

[0046] As an example, the computation graph construction unit 100 may allocate corresponding memory space to each thread according to the total amount of operation data of at least one operation included in the computation graph (i.e., the number of operation data). The computation graph construction unit 100 is used to copy the computation graph for each thread. The steps of creating multiple threads by the computation graph construction unit 100 will be described later. Figures 6 to 10 Describe in detail.

[0047] The parallel mode determination unit 200 determines the parallel execution mode of the computation graph according to the computation task, wherein the parallel execution mode is one of the following modes or a combination of at least two of the following modes: parallel execution within a computing node, parallel execution between computing nodes, and parallel execution between computation graphs.

[0048] Specifically, parallel execution within an operation node may refer to the parallel execution of at least one individual operation node in the computation graph through different threads; parallel execution between operation nodes may refer to the parallel execution of different operation nodes in the computation graph through different threads; and parallel execution between computation graphs may refer to the parallel execution of the computation graph and at least one copy of it through different threads.

[0049] The execution unit 300 executes the computation graph in the multiple threads according to the determined parallel execution mode. As an example, when the computation graph construction unit 100 allocates a corresponding memory space to each thread, the execution unit 300 executes the computation graph in parallel in the multiple threads according to the starting addresses of the memory spaces of the multiple threads and the relative indexes.

[0050] Figure 3 Detailed description is a flowchart illustrating a method for executing a computation graph in parallel according to an embodiment of the present invention.

[0051] In step S1, the computation graph construction unit 100 constructs a computation graph corresponding to the computation task, wherein the computation graph includes operation nodes, data nodes, and edges connecting the operation nodes and the data nodes.

[0052] In step S2, the parallel mode determination unit 200 determines the parallel execution mode of the computation graph according to the computation task, wherein the parallel execution mode is one of the following modes or a combination of at least two of the following modes: parallel execution within a computing node, parallel execution between computing nodes, and parallel execution between computation graphs.

[0053] In step S3, the execution unit 300 executes the computation graph in multiple threads according to the determined parallel execution mode.

[0054] Among them, parallel execution within a computing node means that at least one individual computing node in the computing graph is executed in parallel through different threads; parallel execution between computing nodes means that different computing nodes in the computing graph are executed in parallel through different threads; and parallel execution between computing graphs means that the computing graph and at least one of its copies are executed in parallel through different threads.

[0055] Figure 4 is a schematic diagram illustrating a computation graph including data nodes according to an embodiment of the present invention.

[0056] Reference Figure 4 The computing nodes (represented by “rectangles”) represent the operations performed by the neural network, the data nodes (represented by “circles”) represent the data input and output of the neural network, and the directed edges (represented by “arrows”) only connect computing nodes and data nodes, indicating that a certain data is the input or output of the corresponding calculation.

[0057] Since data is also a node in the computational graph, such as Figure 4 As shown, we can clearly know that the outputs of node A and node B point to the same data (such as Figure 4 (a) shows the data D1), or execute different data (such as Figure 4 (b) shows data D1 and data D2). Therefore, different strategies are adopted according to different situations. If different data (such as Figure 4 (b)), then node A and node B can still be parallel, otherwise, they cannot be parallel.

[0058] Figure 5 is a schematic diagram showing a computation graph including data nodes and thread replicas according to an embodiment of the present invention.

[0059] The thread copy of the computation graph means that the structure of the original computation graph is copied into multiple copies and submitted to other threads for execution. Each copy is called a computation graph thread copy. Figure 5As shown in , the computation graph includes replica G1, replica G2 and replica G3. Different thread replicas can run independently and in parallel. After the computation graph is defined, the computation graph construction unit 100 can create several thread replicas. The steps of executing the computation graph are: first divide the input data into several blocks, and then submit each block to the corresponding thread replica (for example, replica G1, replica G2 and replica G3). After each thread executes all nodes, thread synchronization is performed again. In each replica, parallel execution between nodes and within nodes can still be performed according to the computing tasks. The relationship between the computation graph and the memory space, as well as the method and steps for executing computation graphs in parallel by multiple threads will be referred to. Figures 6 to 10 Describe in detail.

[0060] Compared with the two original degrees of parallelism, computation graph parallelism (i.e., computation graph thread replica parallelism) does not require each node to synchronize threads, nor is it limited by the dependencies between nodes. Figure 2B In the extreme case shown in , it is almost impossible to achieve parallelism within and between computing nodes. Figure 5 After the computation graph is shown, the execution unit 300 can use the thread copies of the computation graph to execute parallelism between the computation graphs, thereby improving the degree of parallelism.

[0061] In addition, since one level of parallelism is added, that is, parallelism between computational graphs is added, it means that users need to configure more parameters. How to configure the ratio of the three parallelisms, namely, parallelism within a computing node, parallelism between computing nodes, and parallelism between computational graphs, is a challenge for users. Therefore, as a preferred method, the present invention also proposes to select different executors according to different computing tasks and automatically configure parallel strategies. Different executors have different configurations for these three parallelisms (for example, turning on / off the corresponding parallelism level).

[0062] Here, the parallel execution mode of the computation graph can be determined based on the time cost of parallel execution of the computation task within the computing node, parallel execution between computing nodes, and / or parallel execution between computation graphs. In other words, the parallel execution mode (also referred to as parallel strategy) is determined by measuring the time cost corresponding to various parallel modes in the specific computation task. For example, the time cost can be determined based on the synchronous operation of the computation task. The time cost is determined by considering the synchronous operation involved in the computation task.

[0063] The following describes a method for determining a parallel strategy in exemplary synchronous computing and asynchronous computing, but it should be understood that exemplary embodiments of the present invention are not limited thereto:

[0064] 1. When the computing task involves asynchronous computing, the parallel execution mode between computing graphs is determined as the parallel execution mode of the computing graphs. As an example, when the asynchronous computing is asynchronous training of a neural network, only thread copies need to be parallel, and the parallel execution mode between computing graphs is determined as the parallel execution mode of the computing graphs;

[0065] 2. When the computing task involves synchronous computing and the computing graph includes synchronous computing nodes, a combination of parallel execution between computing nodes and parallel execution within computing nodes is determined as the parallel execution mode of the computing graph. As an example, when the synchronous computing refers to synchronous training of a neural network; and / or the synchronous computing node refers to a batch normalization node, such computing nodes need to synchronize all thread copies, so the more thread copies there are, the greater the overhead (for example, computing resources and / or time cost). In this case, parallelism between computing graphs can be turned off, and a combination of parallel execution between computing nodes and parallel execution within computing nodes is determined as the parallel execution mode of the computing graph;

[0066] 3. When the computing task involves synchronous computing and the computing graph does not include synchronous computing nodes, the combination of parallel execution between computing nodes and parallel execution between computing graphs is determined as the parallel execution mode of the computing graph. As an example, when the synchronous computing refers to synchronous training of a neural network, if there is no synchronous computing node in the computing graph, the intra-node parallelism should be turned off to avoid the thread synchronization overhead caused by the intra-node parallelism, and therefore the combination of parallel execution between computing nodes and parallel execution between computing graphs is determined as the parallel execution mode of the computing graph.

[0067] The following will be combined Figures 6 to 8 Describe the relationship between the computation graph and the memory space in the prior art and the difference between the present invention.

[0068] Figure 6 It is a diagram showing the relationship between the existing computation graph and the memory space. Figure 7 and Figure 8 is a diagram showing the relationship between the computation graph and the memory space of the present invention. Figures 6 to 8 In the figure, the upper part represents the actual index of the memory space, and the lower part represents the computational graph, where the boxes represent operations and the circles represent the actual index or relative index of the operation data in the memory space.

[0069] like Figure 6As shown in the figure, assuming that there are four memory segments to be operated in the total memory space, whose actual indexes are 0 to 3, the output data of operation A is written to the memory space with the actual index of 1, and the input of operation B is read from the memory space with the actual index of 1. When the computation graph is copied to achieve multi-threaded parallel execution, the output data of operation A and the input data of operation B of the copied computation graph still point to the memory space with the actual index of 1. Therefore, in order to reallocate the memory space used for the copied computation graph, the original corresponding relationship will have to be found, which is difficult to implement and has high complexity.

[0070] like Figure 7 and Figure 8 As shown, in the embodiment of the present invention, the relative index of the output data of operation A and the input data of operation B in the computation graph is declared to be 1 in the memory space. When the computation graph construction unit 100 creates the first thread 0 and the first thread 1, the memory space with actual indexes of 0 to 1 and the memory space with actual indexes of 2 to 3 are allocated to the first thread 0 and the second thread 1 respectively. When executing the computation graph copied for the first thread 0, the memory space of the output data of operation A and the input data of operation B will be determined as the memory space with actual index 0; when executing the computation graph copied for the first thread 1, the memory space of the output data of operation A and the input data of operation B will be determined as the memory space with actual index 3, thereby separating the memory space of the data processed by the computation graphs of the first thread 0 and the first thread 1 to avoid execution errors.

[0071] return Figure 1 As an example, the execution unit 300 may use each first thread to determine the memory address of the operation data of at least one operation included in the corresponding calculation graph according to the start address of the corresponding memory space and the relative index; and use multiple first threads to execute at least one operation included in their respective calculation graphs according to the memory addresses determined by each. That is, in this example, the first thread itself executes the operations of each calculation graph.

[0072] In another embodiment, due to the efficiency problem of parallel processing, for example, in some scenarios, the method of data parallel processing will be limited, such as each time a part of data is processed, the results need to be synchronized between the threads (as an example, this synchronization process is very common in the training process of the machine learning model). At this time, if the degree of parallelism is high, the total amount of data obtained by each thread will be small, and the proportion of additional overhead such as thread waiting and data copying will increase, which will reduce the efficiency. For this reason, a special working thread can be set to obtain the operation to be processed and perform corresponding processing, wherein the operation to be processed is accompanied by the memory location information of the required processing data. Specifically, all information related to the operation can be packaged into an operation package, so that by executing the operation of each operation package by the working thread, there will be no data access conflict. In other words, multiple second threads (i.e., working threads) for performing operations can be created. The first thread that originally holds the calculation graph is responsible for packaging and passing the operation package to the second thread, and the second thread then performs the operation according to all the information in the operation package. The advantage of doing this is that the parallelism of the calculation graph is separated from the parallelism of the actual calculation. Generally speaking, for a computational graph corresponding to a neural network, the computational graph includes multiple operations that have no mutual dependencies and can be executed simultaneously, so the number of first threads can be less than the number of second threads, and even the number of second threads can be several times the number of first threads. This can reduce the additional overhead and increase the parallelism of the calculation.

[0073] Specifically, the first thread is used to pack the operation package, and the calculation graph construction unit 100 also creates multiple second threads, and the second threads are used to execute the operations in the calculation graph.

[0074] In this embodiment, the execution unit 300 uses each first thread to generate an operation package of at least one operation included in the corresponding computation graph according to the starting address and relative index of the corresponding memory space; and uses multiple second threads to execute the operation packages created by multiple first threads.

[0075] The operation packet contains the starting address of the corresponding memory space, the processing process of the corresponding operation and the relative index of the operation data of the corresponding operation in the memory space. Accordingly, when the operation in the operation packet is executed, the memory address of the data involved in the operation is determined by the above information in the operation packet.

[0076] Alternatively, the operation information packet contains the processing process of the corresponding operation and the memory address of the corresponding operation generated based on the starting address of the corresponding memory space and the relative index of the operation data of the corresponding operation in the memory space. In this case, the execution unit 300 uses the first thread to generate the memory address of the corresponding operation based on the starting address of the corresponding memory space and the relative index of the operation data of the corresponding operation in the memory space. Accordingly, when the operation in the operation packet is executed, the memory address of the data involved in the operation included in the operation packet can be directly used.

[0077] As an example, the execution unit 300 uses each first thread to generate an operation packet of the operation to be executed next among at least one operation included in the corresponding calculation graph according to the starting address and relative index of the corresponding memory space; uses multiple first threads to put the generated operation packets into a buffer queue; uses multiple second threads to obtain operation packets from the buffer queue for execution, and continues to obtain new operation packets from the buffer queue after the operation packet is executed; and uses multiple second threads to inform the corresponding first thread of the execution completion status of the operation packet, so that the corresponding first thread determines the operation to be executed next among at least one operation included in the corresponding calculation graph. In other words, when an operation is executed, its subsequent operations will be packaged and passed to the buffer queue for the second thread to obtain. Therefore, the operation packets obtained by the second working thread are all operation packets that can be executed directly without waiting for the execution of upstream dependent operations.

[0078] Fig. 9 is a flow chart showing a method for performing parallelism between computation graphs according to an embodiment of the present invention. Here, as an example, the method can be performed by a computer program, or by a hardware device or a collection of hardware and software resources specifically used to perform machine learning, big data computing, or data analysis, for example, the method can be performed by a machine learning platform for implementing machine learning related services.

[0079] Reference Fig. 9 In step S10, a computation graph including at least one operation is obtained. In the computation graph, a relative index of operation data of the at least one operation in a memory space is declared.

[0080] As an example, at least one operation included in the computation graph may also be referred to as a computing operation. As an example, the at least one operation involves at least one computing operation in a machine learning algorithm. The operational data of the operation may include input data and output data of the operation.

[0081] When a computation graph is declared, the memory space of the computation graph's operation data is represented by a relative index, rather than a direct index (i.e., an actual index). In the computer field, as an example, the actual index of the memory space can be represented by 8 bytes of data, and the memory space of each operation data of the computation graph according to an embodiment of the present invention is represented by a relative index, and the maximum value of the relative index can be set to the total number of operation data of at least one operation included in the computation graph (i.e., the number of operation data). Usually, this total number is small and can be represented by 4 or 2 bytes of data, thereby reducing the memory size occupied by the computation graph.

[0082] In step S20, a plurality of first threads are created, and a corresponding memory space is allocated to each first thread.

[0083] As an example, when the memory of the operation data of the computation graph is uniformly managed by the memory management module, the memory management module can be used to allocate corresponding memory space to each first thread.

[0084] As an example, corresponding memory space may be allocated to each first thread according to the total amount of operation data (ie, the number of operation data) of at least one operation included in the computation graph.

[0085] In step S30 , the computation graph is copied for each first thread.

[0086] In step S40, at least one operation included in the computation graph is executed according to the starting address and relative index of the memory space of each of the plurality of first threads. In other words, at least one operation included in the computation graph is executed according to the starting address of the memory space of each first thread and the relative index of the operation data of at least one operation included in the computation graph corresponding thereto.

[0087] As an example, in step S40, the memory space of each operation can be determined according to the starting address of the memory space of each first thread and the relative index of the operation data of each operation included in the computation graph, and each operation included in the computation graph can be executed according to the memory space of each operation included in the computation graph.

[0088] The relationship between the computation graph and the memory space in the prior art and the difference between the present invention are as described above and will not be repeated here.

[0089] As an example, in step S40, each first thread may be used to determine the memory address of the operation data of at least one operation included in the corresponding computation graph according to the start address and relative index of the corresponding memory space; and multiple first threads may be used to execute at least one operation included in their respective computation graphs according to the memory addresses determined by each. That is, in this example, the operations of each computation graph are executed by the first thread itself.

[0090] As another example, the method for executing parallel computation graphs according to an embodiment of the present invention also includes the following steps: creating multiple second threads. In this embodiment, in step S40, each first thread is used to generate an operation package of at least one operation included in the corresponding computation graph according to the starting address and relative index of the corresponding memory space; and multiple second threads are used to execute the operation packages created by multiple first threads.

[0091] The operation packet contains the starting address of the corresponding memory space, the processing process of the corresponding operation and the relative index of the operation data of the corresponding operation in the memory space.

[0092] Alternatively, the operation information packet contains the processing process of the corresponding operation and the memory address of the corresponding operation generated based on the starting address of the corresponding memory space and the relative index of the operation data of the corresponding operation in the memory space. In this case, in step S40, the memory address of the corresponding operation is generated by the first thread based on the starting address of the corresponding memory space and the relative index of the operation data of the corresponding operation in the memory space.

[0093] The following will refer to Fig.10 To describe Fig. 9 An example of a flowchart of step S40 in FIG.

[0094] In step S401, each first thread is used to generate an operation packet of an operation to be executed next among at least one operation included in the corresponding calculation graph according to the starting address and relative index of the corresponding memory space.

[0095] In step S402, a plurality of first threads are used to put the operation packets generated by the first threads into a buffer queue.

[0096] In step S403, multiple second threads are used to obtain operation packets from the buffer queue for execution, and new operation packets are continuously obtained from the buffer queue after the operation packets are executed. That is, after an operation is executed, its subsequent operations will be packaged and passed to the buffer queue for the second thread to obtain. Therefore, the operation packets obtained by the second working thread are all operation packets that can be executed directly without waiting for the execution of upstream dependent operations.

[0097] In step S404, the plurality of second threads are used to inform the corresponding first thread of the execution completion status of the operation package, so that the corresponding first thread determines the operation to be executed next among at least one operation included in the corresponding computation graph.

[0098] In the method and device for parallel execution of a computational graph according to an embodiment of the present invention, the computational graph declares the relative index of the operation data of the operations included therein in the memory space, so that the computational graph can be directly copied in a distributed environment to achieve the purpose of data parallel acceleration. At the same time, this relative index declaration method does not involve distributed logic, and the user does not need to consider how to handle multi-threaded logic, thereby reducing the user's usage cost and development difficulty. In addition, the method and device for parallel execution of a computational graph according to an embodiment of the present invention can support the processing of streaming data, and even if the amount of data to be processed is very large, both development cost and execution efficiency can be taken into account. The above has been referred to. Figures 1 to 10 A method and apparatus for parallel execution of a computational graph according to an exemplary embodiment of the present invention is described. However, it should be understood that: Figures 1 to 10 The devices, systems, units, etc. used in the invention may be configured as software, hardware, firmware, or any combination of the above items to perform specific functions. For example, these systems, devices, units, etc. may correspond to dedicated integrated circuits, pure software codes, or modules that combine software and hardware. In addition, one or more functions implemented by these systems, devices, units, etc. may also be uniformly executed by components in physical entity devices (e.g., processors, clients, servers, etc.).

[0099] In addition, the above method can be implemented by instructions recorded on a computer-readable medium. For example, according to an exemplary embodiment of the present application, a computer-readable storage medium storing instructions can be provided, wherein, when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the following steps: obtaining a computational graph including at least one operation, wherein the computational graph declares a relative index of operation data of the at least one operation in a memory space; creating multiple first threads and allocating corresponding memory space to each first thread; copying the computational graph for each first thread; and executing the at least one operation included in the computational graph according to the starting address of the memory space of each of the multiple first threads and the relative index.

[0100] The computer program in the computer-readable medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. It should be noted that the computer program can also be used to perform additional steps in addition to the above steps or perform more specific processing when performing the above steps. The contents of these additional steps and further processing have been described in reference to Figure 3 and Fig.10 It is mentioned in the description of the related method, so it will not be repeated here to avoid repetition.

[0101] It should be noted that the parallel execution method and device of the computational graph according to the exemplary embodiment of the present invention can completely rely on the operation of the computer program to realize the corresponding functions, that is, each unit or device corresponds to each step in the functional architecture of the computer program, so that the entire device or system is called through a special software package (for example, lib library) to realize the corresponding function.

[0102] On the other hand, when Figures 1 to 10 When the various units or devices mentioned are implemented in software, firmware, middleware or microcode, the program code or code segments for performing the corresponding operations can be stored in a computer-readable medium such as a storage medium, so that the processor can perform the corresponding operations by reading and running the corresponding program code or code segments.

[0103] On the other hand, the various devices included in the parallel execution device of the computation graph according to the exemplary embodiment of the present invention can also be implemented by hardware, software, firmware, middleware, microcode or any combination thereof. When implemented by software, firmware, middleware or microcode, the program code or code segment for performing the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that the processor can perform the corresponding operation by reading and running the corresponding program code or code segment.

[0104] For example, a parallel execution device that implements a computational graph according to an exemplary embodiment of the present invention may include at least one computing device and at least one storage device that stores instructions, wherein the instructions, when executed by the at least one computing device, prompt the at least one computing device to perform the following steps: obtaining a computational graph including at least one operation, wherein the computational graph declares a relative index of operation data of the at least one operation in a memory space; creating multiple first threads and allocating corresponding memory space to each first thread; copying the computational graph for each first thread; and executing the at least one operation included in the computational graph according to the starting address of the memory space of each of the multiple first threads and the relative index.

[0105] Specifically, the above-mentioned device can be deployed in a server or on a node device in a distributed network environment. In addition, the device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.). All components of the device may be connected to each other via a bus and / or a network.

[0106] Here, the device is not necessarily a single device, but may be any device or circuit collection that can execute the above instructions (or instruction sets) individually or jointly. The device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0107] In the device, the computing device for executing the parallel execution method of the computational graph according to the exemplary embodiment of the present invention may be a processor, such a processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc. The processor may run instructions or codes stored in one of the storage devices, wherein the storage device may also store data. Instructions and data may also be sent and received through a network via a network interface device, wherein the network interface device may use any known transmission protocol.

[0108] The storage device may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the storage device may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The storage device and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the storage device.

[0109] The above describes the exemplary embodiments of the present application. It should be understood that the above description is only exemplary and not exhaustive, and the present application is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present application, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present application should be based on the scope of the claims.

Claims

1. A method for parallel execution of a computation graph, characterized in that: include: Constructing a computational graph corresponding to the computational task, wherein the computational graph includes computing nodes, data nodes, and edges connecting the computing nodes and the data nodes; Determine, according to the computing task, that the parallel execution mode of the computing graph is parallel execution between the computing graphs; and Executing the computation graph in multiple threads according to a determined parallel execution mode, Parallel execution between computation graphs means that the computation graph and at least one of its copies are executed in parallel through different threads. The parallel execution mode also includes: parallel execution within a computing node and / or parallel execution between computing nodes, wherein the parallel execution within a computing node refers to at least one individual computing node in the computing graph being executed in parallel through different threads; the parallel execution between computing nodes refers to different computing nodes in the computing graph being executed in parallel through different threads. The step of determining the parallel execution mode of the computation graph according to the type of the computation task includes: In the case where the computing task involves asynchronous computing, determining the parallel execution mode between the computing graphs as the parallel execution mode of the computing graphs; When the computing task involves synchronous computing and the computing graph includes synchronous computing nodes, a combination of parallel execution between computing nodes and parallel execution within computing nodes is determined as the parallel execution mode of the computing graph; In the case where the computing task involves synchronous computing and the computing graph does not include synchronous computing nodes, a combination of parallel execution between computing nodes and parallel execution between computing graphs is determined as the parallel execution mode of the computing graph.

2. The parallel execution method according to claim 1, wherein: The asynchronous computing refers to asynchronous training of a neural network; and / or, the synchronous computing refers to synchronous training of a neural network; and / or, the synchronous computing node refers to a batch normalization node.

3. The parallel execution method according to claim 1, wherein: The storage location of the operation data of the data node declared in the computation graph is the relative index of the operation data in the memory space. Furthermore, the step of executing the computation graph in multiple threads according to the determined parallel execution mode includes: allocating corresponding memory space to each thread, and executing the computation graph in parallel in the multiple threads according to the starting addresses of the memory spaces of the multiple threads and the relative indexes.

4. The parallel execution method according to claim 3, wherein: The step of executing the computation graph in parallel in the multiple threads according to the start addresses of the memory spaces of the multiple threads and the relative indexes includes: Determine, by each thread, the memory address of the operation data of the at least one data node included in the corresponding computation graph according to the starting address of the corresponding memory space and the relative index; The multiple threads are utilized to execute respective computation graphs according to the respective determined memory addresses.

5. A parallel execution device for a computation graph, characterized in that: include: A computation graph construction unit, which constructs a computation graph corresponding to the computation task, wherein the computation graph includes computing nodes, data nodes, and edges connecting the computing nodes and the data nodes; A parallel mode determination unit, which determines, according to the computing task, that the parallel execution mode of the computing graph is parallel execution between the computing graphs; and An execution unit executes the computation graph in a plurality of threads according to a determined parallel execution mode, Parallel execution between computation graphs means that the computation graph and at least one of its copies are executed in parallel through different threads. The parallel execution mode also includes: parallel execution within a computing node and / or parallel execution between computing nodes, wherein the parallel execution within a computing node refers to at least one individual computing node in the computing graph being executed in parallel through different threads; the parallel execution between computing nodes refers to different computing nodes in the computing graph being executed in parallel through different threads. Among them, the parallel mode determination unit is configured to: when the computing task involves asynchronous computing, determine the parallel execution mode between computing graphs as the parallel execution mode of the computing graphs; when the computing task involves synchronous computing and the computing graph includes synchronous computing nodes, determine the combination of parallel execution between computing nodes and parallel execution within computing nodes as the parallel execution mode of the computing graph; when the computing task involves synchronous computing and the computing graph does not include synchronous computing nodes, determine the combination of parallel execution between computing nodes and parallel execution between computing graphs as the parallel execution mode of the computing graph.

6. The device according to claim 5, wherein: The asynchronous computing refers to asynchronous training of a neural network; and / or, the synchronous computing refers to synchronous training of a neural network; and / or, the synchronous computing node refers to a batch normalization node.

7. The device according to claim 5, wherein: The computation graph construction unit declares in the computation graph that the storage location of the operation data of the data node is the relative index of the operation data in the memory space, and allocates corresponding memory space to each thread; Furthermore, the execution unit executes the computation graph in parallel in the multiple threads according to the start addresses of the memory spaces of the multiple threads respectively and the relative index.

8. The device according to claim 7, wherein: The execution unit uses each thread to determine the memory address of the operation data of at least one data node included in the corresponding calculation graph according to the starting address of the corresponding memory space and the relative index; and uses the multiple threads to execute their respective calculation graphs according to their respective determined memory addresses.

9. A system comprising at least one computing device and at least one storage device storing instructions, wherein: When the instructions are executed by the at least one computing device, the at least one computing device is prompted to perform the parallel execution method of the computation graph according to any one of claims 1 to 4.

10. A computer-readable storage medium storing instructions, wherein: When the instruction is executed by at least one computing device, the at least one computing device is prompted to execute the parallel execution method of the computing graph according to any one of claims 1 to 4.

Images