Generating parallel computing schemes for neural networks

Abstract

Various embodiments relate to determining a parallel computation scheme for a neural network. A device can receive a computation graph and transform the computation graph into a dataflow graph comprising a recursive subgraph. Each recursive subgraph can comprise a tuple of another recursive subgraph and an operator node or an empty graph. The device can determine a number of split recursions based on a number of parallel computing devices. For each split recursion, the device can determine a cost corresponding to an operator node, determine a processing order of the recursive subgraph, and process the recursive subgraph. To process a recursive subgraph, the device can select a split axis for one or more tensors associated with an operator node of the recursive subgraph. The device can output a split scheme, where the split scheme comprises a split axis for each tensor associated with the operator node. Various devices, methods, and computer programs are disclosed.

Description

Technical Field

[0001] This invention generally relates to the fields of neural networks and distributed parallel computing technology. Specifically, some embodiments of this invention relate to generating parallel computing schemes for neural networks. Background Technology

[0002] Neural networks or other machine learning tools can automate tasks for a wide variety of applications. Exemplary applications include the analysis and processing of text, speech, image, or video data. However, even though neural networks provide a powerful tool for task automation, processing complex neural networks can be slow or even impractical on all devices. Summary of the Invention

[0003] The purpose of this summary is to introduce a selection of concepts in a simplified form, which will be further described in the following specific implementations. The purpose of this summary is not to identify key or essential features of the subject matter protected by the claims, nor to limit the scope of the subject matter protected by the claims.

[0004] The object of this invention is to provide an efficient and deterministic method for generating parallel computing schemes for neural networks. The above and other objects can be achieved through the features of the independent claims. Further implementations will be apparent from the dependent claims, detailed description, and drawings.

[0005] In a first aspect, an apparatus is provided for determining a parallel computing scheme for a neural network. The apparatus can be used to receive a computation graph of the neural network and transform the computation graph into a data flow graph comprising multiple recursive subgraphs. Each recursive subgraph may include another recursive subgraph and tuples of operator nodes or an empty graph. The apparatus can be used to determine the number of recursive partitions based on the number of parallel computing devices. The apparatus can be used to: for each recursive partition, determine multiple costs corresponding to multiple operator nodes associated with the data flow graph; determine a processing order for the multiple recursive subgraphs based on a descending order of the multiple costs; and process the multiple recursive subgraphs according to the determined processing order. Processing the recursive subgraphs may include: selecting a partition axis for tensors associated with the operator nodes of the recursive subgraphs. The apparatus can be used to output a partition scheme, wherein the partition scheme includes a partition axis for each tensor associated with the multiple operator nodes. This scheme provides an efficient and deterministic way to determine a parallel computing scheme for a neural network.

[0006] According to one implementation of the first aspect, the device can also be used to determine the number of times the segmentation recursion is performed, such that 2 NThis is equal to the number of parallel computing devices, where N is the number of recursive partitioning operations. This scheme can determine the pre-configured number of recursive partitioning operations.

[0007] According to one implementation of the first aspect, the device can also be used to determine the multiple costs corresponding to the multiple operator nodes based on the amount of data to be transmitted between the parallel computing devices for each operator node. This scheme can prioritize high-cost operator nodes when determining the partitioning scheme.

[0008] According to one implementation of the first aspect, the device can also be used to select the splitting axis for the tensor associated with the operator node based on intra-operator communication cost, wherein the intra-operator communication cost includes the amount of data to be transmitted between the parallel computing devices for the operator node. This scheme can reduce communication between the parallel computing devices.

[0009] According to one implementation of the first aspect, the device can also be used to select the splitting axis for the tensor associated with the operator node based on the intra-operator communication cost if no splitting axis is determined for adjacent operator nodes in the current splitting recursion. This scheme can roughly estimate the communication cost of the operator node.

[0010] According to one implementation of the first aspect, the device can also be used to select the splitting axis for the tensor associated with the operator node based on inter-operator communication cost, wherein the inter-operator communication cost includes the amount of data to be transmitted by the parallel computing devices when executing the adjacent operator node based on the shared tensor between the operator node and the adjacent operator node, or when executing the operator node based on the output of the adjacent operator node. Due to the dependencies between operator nodes, this scheme can reduce communication between parallel computing devices.

[0011] According to one implementation of the first aspect, the device can also be used to select the splitting axis for the tensor associated with the operator node based on the intra-operator communication cost and the inter-operator communication cost if at least one splitting axis has been determined for the adjacent operator node or another adjacent operator node in the current splitting recursion. Due to intra-operator and inter-operator communication, this scheme can reduce communication between parallel computing devices.

[0012] According to one implementation of the first aspect, the device can also be used to determine whether the partitioning scheme meets at least one memory requirement associated with the parallel computing device. The device can also be used to output the partitioning scheme if it is determined that the partitioning scheme meets the at least one memory requirement associated with the parallel computing device. This scheme can verify the determined parallel computing scheme against the capabilities of the parallel computing device.

[0013] Secondly, a method for determining a parallel computing scheme for a neural network is provided. The method may include receiving a computation graph of the neural network and transforming the computation graph into a data flow graph comprising multiple recursive subgraphs. Each recursive subgraph may include another recursive subgraph and tuples of operator nodes or an empty graph. The method may include determining the number of recursive partitions based on the number of parallel computing devices. The method may include: for each recursive partition, determining multiple costs corresponding to multiple operator nodes associated with the data flow graph; determining a processing order for the multiple recursive subgraphs based on a descending order of the multiple costs; and processing the multiple recursive subgraphs according to the determined processing order. Processing the recursive subgraphs may include selecting a partition axis for tensors associated with the operator nodes of the recursive subgraphs. The method may include outputting a partition scheme, wherein the partition scheme includes a partition axis for each tensor associated with the multiple operator nodes. This scheme provides an efficient and deterministic way to determine a parallel computing scheme for a neural network.

[0014] According to one implementation of the second aspect, the method may further include determining the number of times the segmentation recursion is performed, such that 2 N This is equal to the number of parallel computing devices, where N is the number of recursive partitioning operations. This scheme can determine the pre-configured number of recursive partitioning operations.

[0015] According to one implementation of the second aspect, the method may further include determining the plurality of costs corresponding to the plurality of operator nodes based on the amount of data to be transmitted between the parallel computing devices for each operator node. This scheme can prioritize high-cost operator nodes when determining the partitioning scheme.

[0016] According to one implementation of the second aspect, the method may further include selecting the slicing axis for the tensor associated with the operator node based on intra-operator communication cost, wherein the intra-operator communication cost includes the amount of data to be transmitted between the parallel computing devices for the operator node. This scheme can reduce communication between the parallel computing devices.

[0017] According to one implementation of the second aspect, the method may further include selecting the splitting axis for the tensor associated with the operator node based on the intra-operator communication cost if no splitting axis is determined for the adjacent operator node in the current splitting recursion. This scheme can roughly estimate the communication cost of the operator node.

[0018] According to one implementation of the second aspect, the method may further include selecting the slicing axis for the tensor associated with the operator node based on inter-operator communication cost, wherein the inter-operator communication cost includes the amount of data to be transmitted between the parallel computing devices when executing the adjacent operator node based on the shared tensor between the operator node and the adjacent operator node, or when executing the operator node based on the output of the adjacent operator node. Due to the dependencies between operator nodes, this scheme can reduce communication between parallel computing devices.

[0019] According to one implementation of the second aspect, the method may further include selecting the splitting axis for the tensor associated with the operator node based on the intra-operator communication cost and the inter-operator communication cost if at least one splitting axis has been determined for the adjacent operator node or another adjacent operator node in the current splitting recursion. Due to intra-operator and inter-operator communication, this scheme can reduce communication between parallel computing devices.

[0020] According to one implementation of the second aspect, the method may further include determining whether the partitioning scheme meets at least one memory requirement associated with the parallel computing device. The method may further include outputting the partitioning scheme if it is determined that the partitioning scheme meets the at least one memory requirement associated with the parallel computing device. This scheme can verify the determined parallel computing scheme against the capabilities of the parallel computing device.

[0021] Thirdly, a computer program is provided. The computer program may include program code configured to, when executed on a computer, cause the method described according to any implementation of the second aspect to be performed.

[0022] Fourthly, a computer program product is provided, comprising a computer-readable storage medium storing program code. The program code may include instructions for executing any implementation of the second aspect.

[0023] Therefore, implementations of the present invention can provide an apparatus, method, computer program, and computer program product for determining a parallel computing scheme for a neural network. These and other aspects of the invention will become apparent from the embodiments described below. Attached Figure Description

[0024] The accompanying drawings, which provide a further understanding of exemplary embodiments and form part of this specification, illustrate exemplary embodiments and, together with the description, help to explain the exemplary embodiments. In the drawings:

[0025] Figure 1 An example of parallel computation for neural networks with different tensor dimensions is shown according to one embodiment;

[0026] Figure 2 An example of data parallelism according to one embodiment is shown;

[0027] Figure 3 An example of model parallelism according to one embodiment is shown;

[0028] Figure 4 An example of an abstract paradigm computation graph according to one embodiment is shown;

[0029] Figure 5 An example of a computational graph and device topology diagram of a neural network according to an embodiment is shown;

[0030] Figure 6 An example of a device for implementing one or more embodiments is shown;

[0031] Figure 7 An example of a system architecture for determining a parallel computing scheme according to one embodiment is shown;

[0032] Figure 8 An example of a recursive computation graph according to one embodiment is shown;

[0033] Figure 9 An example of transforming a computation graph into a data flow graph including recursive subgraphs is shown according to one embodiment;

[0034] Figure 10 An example of a method for determining a parallel computing scheme for a neural network according to one embodiment is shown;

[0035] Figure 11 An example of a partitioning configuration for matrix multiplication according to one embodiment is shown;

[0036] Figure 12 An example of another partitioning configuration for matrix multiplication according to one embodiment is shown;

[0037] Figure 13 An example of another partitioning configuration for matrix multiplication according to one embodiment is shown;

[0038] Figure 14 An example of the inter-operator cost of tensor rearrangement communication according to one embodiment is shown;

[0039] Figure 15 An example of determining the subgraph processing order of matrix multiplication according to one embodiment is shown;

[0040] Figure 16 An example of the partitioning axis of a tensor associated with matrix multiplication according to one embodiment is shown;

[0041] Figure 17An example of the splitting axis of a tensor associated with another matrix multiplication according to one embodiment is shown;

[0042] Figure 18 An example of the partitioning axis of a subtensor associated with the matrix multiplication according to one embodiment is shown;

[0043] Figure 19 An example of a splitting axis of a subtensor associated with another matrix multiplication according to one embodiment is shown;

[0044] Figure 20 An example of a method for determining a parallel computing scheme for a neural network according to one embodiment is shown.

[0045] Similar reference numerals are used to denote the same parts in the accompanying drawings. Detailed Implementation

[0046] The present invention will now be described in detail with reference to the embodiments illustrated in the accompanying drawings. The following detailed description, provided in conjunction with the accompanying drawings, is intended as a description of embodiments and is not intended to represent the only form in which embodiments can be constructed or utilized. The description illustrates the functionality of the examples and the sequence of steps for constructing and operating the examples. However, the same or equivalent functionality and sequence can be achieved through different examples.

[0047] A neural network can comprise a set of interconnected neurons that can be trained together to perform a specific task. A layer of a neural network can comprise multiple neurons that provide an output based on input received from one or more other layers. Therefore, each neuron or node can be used to perform basic computations. Typically, a neuron can connect to one or more other neurons, and these connections can be associated with weights. Weights can be used to adjust the influence of each neuron on the neural network's output. For example, weights can be adjusted to train a neural network to provide the desired output on a set of training data. Layers of a neural network can be represented as tensors. Tensors can comprise multidimensional arrays containing scalar values ​​representing parameters such as the layer's weights. For example, a convolutional layer can be represented as a tensor containing the weights of multiple filters. Furthermore, the inputs and outputs of a layer can be represented as tensors.

[0048] According to one embodiment, the device can determine a parallel computing scheme for a neural network. The device can receive a computation graph. The device can transform the computation graph into a data flow graph including recursive subgraphs. Each recursive subgraph can include another recursive subgraph and tuples of operator nodes, or an empty graph. The device can determine the number of recursive partitions based on the number of parallel computing devices. For each recursive partition, the device can determine the cost corresponding to the operator node; determine the processing order of the recursive subgraph; and process the recursive subgraph. To process the recursive subgraph, the device can select a partition axis for one or more tensors associated with the operator nodes of the recursive subgraph. The device can output a partition scheme, wherein the partition scheme includes a partition axis for each tensor associated with the operator node. Therefore, an efficient and deterministic method is disclosed for determining a parallel computing scheme for a neural network. This parallel computing scheme can, for example, be applied to training a neural network.

[0049] Figure 1 An example of parallel computation for neural networks with different tensor dimensions according to one embodiment is shown. In data parallelism, tensor 102 can be partitioned into multiple sub-tensors related to different samples (S), which can be processed by different computing devices, such as a central processing unit (CPU), a graphics processing unit (GPU), or a processor core of a processing unit. In model parallelism, tensor 104 can be partitioned into sub-tensors related to parameters (P), such as channels. In a first example of hybrid parallelism, tensor 106 can be partitioned into sub-tensors related to samples (S) and parameters (P), such as channels. In a second example of hybrid parallelism, tensor 108 can be partitioned into sub-tensors related to samples (S), parameters (P), and attributes (A), such as length.

[0050] Different types of neural network layers can be associated with different parallelizable dimensions. For example, one-dimensional (1D) pooling layers can be parallelized in terms of samples or attributes such as length or channels. 1D convolutional layers can be parallelized in terms of samples or attributes such as length. Two-dimensional (2D) convolutional layers can be parallelized in terms of samples, attributes such as length and width, or parameters such as channels. Matrix multiplication can be parallelized in terms of samples or parameters such as channels.

[0051] Based on Measuring parallel efficiency, where, The speedup ratio is given by p, where p represents the number of processors, T1 represents the execution time of the corresponding sequential algorithm, and T... p This represents the execution time of a parallel algorithm with p processors. Parallel efficiency E pThe value of is generally between 0 and 1, representing the utilization of processors involved in computation relative to communication and synchronization costs during problem-solving. A higher value indicates higher utilization.

[0052] Figure 2 An example of data parallelism according to one embodiment is shown. In data parallelism, computing devices can process different sample groups 210, 212, 214, such as training data. Each computing device can use the complete neural network (model) computation graph. For example, it can be grouped only by the batch size (also called samples) of the neural network. More computing devices can be used to increase computing power, thereby reducing training time. Each computing device can communicate with parameter server 202 to receive weights W' and calculate weight updates ΔW based on the training data available to each computing device. Each computing device can provide its weight updates ΔW to parameter server 202, which can merge the weight updates received from the computing devices and determine the resulting weight updates of the neural network, for example, using W' = W – ηΔW, and then send the resulting weight updates to the computing devices to determine further weight updates.

[0053] Figure 3 An example of model parallelism according to one embodiment is illustrated. In model parallelism, multiple computing devices can collaboratively process data in the same data set 312. The neural network computation graph can be divided into multiple subgraphs 302, 304, 306, and 308. Each computing device can process one of the subgraphs. The method may include pipelined partitioning of the computation graph and mathematical partitioning of the neural network's channels or other tensor dimensions such as height or width. Model parallelism not only improves computational power but also solves the problem of not being able to store the entire neural network on a single computing device.

[0054] In hybrid parallelism, data parallelism and model parallelism can be used together to improve parallel efficiency. However, due to the many possible partitioning options, the optimization complexity of hybrid parallelism strategies can be high. The example embodiments described in this paper provide an efficient solution for a hybrid parallel computing scheme for acquiring neural networks or any other suitable type of machine learning model.

[0055] Figure 4 An example of an A-normal form computation graph according to one embodiment is shown. An A-normal form (ANF) computation graph is a computation graph in which the parameters of a function are trivial, for example, readily evaluable. An ANF graph represents the dependencies (edges) between functions and their parameters (nodes). Figure 4The ANF representation of the expression f(g(x),h(y)) is shown. The computation of a neural network can be defined by an ANF graph, which defines operations with corresponding inputs and / or outputs to compute the neural network.

[0056] Figure 5 An example of a computation graph (G) and device topology graph (D) of a neural network according to one embodiment is shown. The computation graph, also referred to as an operator graph, may include multiple operator nodes or computation nodes. For example, a concatenation operation (Concat) can be performed based on the outputs of two convolution operations (Conv). Furthermore, a matrix multiplication operation (MatMul) can be performed on the output of the concatenation operation. Figure 5 A distributed parallel device topology is also illustrated. For example, each of two CPUs can connect to two GPUs. These CPUs can also be connected via a network. The distributed parallel scheme can include a mapping from each operator node of G to a configuration C, where each configuration C describes the parallel implementation of the operator nodes of G on multiple devices in topology D.

[0057] Figure 6 Examples of devices for implementing one or more embodiments are shown. Device 600 may, for example, be used to determine a parallel computing scheme for a neural network. Device 600 may include at least one processor 602. The at least one processor may include one or more of various processing devices, such as a coprocessor, microprocessor, controller, digital signal processor (DSP), processing circuitry with or without a DSP, or various other processing devices including integrated circuits, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontroller units (MCUs), hardware accelerators, dedicated computer chips, etc.

[0058] The device may also include at least one memory 604. The memory can be used to store, for example, computer program code, such as operating system software and application software. The memory 604 may also be used to store one or more neural networks or one or more portions thereof. For example, the memory 604 may be used to store a topology (e.g., a computation graph) and the types of operations and / or the shapes of tensors associated with the topology. As described in the description of model parallelism, the entire neural network (with attribute values) may be too large to be stored in the memory of a single device. Therefore, the neural network or one or more portions thereof may be stored in the memory 604 or received from an external source via the communication interface 608. The memory may include one or more volatile memory devices, one or more non-volatile memory devices, and / or combinations thereof. For example, the memory may be embodied as a magnetic storage device (such as a hard disk drive, floppy disk, magnetic tape, etc.), an optical-magnetic storage device, or a semiconductor memory (such as a mask ROM, a programmable ROM (PROM), an erasable PROM (EPROM), a flash ROM, random access memory (RAM), etc.).

[0059] Device 600 may also include a communication interface 608 for enabling device 600 to send and / or receive information. The communication interface may be used to provide at least one wireless connection, such as a 3GPP mobile broadband connection (e.g., 3G, 4G, 5G); a wireless local area network (WLAN) connection, such as a connection conforming to the IEEE 802.11 series or Wi-Fi Alliance standards; a short-range wireless network connection, such as Bluetooth, near field communication (NFC), or RFID connection; a local wired connection, such as a local area network (LAN) connection or a universal serial bus (USB) connection; or a wired internet connection. The wired connection may include an internal data interface, such as a system bus, peripheral component interconnect express (PCI-E), or any other hardware-appropriate interconnect.

[0060] Device 600 may also include a user interface 610, including at least one input device and / or at least one output device. The input device may take various forms, such as a keyboard, touchscreen, or one or more embedded control buttons. The output device may include, for example, a display, a speaker, a vibration motor, etc.

[0061] When device 600 is used to implement a certain function, one and / or some components of the device, such as at least one processor and / or memory, can be used to implement that function. Furthermore, when at least one processor 602 is used to implement a certain function, that function can be implemented using, for example, program code 606 included in memory 604.

[0062] The functions described herein may be performed at least in part by one or more computer program product components, such as software components. According to one embodiment, the device includes a processor or processor circuitry, such as a microprocessor, which, when executing program code, performs embodiments of the operations and functions described herein. Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. Exemplary types of hardware logic components that may be used, such as but not limited to, include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), and graphics processing units (GPUs).

[0063] Device 600 may be used to perform the methods described herein or may include means for performing the methods described herein. In one example, the means includes at least one processor 602 and at least one memory 604 including program code 606 for causing device 600 to perform the methods when executed by at least one processor 602.

[0064] Device 600 may include computing devices such as servers, desktop computers, laptops, tablets, and mobile phones, etc. Although device 600 is illustrated as a single device, it is understood that, where applicable, the functionality of device 600 may be distributed across multiple devices, for example, as an example embodiment of a cloud computing service implementation.

[0065] Figure 7An example system architecture for determining a parallel computing scheme according to one embodiment is shown. System 700 may receive a computation graph of a neural network as input. The computation graph may, for example, include an ANF graph. Although the example embodiment has been described using an ANF graph as an example, it is understood that the example embodiment can be applied to any suitable computation graph. For example, the ANF graph may be received from a neural network resolver. The neural network resolver may be located on the same device as system 700, or the ANF graph may be received from an external neural network resolver, for example, via communication interface 608. The output of system 700 may include a distributed parallel computing scheme for the input computation graph. The output may be provided as input, for example, via an internal or external communication interface to a distributed executable code generator. Various functions, blocks, or modules of system architecture 700 may be implemented as software and / or hardware components, for example, in device 600.

[0066] The ANF (Application Not Filter) of the recursive graph parser 710 can be used to receive an ANF graph and transform it into a recursively structured computation graph, such as a data flow graph including multiple recursive subgraphs. A recursive graph can include subgraphs and operator nodes. Each recursive subgraph can be a recursive graph or an empty graph.

[0067] Figure 8 An example of a recursive computation graph according to one embodiment is shown. The graph may include operator nodes 801 and subgraph 802. Alternatively, the graph can be considered to include operator nodes 802 having subgraph 801. With 801 as the operator node and 802 as its subgraph, subgraph 802 may include a subgraph, in this example, an empty graph, and its operator node (itself). For example, with respect to the example graph ABC having three operator nodes (A, B, C), the recursive graph may include operator node A and subgraph BC. Subgraph BC may include operator node B and subgraph C. Subgraph C may include operator node C and an empty graph. This order may be described by (A(B(C()))). The graph may be reordered as (((A()B)C), (A(B()C)), or any other order.

[0068] Figure 9An example of transforming a computation graph into a data flow graph including recursive subgraphs according to one embodiment is shown. Operator nodes can be characterized by the type of operator node, the shape of its output tensor and / or a list of input tensors and / or its incoming edges. A first figure (a) shows an example of a matrix multiplication operator node 901 with output tensor C and input tensors A and B. A second figure (b) shows an example of an ANF 902 of E = MatMul(MatMul(A,B)D). Alternatively, the ANF graph can be described as E = (AXB)XD. A third figure (c) shows an example of a recursive graph 903 determined based on the computation data flow graph 902. The recursive graph 903 includes a first matrix multiplication operator node (MatMul-E) and a first subgraph. The first subgraph includes NOP-D (input D without operation) and a second subgraph. The second subgraph includes a second matrix multiplication operator node (MatMul-C) and a third subgraph. The third subgraph includes NOP-A and a fourth subgraph. The fourth subgraph includes NOP-A and the fifth subgraph, which is an empty graph. Therefore, the computation graph of, for example, the ANF graph, can be transformed into a data flow graph including recursive subgraphs, which can be used to efficiently determine parallel computing schemes.

[0069] Refer again Figure 7 System 700 may further include a partitioning scheme generator 720, which can control the number of external recursions based on the number of parallel computing devices. Partitioning scheme generator 720 can also call graph partitioning module 722 to execute internal recursions. Partitioning scheme generator 720 can receive intermediate strategies obtained by graph partitioning module 722 and merge these intermediate strategies into a final scheme. Partitioning scheme generator 720 can also verify the feasibility of the final scheme based on the characteristics of parallel computing, such as memory information of parallel computing devices and / or one or more memory performance models. Finally, partitioning scheme generator 720 can output the final scheme to, for example, a distributed graph generator 730.

[0070] The cost model manager 728 can store and manage one or more symbolic performance cost models, such as communication cost models and memory cost models. Both communication cost models and memory cost models can be algebraic symbolic models that can be combined for computation rather than execution to obtain hardware measurement information. The cost model manager 728 can determine the cost of a specific tensor operation, for example, based on the dimension of the tensor associated with the operator node and the type of the operator node.

[0071] The memory cost model can include multiple tensor elements stored in the partition. The memory cost model can serve as a termination criterion or additional termination criterion for determining whether to perform further partitioning recursion, which will be combined with... Figure 10 Further description. Combined with Figures 11 to 14 An example of a communication cost model is described.

[0072] The graph segmentation module 722 can call the node sorting module 724 to reorganize the recursive sequence of graph nodes for processing. The graph segmentation module 722 can also receive the segmentation axis selected by the node scheme generator 726 for each node. The graph segmentation module 722 can merge the segmentation axes selected by the operator nodes of the recursive graph into an intermediate scheme. The intermediate scheme can be sent to the segmentation scheme generator 720.

[0073] The node sorting module 724 can invoke the cost model manager 728, such as the communication cost model within the cost model manager 728, to rearrange the operator nodes of the recursive graph structure. For example, the operator node with the highest potential internal communication cost can be placed before other operator nodes in the processing order. The graph segmentation module 722 can be invoked according to the rearranged order of the operator nodes.

[0074] The node scheme generator 726 can call the cost model manager 728, such as the communication cost model within the cost model manager 728, to select a splitting axis for the current operator node. A splitting axis can be selected for each node in the recursive graph to provide intermediate splitting schemes for the splitting scheme generator 720.

[0075] Figure 10 An example of a method for determining a parallel computing scheme for a neural network according to one embodiment is shown.

[0076] In 1001, a device, such as device 600, can receive a computation graph of a neural network. The computation graph can be of any suitable form. The device can transform the computation graph into a data flow graph that includes multiple recursive subgraphs. However, according to one embodiment, the input graph can include a recursive graph, so it may not be necessary to transform the input computation graph. The recursive subgraphs can include tuples of other subgraphs and operator nodes. Alternatively, the subgraphs can be empty graphs.

[0077] The device can also determine the number of partitioning recursions based on the number of parallel computing devices. At each partitioning recursion, the tensor associated with the operator node can be partitioned into two sub-tensors. For example, at the first partitioning recursion, the tensor can be partitioned into two sub-tensors. At the second partitioning recursion, each of the two sub-tensors can be further partitioned into two sub-tensors, and so on. The partitioning recursion can be iterated until the number of sub-tensors determined by the recursion for each original tensor equals the number of parallel computing devices. For example, the device can determine the number of partitioning recursions such that 2 N It equals the number of parallel computing devices, where N is the number of recursive divisions.

[0078] In 1002, the device can sort operator nodes. The device can, for example, determine the processing order of multiple recursive subgraphs. The device can determine multiple costs corresponding to multiple operator nodes associated with a data flow graph, and determine the processing order of multiple recursive subgraphs based on a descending order of the multiple costs. The multiple costs corresponding to multiple operator nodes can be determined based on the amount of data to be transferred between parallel computing devices for each operator node. The amount of data can be determined based on the upper limit of the data to be transferred by the executing operator node. Alternatively, the amount of data can be determined based on the lower limit of the data to be transferred by the executing operator node. Different amounts of data to be transferred can correspond to different slicing axes. For example, a recursive subgraph including operator nodes with the highest potential intra-operator communication cost can be selected as the first recursive subgraph to be processed. Processing a recursive subgraph can include: processing the operator nodes of the recursive subgraph. Subgraphs can be processed in descending order of intra-operator communication cost. For example, the node sorting module 724 can retrieve multiple costs associated with the operator nodes of each subgraph from the cost model manager 728. The node sorting module 724 can determine the processing order based on the retrieved information. Furthermore, the node sorting module 724 can provide instructions on the processing order to the node scheme generator 726 and / or the graph partitioning module 722.

[0079] In 1003, the device can segment operator nodes. The device can process multiple recursive subgraphs in the processing order determined in 1002. Processing a recursive subgraph may include selecting a segmentation axis for one or more tensors associated with the operator nodes of the recursive subgraph. For example, a segmentation axis may be determined for one or more input tensors and / or output tensors. For example, the device may start with the first recursive subgraph in the processing order determined in 1002. The device may select a segmentation axis for at least one input and / or output tensor associated with the operator nodes of the first recursive subgraph.

[0080] According to one embodiment, the device can select a slicing axis for a tensor associated with an operator node based on intra-operator communication costs. Intra-operator communication costs can include the amount of data to be transferred between parallel computing devices for the operator node. The amount of data can be measured, for example, by the number of tensor elements that need to be interacted between parallel computing devices to execute the operator node. For example, multiple parallel computing devices may need a specific portion of the input tensor to perform a part of the operation. Multiple computing devices may need the same data, and therefore data may need to be transferred between parallel computing devices, which can lead to high communication costs. Determining the slicing axis based on intra-operator communication costs can reduce the communication overhead caused by parallel processing.

[0081] According to one example embodiment, if no splitting axis has been determined for adjacent operator nodes in the current splitting recursion, the device can select a splitting axis for the tensor associated with the operator node based on the intra-operator communication cost. This allows for cost estimation of the operator node even if the splitting scheme for one or more adjacent nodes has not yet been determined. For example, if no splitting axis has been determined for one or more adjacent operator nodes, it may be impossible to estimate the interaction cost between nodes. Therefore, the splitting axis can be selected based on the intra-operator communication cost. For example, the device can select the splitting axis associated with the lowest intra-operator communication cost.

[0082] According to one example embodiment, a device can select a partitioning axis for a tensor associated with an operator node based on inter-operator communication costs, where inter-operator communication costs include the amount of data that needs to be transferred between parallel computing devices to execute adjacent operators based on shared tensors between the operator node and adjacent operators, or to execute operators based on the outputs of adjacent operators. The amount of data can be measured, for example, by the number of tensor elements that need to be exchanged between parallel computing devices to execute adjacent operators or operators. This allows for taking into account interactions between operator nodes when determining the partitioning axis. For example, tensors may be shared with adjacent operator nodes, and these shared tensors or portions thereof may need to be transferred between different computing devices to execute adjacent operator nodes or operators. Alternatively, or additionally, the outputs of operator nodes or one or more portions thereof may need to be transferred between different computing devices. Taking into account these inter-operator communication costs allows for the selection of a partitioning axis, thereby reducing the overhead caused by additional communication between parallel computing devices due to interactions between operator nodes. For example, the device can select a partitioning axis associated with the lowest inter-operator communication cost.

[0083] According to one example embodiment, if at least one splitting axis has been determined for an adjacent operator node or another adjacent operator node in the current splitting recursion, the device can select a splitting axis for the tensor associated with the operator node based on intra-operator communication cost and inter-operator communication cost. For example, if a splitting axis has already been determined for one of a plurality of adjacent operator nodes, both intra-operator communication cost and inter-operator communication cost can be considered simultaneously. For example, the device can select a splitting axis based on the sum of intra-operator communication cost and inter-operator communication cost or other combinations thereof. For example, a splitting axis associated with the minimum value of the sum can be selected.

[0084] In step 1004, the device can determine whether the inner loop of method 1000 has ended. For example, the device can determine whether multiple recursive subgraphs of the data flow graph have been processed. If there are still subgraphs to be processed, the device can return to step 1003 and process the next subgraph in the processing order. For example, a splitting axis can be determined for the next recursive subgraph. The selected splitting axis can be different from the splitting axis selected for the previously processed recursive subgraphs. If multiple recursive subgraphs have been processed, the device can determine an intermediate parallel scheme, which may include the splitting axes selected for the multiple recursive subgraphs.

[0085] In step 1005, the device can determine whether the outer loop has ended. For example, the device can determine whether the determined number of splitting recursions has been completed. If there are still splitting recursions to be executed, the device can return to step 1002 to sort the operator nodes. Therefore, the device can redetermine the processing order of the recursive subgraph. The processing order can be different from the processing order determined for the previous splitting recursion. Alternatively, if the recursive subgraph is configured to be processed in the same order as the previous splitting recursion, the device can return to step 1003.

[0086] Alternatively, or as an alternative to a determined number of partitioning recursions, a memory cost model can be used to determine whether the outer loop has ended. For example, if it is determined that all tensor partitions are suitable for the number of parallel computing devices associated with the current partitioning recursion, the device can terminate the partitioning. Thus, according to an example embodiment, the device can determine the number of partitioning recursions based on a memory cost model.

[0087] In method 1003, the device can process recursive subgraphs in processing order until each of the multiple recursive subgraphs has been processed. At each iteration of the inner loop, the device can select a splitting axis for the (sub)tensors of the operator nodes generated in the previous iteration of the outer loop. Therefore, each tensor can ultimately be split into multiple subtensors, where the number of subtensors corresponds to the number of parallel computing devices. Typically, in method 1000, the inner loop can determine the splitting axis for the tensors of the recursive subgraph, and the outer loop further splits the tensors based on the number of parallel computing devices.

[0088] In step 1004, the device can determine whether the inner loop has ended and move to step 1003 to perform a split on the next operator node, or if there are no other operator nodes to be processed in this split recursion, it can move to step 1005.

[0089] In step 1005, the device can determine whether the outer loop has ended. For example, the device can determine whether the determined number of splitting recursions has been completed. If there are no other splitting recursions to be executed, the device can move to step 1006.

[0090] In 1006, the device can output a selected partitioning scheme. The partitioning scheme can include partitioning axes for each tensor associated with multiple operator nodes. For example, the partitioning scheme can include an indication of the partitioning axis for each tensor. The partitioning scheme can also include: an indication of the partitioning axis of the (sub)tensors obtained by partitioning the tensors according to the corresponding partitioning axes.

[0091] According to one example embodiment, the device can determine whether a partitioning scheme meets at least one memory requirement associated with a parallel computing device. Furthermore, if it is determined that the partitioning scheme meets at least one memory requirement associated with a parallel computing device, the device can output the partitioning scheme. For example, the device can determine whether the parallel computing device has sufficient memory and / or processing resources to store and / or process the (sub)tensors and operations allocated to each computing device. According to one example embodiment, if it is determined that the partitioning scheme meets all memory requirements associated with a parallel computing device, the device can output the partitioning scheme.

[0092] Although method 1000 is described as being executed by a device, it should be understood that the method can be executed by any suitable entity, such as a system comprising one or more software and / or hardware components or multiple distributed computing devices.

[0093] Figure 11 An example of a partitioning configuration for matrix multiplication according to one embodiment is shown. An example is provided with matrices A and B as input tensors. In a first partitioning configuration, matrix A can be partitioned with reference to axis i. For example, the upper half of matrix A can be considered as a first subtensor, and the lower half of matrix A can be considered as a second subtensor. The computation of matrix multiplication can be distributed between a first computing device and a second computing device. To compute the partitioned matrix multiplication, the first computing device may need to access the upper half of matrix A with dimension i,k and the entire matrix B with dimension k,j. The second computing device may need to access the lower half of matrix A and the entire matrix B; therefore, based on the dimension of matrix B, for example, The intra-operator communication cost of the first segmentation configuration can be determined. The output tensor may include an upper subtensor containing the output matrix of the first computing device and a lower subtensor containing the output matrix of the second computing device.

[0094] Figure 12An example of another partitioning configuration for matrix multiplication according to one embodiment is shown. Again, matrices A and B are provided as examples of input tensors. In this second partitioning configuration, matrix B can be partitioned with reference to axis j. For example, the left half of matrix B can be considered a first subtensor, and the right half of matrix B can be considered a second subtensor. To compute the partitioned matrix multiplication, a first computing device may need access to the left half of matrix B and the entire matrix A, while a second computing device needs access to the right half of matrix B and the entire matrix A. Therefore, based on the dimensions of matrix B, for example, The communication cost within the operator can be determined. The output tensor may include a left subtensor containing the output matrix of the first computing device and a right subtensor containing the output matrix of the second computing device.

[0095] Figure 13 An example of another partitioning configuration for matrix multiplication according to one embodiment is shown. Again, matrices A and B are provided as examples of input tensors. In this third partitioning configuration, matrices A and B can be partitioned with reference to axis k. For example, the left half of matrix A can be considered a first subtensor, and the right half a second subtensor. The upper half of matrix B can be considered a third subtensor, and the lower half a fourth subtensor. To compute the partitioned matrix multiplication, the first computing device may only need to access the left half of matrix A and the upper half of matrix B, while the second computing device may only need to access the right half of matrix A and the lower half of matrix B. However, it may be necessary to sum the output matrices of the computing devices and redistribute them. Therefore, based on the dimension of the output, for example, based on... The cost of communication within the operator can be determined.

[0096] Figure 14 An example of determining the inter-operator cost of tensor rearrangement communication according to one embodiment is shown. The dotted and short dashed portions can be associated with different computing devices. For example, if a node is configured to partition one operator node along axis i and another along axis k, it may be necessary to transmit half of the dotted and short dashed portions between computing devices because the other half may already exist on their respective computing devices. If network bandwidth is not saturated, communication can occur simultaneously for both the dotted and short dashed portions. Therefore, the cost can be determined based on the dimensions of the tensors required by the two computing devices, for example, Determine the communication cost between operators.

[0097] Figures 15 to 19 An example is shown of determining a partitioning scheme on four devices in a distributed parallel manner to compute a graph involving two consecutive matrix multiplications. Based on the number of parallel computing devices (4), the number of partitioning recursions N can be determined as N = 2, because 2 N =22 =4.

[0098] Figure 15 An example of determining the subgraph processing order for matrix multiplication according to one embodiment is shown. The above intra-operator communication cost model can be used to determine the processing order of the first operator node (Op1) and the second operator node (Op2), both of which include matrix multiplication in this example. The dimensions of the matrix (a=30, b=10, c=20, d=40) can be used to determine the intra-operator communication cost for different candidate split axes (i,j,k), as follows:

[0099] Operator node 1:

[0100]

[0101]

[0102] Operator node 2:

[0103]

[0104]

[0105] The processing order can be determined by comparing the lowest cost of each operator node and selecting the operator node associated with the highest compared cost. In this example, the second operator node can be processed first because cost(j) is the highest among the compared costs. Op2 =300 (representing the minimum cost of the second operator node) is greater than cost(i). Op1 =100 (representing the minimum cost of the first operator node).

[0106] Figure 16 An example of the partitioning axis of a tensor associated with matrix multiplication according to one embodiment is shown. The lowest intra-operator communication cost cost(j) based on the second operator node is also presented. Op2 =300, allowing us to select the splitting axis j for the second operator node. Therefore, the lower input matrix of the second operator node can be split into a left part 1601 and a right part 1602. The output matrix of the second operator node can be split in a similar manner.

[0107] Figure 17 An example of the partitioning axis of a tensor associated with another matrix multiplication according to one embodiment is shown. The lowest intra-operator communication cost cost(i) is based on the first operator node. Op1=100, allowing the selection of a splitting axis i for the first operator node. Typically, the splitting axis can be selected based on intra-operator communication cost and inter-operator communication cost. However, in this example, the tensor axc shared between the two operator nodes is in an abstract state, so selection can be based solely on intra-operator cost. If a tensor is unrelated to the selected splitting axis, it can be considered to be in an abstract state. For example, combining... Figures 11 to 13 As stated above, for matrix multiplication A*B=C, there can be three shared axes i, j, and k. Consider the partitioning axis i ( Figure 11 Matrix A and matrix C, with axis i, can be partitioned into two parts in a deterministic manner. However, matrix B is independent of axis i and can be partitioned later based on axis j or k without incurring different costs. The partitioning axis j or k can be determined later, for example, based on minimizing the communication cost between operators, e.g., after partitioning recursion. Similarly, in Figure 12 In the example, the chosen axis is j, therefore matrix A is in an abstract state. Figures 16 to 19 In the diagram, the diagonal dashed lines represent the abstract state of a tensor or subtensor. Figure 17 In the example, the matrix axc of the second operator node is in an abstract state, while the matrix axc of the first operator node is not in an abstract state.

[0108] The second operator node can be notified that the partition axis of its upper input tensor in the current recursion should be the same as the partition axis of the first operator node's output tensor. This information can be provided, for example, after all partitioning recursions. Therefore, the upper input matrix of the first operator node can be partitioned into an upper part 1701 and a lower part 1702. The output matrix of the first operator node can be partitioned in a similar manner.

[0109] After selecting the splitting axis for the first and second operator nodes, the first splitting recursion (inner loop) is completed. Since N=2, a second splitting recursion can be initiated to further split the subtensors obtained from the first splitting recursion. To determine the processing order of the second splitting recursion, the dimensions of the subtensors can be used to determine the intra-operator communication cost for different splitting axes (i,j,k), as follows:

[0110] Operator node 1:

[0111]

[0112]

[0113] Operator node 2:

[0114]

[0115]

[0116] Therefore, it can be determined that the processing order is the same, that is, in the second recursive process of the split, the second operator node can be processed first.

[0117] Figure 18 An example of the partitioning axis of a subtensor associated with matrix multiplication is shown according to one embodiment. The lowest intra-operator communication cost cost(i) based on the second operator node is also presented. Op2 =200, which allows you to select the splitting axis i for the second operator node in the second splitting recursion.

[0118] Figure 19 An example of the partitioning axis of a subtensor associated with another matrix multiplication according to one embodiment is shown. The lowest intra-operator communication cost cost(j) based on the first operator node is also shown. Op1 =75, the splitting axis j can be selected for the second operator node in the second splitting recursion. Based on Figure 18 The tensor axc in the second operator node is associated with two partitions, one with axis i and the other an abstract state. So far, similar to the second operator node, the first operator node is associated with a partition with axis i. Therefore, the abstract state still applies, allowing the use of intra-operator communication costs without inter-operator communication costs.

[0119] After selecting the splitting axis for the two operator nodes, the number of splitting recursions completed is two, therefore there are no other splitting recursions to be executed in the outer loop. If the devices can communicate with each other simultaneously and the network bandwidth is not saturated, the total communication cost can be determined, for example, based on the intra-operator communication cost of the operator nodes and the inter-operator communication cost between the operator nodes. If these devices communicate one by one, the total communication cost can be Subsequently, a final parallel computation scheme, including the defined splitting axis, can be output.

[0120] The following is combined with Figures 15 to 19 Provide an example of applying a memory cost model to determine the number of recursive partitioning operations. In this example, it is assumed that each parallel computing device can hold 1000 elements, which can be considered the memory requirement. Figure 15 In this example, the first operator node requires 1100 elements (ab + bc + ac), and the second operator node requires 2600 elements (ac + cd + ad). Since each parallel computing device requires 1000 elements, partitioning is unsuitable for parallel computing devices, thus allowing for further recursive partitioning (e.g., ...). Figure 10 (Outer ring) to distribute computing across more devices.

[0121] exist Figure 16 In this context, the first operator node still requires 1100 elements. The second operator node requires ac + cd / 2 + ad / 2 = 1600 elements, which still exceeds the memory requirement of 1000 elements. Figure 17 In the first operator node, the memory required is ab / 2 + bc + ac / 2 = 650 elements, which is less than the memory requirement. However, the memory required by the second operator node is still 1600. Therefore, the first operator node will meet the memory requirement, but the second operator node will not. Therefore, the device can determine to perform further partitioning. Figure 18 In the second operator node, the required memory is ac / 2 + cd / 2 + ad / 4 = 1000, which meets the memory requirements. Figure 19 The operations described can be used to balance the splitting of two operator nodes. Although the same memory requirement of 1000 elements is used in this example, it is understandable that parallel computing devices can have different memory requirements.

[0122] Figure 20 An example of a method 2000 for determining a parallel computing scheme for a neural network according to an embodiment is shown.

[0123] In 2001, the method may include: receiving a computational graph of a neural network.

[0124] In 2002, the method may include transforming a computation graph into a data flow graph comprising multiple recursive subgraphs. According to one example, each recursive subgraph may include another recursive subgraph and tuples of operator nodes, or an empty graph.

[0125] In 2003, the method may include: determining the number of recursive partitions based on the number of parallel computing devices.

[0126] In 2004, the method may include: for each split recursion, determining multiple costs corresponding to multiple operator nodes associated with the data flow graph.

[0127] In 2005, the method may include: for each recursive split, determining the processing order of multiple recursive subgraphs based on a descending order of multiple costs.

[0128] In 2006, the method could include processing multiple recursive subgraphs in a determined processing order for each recursive segmentation. According to one example, processing a recursive subgraph could include selecting a segmentation axis for the tensors associated with the operator nodes of the recursive subgraph.

[0129] In 2007, the method may include: outputting a segmentation scheme, wherein the segmentation scheme includes a segmentation axis for each tensor associated with a plurality of operator nodes.

[0130] Further features of the method are derived directly from the functions and parameters of the method and apparatus described in the appended claims and specification, and therefore will not be repeated here.

[0131] The example implementation provides an efficient and deterministic method for determining a parallel computing scheme for a neural network. For example, using the reference architecture ResNet 50, the time spent determining the parallel computing scheme can be significantly reduced. For instance, a 72x speedup can be achieved with 64 parallel computing devices compared to the reference solution. This approach may be even more beneficial as the number of parallel computing devices increases. For example, a 220x speedup can be achieved with 512 parallel computing devices. Therefore, the parallel efficiency of determining a parallel computing scheme for a neural network can be significantly improved.

[0132] An apparatus or system may be used to perform or cause to perform any aspect of one or more methods described herein. Further, a computer program may include program code for: causing to perform one aspect of one or more methods described herein, and then executing the computer program on a computer. Further, a computer program product may include a computer-readable storage medium having program code stored thereon, the program code including instructions for performing any aspect of one or more methods described herein. Further, an apparatus may include means for performing any aspect of one or more methods described herein. According to an example embodiment, the means includes at least one processor and a memory including program code. The program code is used to cause to perform any aspect of one or more methods when executed by at least one processor.

[0133] Any ranges or device values ​​given herein are expandable or modifiable without losing the desired effect. Furthermore, unless expressly prohibited, any embodiment may be combined with another embodiment.

[0134] Although the subject matter has been described in language specific to structural features and / or actions, it should be understood that the subject matter defined in the claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as examples of implementing the claims, and other equivalent features and actions are intended to fall within the scope of the claims.

[0135] It should be understood that the above advantages and benefits may relate to one embodiment or several embodiments. These embodiments are not limited to embodiments that solve any or all of the described problems, or those that have any or all of the described advantages and benefits. It is further understood that a reference to "a" may refer to one or more of those items. Furthermore, a reference to "at least one" or "one or more" may refer to one or more of those items.

[0136] The steps or operations described herein can be performed in any suitable order, or simultaneously where appropriate. Furthermore, individual boxes can be removed from any method without departing from the scope of the subject matter described herein. Aspects of any of the foregoing embodiments can be combined with aspects of any other described embodiments to form further embodiments without losing the desired effects.

[0137] As used herein, the term "including" means including the identified method, box, or element, but such boxes or elements do not constitute an exclusive list, and the method or device may include other boxes or elements.

[0138] Although a subject may be referred to as the "first" or "second" subject, this does not necessarily indicate any order or importance of these subjects. Rather, these attributes can be used simply to distinguish between subjects.

[0139] It should be understood that the above description is given by way of example only, and various modifications can be made by those skilled in the art. The foregoing specification, examples, and data provide a complete description of the structure and use of the exemplary embodiments. Although various embodiments have been described above with some degree of specificity or with reference to one or more individual embodiments, those skilled in the art can make various changes to the disclosed embodiments without departing from the scope of this specification.

Claims

1. An apparatus for determining a parallel computing scheme for a neural network, said parallel computing scheme being executed by a plurality of parallel computing devices, characterized in that, The device is used for: (a) Receive the computation graph of the neural network; (b) Transform the computation graph into a recursive data flow graph comprising multiple recursive subgraphs, wherein each recursive subgraph is a tuple of another recursive subgraph and operator nodes; (c) Determine the number of recursive partitions based on the number of parallel computing devices; (d) Perform the predetermined number of recursive splits, wherein for each recursive split, (i) For at least the first recursion in the multiple segmentation recursions, Determine multiple costs corresponding to multiple operator nodes associated with the data flow graph; and Based on the order of the multiple costs from high to low, the processing order of the multiple recursive subgraphs is determined; (ii) Processing the plurality of recursive subgraphs according to the determined processing order, wherein processing each recursive subgraph includes: selecting a splitting axis for a tensor associated with an operator node of the recursive subgraph, thereby splitting the tensor into a plurality of mutually constrained subtensors by the selected splitting axis, wherein after outputting a splitting scheme including the splitting axis of each tensor associated with the plurality of operator nodes, each of the plurality of subtensors will be processed in parallel by different computing devices in the plurality of parallel computing devices, and splitting the tensor into a plurality of subtensors includes splitting the tensor according to a sample, parameter or attribute; (e) Output the partitioning scheme, wherein the partitioning scheme includes one or more partitioning axes for each tensor associated with the plurality of operator nodes; The device is also used for: The partitioning axis is selected for the tensor associated with the operator node based on the intra-operator communication cost, wherein the intra-operator communication cost includes the amount of data to be transmitted between the parallel computing devices for the operator node, and different amounts of data transmitted correspond to different partitioning axes.

2. The device according to claim 1, characterized in that, The device is also used for: Determine the number of recursive steps for the segmentation, such that 2 N It is equal to the number of parallel computing devices, where N is the number of times the partitioning recursion is performed.

3. The device according to claim 1 or 2, characterized in that, The device is also used for: Based on the amount of data to be transmitted between the parallel computing devices for each operator node, the costs corresponding to the multiple operator nodes are determined.

4. The device according to claim 1, characterized in that, The device is also used for: If no splitting axis is determined for adjacent operator nodes in the current splitting recursion, then the splitting axis is selected for the tensor associated with the operator node based on the intra-operator communication cost.

5. The device according to claim 1, characterized in that, The device is also used for: Based on inter-operator communication costs, the splitting axis is selected for the tensor associated with the operator node, wherein the inter-operator communication costs include the amount of data to be transmitted by the parallel computing devices when executing the adjacent operator node based on the shared tensor between the operator node and the adjacent operator node or when executing the operator node based on the output of the adjacent operator node.

6. The device according to claim 1 or 5, characterized in that, The device is also used for: If at least one splitting axis is determined for an adjacent operator node or another adjacent operator node in the current splitting recursion, then the splitting axis is selected for the tensor associated with the operator node based on the intra-operator communication cost and the inter-operator communication cost.

7. The device according to claim 1 or 2, characterized in that, The device is also used for: Determine whether the partitioning scheme meets at least one memory requirement associated with the parallel computing device; and If it is determined that the partitioning scheme meets the at least one memory requirement associated with the parallel computing device, then the partitioning scheme is output.

8. A method for determining a parallel computing scheme for a neural network, wherein, The parallel computing scheme is executed by multiple parallel computing devices, characterized in that the method includes: (a) Receive the computation graph of the neural network; (b) Transform the computation graph into a recursive data flow graph comprising multiple recursive subgraphs, wherein each recursive subgraph is a tuple of another recursive subgraph and operator nodes; (c) Determine the number of recursive partitions based on the number of parallel computing devices; (d) Perform the predetermined number of recursive splits, wherein for each recursive split, (i) For at least the first recursion in the multiple segmentation recursions, Determine multiple costs corresponding to multiple operator nodes associated with the data flow graph; and Based on the order of the multiple costs from high to low, the processing order of the multiple recursive subgraphs is determined; (ii) Processing the plurality of recursive subgraphs according to the determined processing order, wherein processing each recursive subgraph includes: selecting a splitting axis for a tensor associated with an operator node of the recursive subgraph, thereby splitting the tensor into a plurality of mutually constrained subtensors by the selected splitting axis, wherein after outputting a splitting scheme including the splitting axis of each tensor associated with the plurality of operator nodes, each of the plurality of subtensors will be processed in parallel by different computing devices in the plurality of parallel computing devices, and splitting the tensor into a plurality of subtensors includes splitting the tensor according to a sample, parameter or attribute; (e) Output the partitioning scheme, wherein the partitioning scheme includes one or more partitioning axes for each tensor associated with the plurality of operator nodes; The method further includes: The partitioning axis is selected for the tensor associated with the operator node based on the intra-operator communication cost, wherein the intra-operator communication cost includes the amount of data to be transmitted between the parallel computing devices for the operator node, and different amounts of data transmitted correspond to different partitioning axes.

9. The method according to claim 8, characterized in that, Also includes: Determine the number of recursive steps for the segmentation, such that 2 N It is equal to the number of parallel computing devices, where N is the number of times the partitioning recursion is performed.

10. The method according to claim 8 or 9, characterized in that, Also includes: Based on the amount of data to be transmitted between the parallel computing devices for each operator node, the costs corresponding to the multiple operator nodes are determined.

11. The method according to claim 8, characterized in that, Also includes: If no splitting axis is determined for adjacent operator nodes in the current splitting recursion, then the splitting axis is selected for the tensor associated with the operator node based on the intra-operator communication cost.

12. The method according to claim 8, characterized in that, Also includes: Based on inter-operator communication costs, the splitting axis is selected for the tensor associated with the operator node, wherein the inter-operator communication costs include the amount of data to be transmitted by the parallel computing devices when executing the adjacent operator node based on the shared tensor between the operator node and the adjacent operator node or when executing the operator node based on the output of the adjacent operator node.

13. The method according to claim 8 or 12, characterized in that, Also includes: If at least one splitting axis is determined for an adjacent operator node or another adjacent operator node in the current splitting recursion, then the splitting axis is selected for the tensor associated with the operator node based on the intra-operator communication cost and the inter-operator communication cost.

14. The method according to claim 8 or 9, characterized in that, Also includes: Determine whether the partitioning scheme meets at least one memory requirement associated with the parallel computing device; as well as If it is determined that the partitioning scheme meets the at least one memory requirement associated with the parallel computing device, then the partitioning scheme is output.

15. A computer program, comprising program code, characterized in that, The program code is used to: when the computer program is executed on a computer, cause the method according to any one of claims 8 to 14 to be performed.

16. A computer program product comprising a computer-readable storage medium for storing program code thereon, characterized in that, The program code includes instructions for performing the method according to any one of claims 8 to 14.

Images