Computational graph processing method, device, readable medium and electronic device

Abstract

The present disclosure relates to a method, device, readable medium and electronic device for processing a computational graph. The method for processing a computational graph obtains a target computational graph corresponding to a machine learning model to be trained, performs node fusion processing on the target computational graph to obtain a fused computational graph including multiple subgraphs; according to multiple preset segmentation methods of each operator node, effectively segments the operator nodes in the fused computational graph to obtain multiple segmentation data flow graphs corresponding to the fused computational graph, and the segmentation data flow graph is a target computational graph with an effective segmentation method; then, a parallel configuration strategy corresponding to the target computational graph is searched according to the segmentation data flow graph, so that a model training strategy that is a hybrid of pipeline parallelism, data parallelism and model parallelism can be automatically and efficiently generated, which can effectively improve the model training efficiency of the machine learning model and shorten the model training time.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular, to a method, device, readable medium and electronic device for processing a computational graph. Background Art

[0002] As a powerful tool for task automation, machine learning can automate tasks in various types of applications. That is why in recent years, machine learning, especially deep neural networks, has been widely used in various challenging task processing, such as computer vision, natural language processing, speech recognition, etc., and has achieved major breakthroughs. However, with the further deepening of neural network applications, the parameters and computational complexity of neural network models are also increasing, and the training time of neural network models is also extended. As a result, how to efficiently train neural network models has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] The present disclosure provides a method, device, readable medium and electronic device for processing a computational graph.

[0005] In a first aspect, the present disclosure provides a method for processing a computational graph, the method comprising:

[0006] Obtain a target computation graph corresponding to the machine learning model to be trained, wherein the target computation graph includes an operator node and a directed edge connecting two operator nodes;

[0007] Performing node fusion processing on the target computation graph to obtain a fused computation graph including multiple subgraphs;

[0008] According to a plurality of preset segmentation methods of each operator node, the operator nodes in the fusion calculation graph are effectively segmented to obtain a plurality of segmentation data flow graphs corresponding to the fusion calculation graph, wherein the segmentation data flow graph is a target calculation graph with an effective segmentation method;

[0009] The parallel configuration strategy corresponding to the target computational graph is searched according to the partitioned data flow graph, wherein the parallel configuration strategy includes a pipeline parallel set, a data parallel set and a model parallel set, and the parallel configuration strategy is used for parallel training of the machine learning model.

[0010] In a second aspect, the present disclosure provides a computing graph processing device, the device comprising:

[0011] An acquisition module is configured to acquire a target computation graph corresponding to the machine learning model to be trained, wherein the target computation graph includes an operator node and a directed edge connecting two operator nodes;

[0012] A graph fusion module is configured to perform node fusion processing on the target computation graph to obtain a fused computation graph including multiple subgraphs;

[0013] A splitting module is configured to effectively split the operator nodes in the fusion computing graph according to a plurality of preset splitting methods for each operator node, so as to obtain a plurality of split data flow graphs corresponding to the fusion computing graph, wherein the split data flow graph is a target computing graph with an effective splitting method;

[0014] The search module is configured to search for the parallel configuration strategy corresponding to the target computational graph according to the partitioned data flow graph, wherein the parallel configuration strategy includes a pipeline parallel set, a data parallel set and a model parallel set, and the parallel configuration strategy is used for parallel training of the machine learning model.

[0015] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect above.

[0016] In a fourth aspect, the present disclosure provides an electronic device, including:

[0017] a storage device having a computer program stored thereon;

[0018] A processing device is used to execute the computer program in the storage device to implement the steps of the method described in the first aspect above.

[0019] The above technical scheme obtains the target computation graph corresponding to the machine learning model to be trained, wherein the target computation graph includes operator nodes and directed edges connecting two operator nodes; performs node fusion processing on the target computation graph to obtain a fused computation graph including multiple subgraphs; effectively splits the operator nodes in the fused computation graph according to multiple preset splitting methods of each operator node to obtain multiple split data flow graphs corresponding to the fused computation graph, wherein the split data flow graph is a target computation graph with an effective splitting method; then searches for the parallel configuration strategy corresponding to the target computation graph according to the split data flow graph, wherein the parallel configuration strategy includes a pipeline parallel set, a data parallel set and a model parallel set, wherein the parallel configuration strategy is used for parallel training of the machine learning model, and can automatically and efficiently generate a model training strategy that is a hybrid of pipeline parallelism, data parallelism and model parallelism, which can effectively improve the model training efficiency of the machine learning model and shorten the model training time.

[0020] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0022] Figure 1 is a flowchart of a method for processing a computational graph shown in an exemplary embodiment of the present disclosure;

[0023] Figure 2a is a schematic diagram of node fusion processing shown in an exemplary embodiment of the present disclosure;

[0024] Figure 2b This is an example diagram of a data flow graph segmentation diagram shown in an example embodiment of the present disclosure;

[0025] Figure 3 is based on Figure 1 A flowchart of a method for processing a computational graph is shown in the illustrated embodiment;

[0026] Figure 4 is a schematic diagram of a directed edge annotation shown in an exemplary embodiment of the present disclosure;

[0027] Figure 5 is based on Figure 1 A flowchart of another method for processing a computational graph shown in the illustrated embodiment;

[0028] Figure 6 is a schematic diagram of coarse-grained processing of operator nodes shown in an exemplary embodiment of the present disclosure;

[0029] Figure 7 is a block diagram of a computing graph processing device shown in an exemplary embodiment of the present disclosure;

[0030] Figure 8 It is a block diagram of an electronic device shown in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0032] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0033] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0034] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0035] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0036] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0037] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0038] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0039] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0040] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0041] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0042] Before introducing the specific implementation methods of the present disclosure in detail, the application scenarios of the present disclosure are first described as follows. The present disclosure can be applied to the process of determining the parallel training strategy of a machine learning model. The machine learning model can be a neural network model, a decision tree model, or a Bayesian algorithm model, etc. Taking a neural network as an example, in order to improve the training efficiency of the machine learning model, the neural network is usually trained in parallel on multiple hardware accelerators (such as GPU (graphics processing unit) and TPU (Tensor Processing Unit). There are currently three main parallel training methods, namely: Data Parallel, Model Parallel, and Pipeline Parallel. Among them, data parallelism refers to dividing the data into several blocks and mapping them to different processors respectively. Each processor runs the same processing program to process the assigned data, that is, the training data set is divided and mapped on multiple computing devices. Each computing device has a complete model. The computing devices use some communication primitives to converge the global gradient and update the parameters. Model parallelism is the parameters of the neural network model (such as Linear / Dense Layer) to split a single large model parameter to multiple computing devices for parallel computing. Pipeline parallelism uses the stacked layer characteristics of neural networks to segment the entire deep learning model. Different segments are on different devices, and the front and back stages are pipelined in batches to work in parallel in a relay manner. A distributed training of a deep neural network can use only one parallel method or a mixture of different parallel methods (for example, for a convolutional classification network, the first few layers of the network use data parallelism, and the last layer uses model parallelism).

[0043] In the related art, when determining the parallel scheme for model training, it is either manually set or two of the parallel methods are mixed. It is impossible to automatically achieve comprehensive mixed parallelism of the three parallel methods. Therefore, it is not conducive to better improving the training efficiency of the neural network model, and cannot further shorten the training time of the neural network model.

[0044] In order to solve the above technical problems, the present disclosure provides a method, device, readable medium and electronic device for processing a computational graph. The method for processing the computational graph obtains a target computational graph corresponding to a machine learning model to be trained, performs node fusion processing on the target computational graph, so as to obtain a fused computational graph including multiple subgraphs; according to multiple preset segmentation methods of each operator node, the operator nodes in the fused computational graph are effectively segmented to obtain multiple segmentation data flow graphs corresponding to the fused computational graph, and the segmentation data flow graph is a target computational graph with an effective segmentation method; then, according to the segmentation data flow graph, a parallel configuration strategy corresponding to the target computational graph is searched, and the parallel configuration strategy includes a pipeline parallel set, a data parallel set and a model parallel set. In this way, a model training strategy that is a hybrid of pipeline parallelism, data parallelism and model parallelism can be automatically and efficiently generated, which can effectively improve the model training efficiency of the machine learning model and shorten the model training time.

[0045] The technical solution of the present disclosure is described in detail below in conjunction with specific embodiments.

[0046] Figure 1 is a flowchart of a method for processing a computational graph shown in an exemplary embodiment of the present disclosure; Figure 1 As shown, the method may include:

[0047] Step 101, obtain a target computation graph corresponding to the machine learning model to be trained, where the target computation graph includes an operator node and a directed edge connecting two operator nodes.

[0048] The processing method of the computational graph can be applied to a terminal or a server, and the machine learning model can be a neural network model, a decision tree model or a Bayesian algorithm model, or other machine learning models in the prior art. The operator node can be a convolution operator node, an activation function node, an addition operator node, a subtraction operator node, a multiplication operator node, a division operator node, a power operator node, a logarithm operator node, a function operator node, an array operator node, a limit operator node, a derivative operator node, an integral operator node, etc. The directed edge can represent the direction of data transmission between two operator nodes.

[0049] In this step, the target computation graph may be obtained through a preset interface, for example, by accessing an external storage device through a preset interface and reading the target computation graph from the external storage device.

[0050] Step 102: Perform node fusion processing on the target computation graph to obtain a fused computation graph including multiple subgraphs.

[0051] In this step, a preset dynamic graph fusion algorithm can be used to minimize the amount of communication data between the multiple subgraphs and balance the amount of calculation corresponding to different subgraphs as the optimization goal, and each subgraph can be used as a closure graph as a constraint to determine the multiple target subgraphs corresponding to the target calculation graph, so as to obtain the fused calculation graph including the multiple target subgraphs.

[0052] Among them, the preset dynamic graph fusion algorithm can be a Kernighan-Lin (KL) algorithm, and the amount of communication data between multiple subgraphs is the amount of data generated when different subgraphs interact with each other. The balance of the amount of calculation corresponding to different subgraphs can be determined by the difference in the calculation time corresponding to different subgraphs. For example, when the difference in the calculation time of any two subgraphs is less than or equal to the preset difference threshold, the balance of the amount of calculation corresponding to different subgraphs is determined. The closure graph is used to characterize a subgraph that does not need to communicate with other subgraphs during execution except for input and output.

[0053] For example, Figure 2a is a schematic diagram of node fusion processing shown in an exemplary embodiment of the present disclosure, such as Figure 2a As shown in Figure 1, the entire computational graph from the first operator node (dateinput, data input node) to the last operator node (loss, loss calculation node) can be divided into three subgraphs, where Figure 1 The output of is used as the input of subgraph 2, and the output of subgraph 2 is used as the input of subgraph 2. Figure 3 Input, sub Figure 1 , sub-graph 2, sub-graph Figure 3 are all closure graphs, and Figure 1 , sub-graph 2 and sub-graph Figure 3 neutron Figure 1 The difference between the calculation time of sub-graph 1 and the calculation time of sub-graph 2 is less than the preset difference threshold, and the calculation time of sub-graph 2 is Figure 3 The difference in the calculated duration is less than the preset difference threshold, Figure 1 The calculation time and sub Figure 3 The difference in the calculated duration is less than the preset difference threshold.

[0054] Step 103, according to a plurality of preset segmentation methods of each operator node, the operator nodes in the fusion calculation graph are effectively segmented to obtain a plurality of segmentation data flow graphs corresponding to the fusion calculation graph.

[0055] Among them, the split data flow graph is a target calculation graph with a valid splitting method. The preset splitting method can be a legal splitting (valid splitting) of any dimension, such as no splitting (indicating copying the tensor), splitting along the first dimension of the tensor, splitting along the second dimension of the tensor or splitting in other dimensions, or mixed dimension splitting (for example, the first input tensor of the matrix multiplication operator is split into the second dimension, and the second input tensor is split into the first dimension). It can also be other feasible splitting methods in the prior art. When splitting, it may be necessary to aggregate the calculation results of each slice through collective communication. As long as the semantics of the operator can be guaranteed to be the same after splitting (that is, the calculation results before and after splitting are the same), it can be regarded as a legal splitting (valid splitting). In order to encode the splitting method into the data flow graph, we convert the target calculation graph into a form to obtain the data flow graph corresponding to the target calculation graph, and add the splitting dimension attribute to the data flow graph to form a split data flow graph. The nodes in the split data flow graph are the input tensors and output tensors corresponding to the operator nodes in the fusion calculation graph, and the nodes in the split data flow graph carry the dimension attributes of the tensor splittable.

[0056] For example, the addition operator has two input tensors and one output tensor. The dimensions that each tensor can be split into include (one-dimensional splitting, two-dimensional splitting, and three-dimensional splitting). Then, Figure 2b ( Figure 2b This is an example diagram of a data flow graph segmentation diagram shown in an example embodiment of the present disclosure; all tensors associated with the addition operator node (tensor node 1 and tensor node 2, tensor node 3), as well as the connections between all tensors associated with the addition operator node, are labeled in the labeling method shown in FIG. Different wiring methods represent different segmentation methods of the addition operator, for example, the nodes of the segmentation data flow graph represent (tensor, segmentable dimensions), and the lines represent (operator, segmentation method). For example, if a legal segmentation method of a certain operator is to segment along the second dimension of the first tensor and along the first dimension of the second tensor, then we add a line between the two nodes (tensor 1, dimension 2) and (tensor 2, dimension 1), that is, use the line to represent the legal segmentation method of segmenting along the second dimension of the first tensor and along the first dimension of the second tensor. Different preset segmentation methods can be labeled in different ways, such as using different line types (taking the addition operator node as an example, such as Figure 2bAs shown, the one-dimensional split of the addition operator node is marked with a dotted line, where the one-dimensional splitting method corresponding to the addition operator node means that each tensor of the addition operator node is split according to the first dimension, the two-dimensional split is marked with a solid line, that is, each tensor of the addition operator node is split according to the second dimension, and the three-dimensional split is marked with a dashed line, that is, each tensor of the addition operator node is split according to the third dimension), marked with different colors (for example, the one-dimensional split of the addition operator node is marked with red, and the two-dimensional split of the addition operator node is marked with blue), and can also be marked with different line widths of the same line type (for example, the one-dimensional split of the addition operator node is marked with a line width of 0.5 cm, and the two-dimensional split of the addition operator node is marked with a line width of 0.7 cm).

[0057] In this step, the operator nodes in the fusion calculation graph can be divided respectively according to the multiple preset division methods corresponding to each operator node, and then each operator node in the fusion calculation graph is traversed in turn through the preset effective division judgment strategy, and invalid division methods are eliminated to obtain the effective division method corresponding to the fusion calculation graph, thereby obtaining the division data flow graph with the effective division method.

[0058] It should be noted that the preset effective splitting judgment strategy may include that two operator nodes with direct data dependency relationships need to be split in the same splitting method. For example, operator node A is an addition operator, and the addition operator is split in a one-dimensional splitting method, that is, the input tensor and output tensor of operator node A are both split into one-dimensional arrays. If the output of operator node A is used as the input of the subtraction operator node (operator node B), then the splitting method of the subtraction operator node should be one-dimensional splitting, and other operator nodes with direct data dependency relationships with the subtraction operator node (for example, operator node C provides another subtraction parameter to the operator node B) should also be split in a one-dimensional splitting method. In this way, each operator node is traversed to determine the effective splitting method of each operator node in the fusion calculation graph, thereby obtaining the split data flow graph. It should be emphasized that the effective splitting of each operator node is common knowledge in the prior art, and the present disclosure will not repeat it here.

[0059] Step 104, searching for a parallel configuration strategy corresponding to the target computational graph according to the segmented data flow graph, wherein the parallel configuration strategy includes a pipeline parallel set, a data parallel set, and a model parallel set, and the parallel configuration strategy is used for parallel training of the machine learning model.

[0060] In this step, each of the split data flow graphs can be reduced to obtain multiple coarse-grained pending data flow graphs corresponding to the multiple split data flow graphs; according to the pending data flow graphs, the pipeline parallel set, data parallel set and model parallel set corresponding to the target computation graph are determined by a dynamic programming algorithm. The dynamic programming algorithm can be a two-layer dynamic programming algorithm in the prior art.

[0061] Through the scheme shown in the above steps 101 to 104, a model training strategy that is a hybrid of pipeline parallelism, data parallelism, and model parallelism can be automatically and efficiently generated, which can effectively improve the model training efficiency of the machine learning model and shorten the model training time.

[0062] Figure 3 is based on Figure 1 The embodiment shown is a flowchart of a method for processing a computational graph; Figure 3 As shown, Figure 1 The step 103 described in step 104 of effectively segmenting the operator nodes in the fusion calculation graph according to the multiple preset segmentation methods of each operator node to obtain multiple segmentation data flow graphs corresponding to the fusion calculation graph may include:

[0063] Step 1031, the input tensor and output tensor of each operator node in the fusion calculation graph are represented as tensor nodes.

[0064] The implementation method of this step can be found in the above Figure 2b The related description shown in the figure will not be repeated in this disclosure.

[0065] Step 1032, add the divisible dimension attribute of the tensor to the tensor node, and use different connections between tensor nodes to represent multiple preset splitting methods of the operator node to obtain the splitting data flow graph.

[0066] Among them, each preset segmentation method corresponding to each operator node includes a segmentation dimension attribute for each tensor in the operator node.

[0067] For example, if the multiple preset splitting methods corresponding to the addition operator node include splitting the first dimension (i.e., each tensor of the addition operator node is split according to the first dimension), splitting the second dimension (i.e., each tensor of the addition operator node is split according to the second dimension), and splitting the third dimension (i.e., each tensor of the addition operator node is split according to the third dimension), then the addition operator node in the fusion calculation graph can be split by splitting the first dimension, splitting the second dimension, and splitting the third dimension respectively. If the multiple preset splitting methods corresponding to the matrix multiplication operator node include splitting the first input tensor by the second dimension and the second input tensor by the first dimension, then the matrix multiplication operator node in the fusion calculation graph can be split by this splitting method.

[0068] The above step 1032 can be implemented by the following method shown in S11 to S14:

[0069] S11, use the same marking method to mark the connections between all tensor nodes associated with the same operator node that is segmented using the same segmentation method.

[0070] The connection can be used to characterize the properties of the splitting method. The tensors associated with an operator node include all input tensors and all output tensors of the operator.

[0071] S12, using different notation methods to distinguish the connections of each operator corresponding to different segmentation methods.

[0072] In this step, the connection line may be labeled according to the splitting mode attribute of each operator node.

[0073] For example, different segmentation attributes can be marked with different line types, for example Figure 4 As shown ( Figure 4 This is a schematic diagram of edge annotation shown in an exemplary embodiment of the present disclosure.) The lines of the first segmentation method are annotated with solid lines, the lines of the second segmentation method are annotated with dashed lines, the lines of the third segmentation method are annotated with dotted lines, etc. The same line may be annotated with multiple annotation methods at the same time; lines with different segmentation method attributes may also be colored with different colors, that is, lines with different segmentation method attributes may be annotated with different colors, for example, lines of the first segmentation method are annotated with red, lines of the second segmentation method are annotated with blue, lines of the third segmentation method are annotated with green, etc.; lines with different segmentation method attributes may also be annotated with different line widths of the same line type. For example, the lines of the first segmentation method are annotated with a line width of 0.5 cm, the lines of the second segmentation method are annotated with a line width of 0.7 cm, the lines of the third segmentation method are annotated with a line width of 0.9 cm, etc. Among them, the first splitting method can be that all tensor nodes associated with an operator node are split according to the first dimension, the second splitting method can be that the first tensor node associated with an operator node is split according to the first dimension, and the second tensor node is split according to the second dimension, and the third splitting method can be that all tensor nodes associated with an operator node are split according to the second dimension. The specific assignments in the above examples are only used to illustrate how to use different annotation methods to distinguish the connections corresponding to different splitting methods for each operator node, and are not used to limit the specific protection scope.

[0074] S13, determining multiple valid segmentation methods corresponding to the fusion calculation graph according to the connecting lines marked with the segmentation methods.

[0075] Among them, each operator node in the fusion calculation graph is traversed in turn, and the connection lines of each annotation method of different operator nodes are combined to obtain multiple effective segmentation methods.

[0076] S14, taking each effective segmentation method corresponding to the fusion calculation graph as a segmentation data flow graph to obtain a plurality of the segmentation data flow graphs.

[0077] For example, if a fusion computing graph includes operator node 1, operator node 2 to operator node 50, the 50 operator nodes include an addition operator node, a subtraction operator node, a multiplication operator node, and a division operator node, wherein multiple preset segmentation methods of the addition operator node, the subtraction operator node, the multiplication operator node, and the division operator node are pre-set (for example, one-dimensional segmentation, two-dimensional segmentation, and three-dimensional segmentation). Then each operator node is split in a one-dimensional splitting method, a two-dimensional splitting method, and a three-dimensional splitting method, and then each operator node is traversed, and at least a split data flow graph in which all operator nodes are split in a one-dimensional splitting method, a split data flow graph in which all operator nodes are split in a two-dimensional splitting method, and a split data flow graph in which all operator nodes are split in a three-dimensional splitting method can be obtained. Other split data flow graphs that are a mixture of the three splitting methods may also be obtained. Since different computational graph topologies correspond to different legal splitting methods, the present disclosure does not limit which specific split data flow graphs are obtained.

[0078] The above technical scheme uses the same annotation method to mark the connections between all tensor nodes associated with the same operator node that is split using the same splitting method; uses different annotation methods to distinguish the connections corresponding to different splitting methods of each operator node, and then determines the multiple valid splitting methods corresponding to the fused calculation graph based on the connections marked with the splitting methods; uses each valid splitting method corresponding to the fused calculation graph as a splitting data flow graph to obtain multiple splitting data flow graphs, which can effectively improve the speed of obtaining the effective splitting methods of the entire calculation graph, and is conducive to improving the efficiency of generating a comprehensive hybrid parallel model training strategy.

[0079] Figure 5 is based on Figure 1 The embodiment shown is a flowchart of another method for processing a computational graph; Figure 5 As shown, Figure 1 The step 104 of searching the parallel configuration strategy corresponding to the target computation graph according to the segmented data flow graph, wherein the parallel configuration strategy includes a pipeline parallel set, a data parallel set, and a model parallel set, may include the following steps:

[0080] Step 1041, performing reduction processing on each of the split data flow graphs to obtain multiple coarse-grained pending data flow graphs corresponding to the multiple split data flow graphs.

[0081] In this step, a possible implementation method is to reduce the segmented data flow graph through a preset reduction algorithm to obtain the pending data flow graph. The preset reduction algorithm can be any data reduction algorithm in the prior art. There are many data reduction algorithms involved in existing patents and papers, which will not be described in detail in this disclosure.

[0082] Another possible implementation may include the following steps S1 to S3:

[0083] S1, obtain the calculation time attribute corresponding to each operator node in the target calculation graph.

[0084] The operator nodes in the target computation graph are operator nodes without a splitting mode attribute, and the computation time attribute is the execution time of each operator node when it is not split.

[0085] It should be noted that current deep learning frameworks usually come with built-in tools, which can be used to measure the execution time and memory consumption of each operator node, as well as the amount of data represented by each directed edge. In the process of implementing S1, built-in tools can be used to measure the computing time attributes corresponding to each operator node in the target computational graph.

[0086] S2, determining the consumption time attribute corresponding to each operator node in the split data flow graph according to the calculation time attribute.

[0087] Among them, according to the segmentation method of each operator node in the target calculation graph and the calculation time attribute, the target calculation time attribute of the node operator in the segmented data flow graph is determined through a preset mapping relationship; the communication time attribute corresponding to each operator node in the segmented data flow graph is obtained; and the consumed time attribute is determined according to the target calculation time attribute and the communication time attribute.

[0088] It should be noted that the preset mapping relationship is used to characterize the functional relationship between the calculation time attribute of each operator node in the target calculation graph when it is not split and the target calculation time attribute of the operator node after being split according to the preset splitting method. The functional relationship can be a linear relationship (for example, a proportional relationship, a linear function relationship, a quadratic function relationship, etc.) or a nonlinear relationship.

[0089] For example, if the execution time (i.e., the computing time attribute) of operator node A in the target computational graph is 20 seconds when it is not split, and the operator node A is split into two halves of the same dimension, then the target computing time attribute corresponding to the split operator node A can be half of 20 seconds, that is, after each operator node is split, the computing time of each operator node can be adjusted proportionally.

[0090] In addition, it should be noted that the communication time attribute corresponding to the operator node can be obtained in the following ways: one way to obtain it is to test-run the fusion calculation graph corresponding to the target calculation graph in the standby device topology network to obtain the communication time of each operator node, that is, to obtain the communication time attribute of the operator node; another way to obtain it is to first measure the data volume of each operator node through the built-in tools of the deep learning framework, and then calculate the communication time required to transmit the data volume according to the preset communication speed, so as to obtain the communication time attribute of the operator node.

[0091] Furthermore, the above-mentioned implementation method of determining the consumed time attribute based on the target calculation time attribute and the communication time attribute may be to use the sum of the attribute value corresponding to the target calculation time attribute (i.e., the execution time after segmentation) and the attribute value corresponding to the communication time attribute (communication time) as the attribute value of the consumed time attribute.

[0092] S3, coarse-grained processing is performed on the operator nodes in the split data flow graph according to the consumption time attribute corresponding to each operator node to obtain the pending data flow graph.

[0093] For example, Figure 6 is a schematic diagram of coarse-grained processing of operator nodes shown in an exemplary embodiment of the present disclosure, such as Figure 6 As shown in the left figure of the first row in j To operator node V i Then to the operator node V k , if the operator node V j To operator node V i The attribute value of the consumption time attribute is COST ji , operator node V i To operator node V k The attribute value of the consumption time attribute is COST ik , then the operator node V j To operator node V i Then to the operator node V k Can be reduced to operator node V j To operator node V k , operator node V j To operator node V k The corresponding consumption time attribute is updated to COST ji +COST ik Similarly, according to the principle that the corresponding attribute values ​​of the consumption time attribute are equivalent, the coarse-grained processing of other operator nodes can be realized, such as Figure 6 The first row on the right can be used to convert the operator node V in the original computation graph i To operator node V jThe two directed edges in are coarse-grained into one directed edge, and the consumption time attribute corresponding to the coarse-grained directed edge is equal to the sum of the consumption time attribute values ​​of the two directed edges before the coarse-graining.

[0094] The steps shown in S1 to S3 above can effectively obtain the pending data flow graph after coarse-grained processing, which is conducive to providing a reliable data basis for the generation of subsequent parallel configuration strategies.

[0095] Step 1042, determine the pipeline parallel set, data parallel set and model parallel set corresponding to the target computation graph through a dynamic programming algorithm based on the pending data flow graph.

[0096] In this step, it can be implemented through the following steps S4 to S6, which are as follows:

[0097] S4, constructing a preset function model according to the preset split quantity and the pending data flow graph,

[0098] Among them, the preset function model is used to characterize the correspondence between different segmentation methods and splitting methods and the total time consumption for training the machine learning model.

[0099] S5, finding the optimal solution of the preset function model to obtain the optimal splitting method and the optimal segmentation method under the optimal splitting method.

[0100] Among them, the memory consumption corresponding to each group of solutions can be obtained; the memory consumption is less than or equal to the preset memory consumption threshold, and the target solution when the preset function model takes the minimum function value is taken as the optimal solution.

[0101] For example, let S represent the set of parallel strategy solutions, S = {(p1, t1, m1), (p2, t2, m2), ...}, where p1 represents the operator splitting strategy of the first parallel branch in the parallel strategy, t1 represents the operator execution time corresponding to the operator splitting strategy of the first parallel branch, and m1 represents the operator memory consumption corresponding to the operator splitting strategy of the first parallel branch; p2 represents the operator splitting strategy of the second parallel branch in the parallel strategy, t2 represents the operator execution time corresponding to the operator splitting strategy of the second parallel branch, and m2 represents the operator memory consumption corresponding to the operator splitting strategy of the second parallel branch. Assume that there are two sets S and S', each of size N and N', and define the Cartesian product operation for S to merge the solutions:

[0102] Prod(S,S′)=U 1≤n≤N,1≤n′≤N′ {[p,p′],t n +t n ′,m n +m n ′}

[0103] It should be noted that in the process of searching for the optimal solution, the solution set can be initialized to S first, and the M 2 The subgraph uses the dynamic programming algorithm to solve the pipeline parallelism, and updates S according to the outer dynamic programming. The subprocess will be called during the solution. The dynamic programming algorithm is used to solve the data parallelism and model parallelism for each reduced subgraph. The complexity of the outer dynamic programming of the entire search algorithm is O(N 2 M 2 ), the inner layer dynamic programming complexity is O(n 2 K 2 log(K)(log(n)+log(K))), so the overall complexity is O(N 2 M 2 n 2 K 2 log(K)(log(n)+log(K))), where M is the number of devices, S is the solution, N is the number of subgraphs after graph fusion, n is the number of subgraph operators, and K is the number of parallel strategies for a single operator.

[0104] S6, split the target computation graph into multiple pipeline parallel sets according to the optimal splitting method, and determine the data parallel set and the model parallel set in each of the pipeline parallel sets according to the optimal splitting method.

[0105] It should be noted that after obtaining the optimal solution of the preset function model, the reduction graph can be expanded to record the data parallel and model parallel solutions of the original calculation graph, and the fusion graph can be expanded to record the pipeline parallel solution of the original calculation graph, so as to obtain the optimal pipeline parallel set, data parallel set and model parallel set.

[0106] For example, if the optimal splitting method for operator node 1 to operator node 50 is operator node 1 to operator node 15, operator node 16 to operator node 23, operator node 24 to operator node 37, and operator node 38 to operator node 50, then the multiple pipeline parallel sets are: the computational graph set corresponding to operator node 1 to operator node 15, the computational graph set corresponding to operator node 16 to operator node 23, the computational graph set corresponding to operator node 24 to operator node 37, and the computational graph set corresponding to operator node 38 to operator node 50. The data parallel set includes the splitting method of each operator node from operator node 1 to operator node 15, the splitting method of each operator node from operator node 16 to operator node 23, the splitting method of each operator node from operator node 24 to operator node 37, and the splitting method of each operator node from operator node 38 to operator node 50. The model parallel set includes the splitting method of the relevant parameters of the model parallelism from operator node 1 to operator node 15, the splitting method of the relevant parameters of the model parallelism from operator node 16 to operator node 23, the splitting method of the relevant parameters of the model parallelism from operator node 24 to operator node 37, and the splitting method of the relevant parameters of the model parallelism from operator node 38 to operator node 50.

[0107] It should be noted that in the above process of obtaining the optimal splitting method and the optimal segmentation method under the optimal splitting method, a dynamic programming algorithm can be used. The dynamic programming algorithm is a mature algorithm in the prior art. The specific implementation details of the solution can refer to the relevant description of the dynamic programming algorithm in the prior art, and the present disclosure will not go into details here.

[0108] The above steps S4 to S6 can efficiently and accurately determine the pipeline parallel set, data parallel set and model parallel set corresponding to the target computation graph according to the pending data flow graph.

[0109] The above technical solution can automatically and efficiently obtain the parallel configuration strategy corresponding to the target calculation graph according to the segmented data flow graph, and can automatically and efficiently generate a model training strategy that is a hybrid of pipeline parallelism, data parallelism and model parallelism, which can effectively improve the model training efficiency of the machine learning model and shorten the model training time.

[0110] Figure 7 is a block diagram of a computing graph processing device shown in an exemplary embodiment of the present disclosure; Figure 7 As shown, the device may include:

[0111] An acquisition module 701 is configured to acquire a target computation graph corresponding to a machine learning model to be trained, the target computation graph comprising an operator node and a directed edge connecting two operator nodes;

[0112] A graph fusion module 702 is configured to perform node fusion processing on the target computation graph to obtain a fused computation graph including multiple subgraphs;

[0113] The segmentation module 703 is configured to effectively segment the operator nodes in the fusion computing graph according to multiple preset segmentation methods of each operator node, so as to obtain multiple segmentation data flow graphs corresponding to the fusion computing graph, and the segmentation data flow graph is a target computing graph with an effective segmentation method;

[0114] The search module 704 is configured to search for the parallel configuration strategy corresponding to the target computational graph according to the partitioned data flow graph. The parallel configuration strategy includes a pipeline parallel set, a data parallel set and a model parallel set. The parallel configuration strategy is used for parallel training of the machine learning model.

[0115] The above technical solution can automatically and efficiently generate a model training strategy that is a hybrid of pipeline parallelism, data parallelism, and model parallelism, which can effectively improve the model training efficiency of the machine learning model and shorten the model training time.

[0116] Optionally, the segmentation module 703 is configured to:

[0117] Representing the input tensor and the output tensor of each operator node in the fusion computation graph as tensor nodes;

[0118] A tensor-divisible dimension attribute is added to the tensor node, and different connections between the tensor nodes are used to represent multiple preset division methods of the operator node to obtain a division data flow graph.

[0119] Optionally, the segmentation module 703 is configured to:

[0120] Use the same annotation method to mark the connections between all tensor nodes associated with the same operator node that is split using the same splitting method;

[0121] Use different annotation methods to distinguish the connections of each operator node corresponding to different segmentation methods;

[0122] Determine multiple valid segmentation methods corresponding to the fusion calculation graph according to the lines marked with segmentation methods;

[0123] Each effective segmentation method corresponding to the fusion calculation graph is used as a segmentation data flow graph to obtain multiple segmentation data flow graphs.

[0124] Optionally, the search module 704 is configured to:

[0125] Performing reduction processing on each of the split data flow graphs to obtain a plurality of coarse-grained pending data flow graphs corresponding to the plurality of split data flow graphs;

[0126] According to the pending data flow graph, a pipeline parallel set, a data parallel set and a model parallel set corresponding to the target computation graph are determined by a dynamic programming algorithm.

[0127] Optionally, the search module 704 is configured to:

[0128] Get the calculation time attribute corresponding to each operator node in the target calculation graph;

[0129] Determine the consumption time attribute corresponding to each operator node in the split data flow graph according to the calculation time attribute;

[0130] The operator nodes in the split data flow graph are coarse-grainedly processed according to the consumption time attribute corresponding to each operator node to obtain the pending data flow graph.

[0131] Optionally, the search module 704 is configured to:

[0132] According to the segmentation mode of each operator node in the target computation graph and the computation time attribute, a target computation time attribute of the node operator in the segmented data flow graph is determined through a preset mapping relationship;

[0133] Obtain the communication time attribute corresponding to each operator node in the split data flow graph;

[0134] The consumed time attribute is determined according to the target calculation time attribute and the communication time attribute.

[0135] Optionally, the search module 704 is configured to:

[0136] Constructing a preset function model according to the preset number of splits and the pending data flow graph, wherein the preset function model is used to characterize the correspondence between different segmentation methods and splitting methods and the total time consumption for training the machine learning model;

[0137] Obtaining the optimal solution of the preset function model to obtain the optimal splitting method and the optimal segmentation method under the optimal splitting method;

[0138] The target computation graph is split into multiple pipeline parallel sets according to the optimal splitting method, and the data parallel set and the model parallel set in each pipeline parallel set are determined according to the optimal splitting method.

[0139] Optionally, the search module 704 is configured to:

[0140] Get the memory consumption corresponding to each set of solutions;

[0141] The target solution when the memory consumption is less than or equal to the preset memory consumption threshold and the preset function model takes the minimum function value is taken as the optimal solution.

[0142] Optionally, the graph fusion module 702 is configured to:

[0143] By presetting a dynamic graph fusion algorithm, with the minimum amount of communication data between the multiple subgraphs and the balanced amount of calculation corresponding to different subgraphs as the optimization goal, and with each subgraph as a closure graph as a constraint, the multiple target subgraphs corresponding to the target calculation graph are determined to obtain the fused calculation graph including the multiple target subgraphs.

[0144] The above technical solution can automatically and efficiently obtain the parallel configuration strategy corresponding to the target calculation graph according to the segmented data flow graph, and can automatically and efficiently generate a model training strategy that is a hybrid of pipeline parallelism, data parallelism and model parallelism, which can effectively improve the model training efficiency of the machine learning model and shorten the model training time.

[0145] Reference below Figure 8 , which shows a schematic diagram of the structure of an electronic device (such as a terminal device or a server) 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0146] like Figure 8 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0147] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0148] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0149] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0150] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0151] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0152] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains a target computational graph corresponding to the machine learning model to be trained, and the target computational graph includes operator nodes and directed edges connecting two operator nodes; performs node fusion processing on the target computational graph to obtain a fused computational graph including multiple subgraphs; according to multiple preset segmentation methods of each operator node, effectively segments the operator nodes in the fused computational graph to obtain multiple segmentation data flow graphs corresponding to the fused computational graph, and the segmentation data flow graph is a target computational graph with an effective segmentation method; searches for a parallel configuration strategy corresponding to the target computational graph according to the segmentation data flow graph, and the parallel configuration strategy includes a pipeline parallel set, a data parallel set and a model parallel set, and the parallel configuration strategy is used to parallel train the machine learning model.

[0153] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0154] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0155] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the acquisition module may also be described as "a module for obtaining a target computational graph corresponding to a machine learning model to be trained, wherein the target computational graph includes operator nodes and directed edges connecting two operator nodes".

[0156] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0157] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0158] According to one or more embodiments of the present disclosure, Example 1 provides a method for processing a computation graph, the method comprising:

[0159] Obtain a target computation graph corresponding to the machine learning model to be trained, wherein the target computation graph includes an operator node and a directed edge connecting two operator nodes;

[0160] Performing node fusion processing on the target computation graph to obtain a fused computation graph including multiple subgraphs;

[0161] According to a plurality of preset segmentation methods of each operator node, the operator nodes in the fusion calculation graph are effectively segmented to obtain a plurality of segmentation data flow graphs corresponding to the fusion calculation graph, wherein the segmentation data flow graph is a target calculation graph with an effective segmentation method;

[0162] The parallel configuration strategy corresponding to the target computational graph is searched according to the partitioned data flow graph, wherein the parallel configuration strategy includes a pipeline parallel set, a data parallel set and a model parallel set, and the parallel configuration strategy is used for parallel training of the machine learning model.

[0163] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein the operator nodes in the fusion computing graph are effectively segmented according to multiple preset segmentation methods of each operator node to obtain multiple segmentation data flow graphs corresponding to the fusion computing graph, including:

[0164] Representing the input tensor and the output tensor of each operator node in the fusion computation graph as tensor nodes;

[0165] A tensor-divisible dimension attribute is added to the tensor node, and different connections between the tensor nodes are used to represent multiple preset division methods of the operator node to obtain a division data flow graph.

[0166] According to one or more embodiments of the present disclosure, Example 3 provides the method described in Example 2, wherein adding a tensor-divisible dimension attribute to the tensor node and using different connections between the tensor nodes to represent multiple preset division methods of the operator node to obtain a division data flow graph includes:

[0167] Use the same annotation method to mark the connections between all tensor nodes associated with the same operator node that is split using the same splitting method;

[0168] Use different annotation methods to distinguish the connections of each operator node corresponding to different segmentation methods;

[0169] Determine multiple valid segmentation methods corresponding to the fusion calculation graph according to the lines marked with segmentation methods;

[0170] Each effective segmentation method corresponding to the fusion calculation graph is used as a segmentation data flow graph to obtain multiple segmentation data flow graphs.

[0171] According to one or more embodiments of the present disclosure, Example 4 provides the method described in Example 1, wherein searching for a parallel configuration strategy corresponding to the target computation graph according to the segmented data flow graph, wherein the parallel configuration strategy includes a pipeline parallel set, a data parallel set, and a model parallel set, including:

[0172] Performing reduction processing on each of the split data flow graphs to obtain a plurality of coarse-grained pending data flow graphs corresponding to the plurality of split data flow graphs;

[0173] According to the pending data flow graph, a pipeline parallel set, a data parallel set and a model parallel set corresponding to the target computation graph are determined by a dynamic programming algorithm.

[0174] According to one or more embodiments of the present disclosure, Example 5 provides the method described in Example 4, wherein the reduction processing is performed on each of the split data flow graphs to obtain multiple coarse-grained pending data flow graphs corresponding to the multiple split data flow graphs, including:

[0175] Obtain the computation time attribute corresponding to each operator node in the target computation graph;

[0176] Determine the consumption time attribute corresponding to each operator node in the split data flow graph according to the calculation time attribute;

[0177] The operator nodes in the split data flow graph are coarse-grainedly processed according to the consumption time attribute corresponding to each operator node to obtain the pending data flow graph.

[0178] According to one or more embodiments of the present disclosure, Example 6 provides the method described in Example 5, wherein determining the consumption time attribute corresponding to each operator node in the split data flow graph according to the calculation time attribute includes:

[0179] According to the segmentation mode and the computation time attribute of each operator node in the target computation graph, the target computation time attribute of the node operator in the segmented data flow graph is determined by a preset mapping relationship;

[0180] Obtaining the communication time attribute corresponding to each operator node in the split data flow graph;

[0181] The consumed time attribute is determined based on the target calculation time attribute and the communication time attribute.

[0182] According to one or more embodiments of the present disclosure, Example 7 provides the method described in Example 4, wherein determining the pipeline parallel set, the data parallel set, and the model parallel set by a dynamic programming algorithm according to the pending data flow graph includes:

[0183] Constructing a preset function model according to the preset number of splits and the pending data flow graph, wherein the preset function model is used to characterize the correspondence between different segmentation methods and splitting methods and the total time consumption for training the machine learning model;

[0184] Obtaining the optimal solution of the preset function model to obtain an optimal splitting method and an optimal segmentation method under the optimal splitting method;

[0185] The target computation graph is split into a plurality of pipeline parallel sets according to the optimal splitting method, and a data parallel set and a model parallel set in each of the pipeline parallel sets are determined according to the optimal splitting method.

[0186] According to one or more embodiments of the present disclosure, Example 8 provides the method of Example 7, wherein obtaining the optimal solution of the preset function model to obtain the optimal splitting method and the optimal segmentation method under the optimal splitting method includes:

[0187] Get the memory consumption corresponding to each set of solutions;

[0188] The target solution when the memory consumption is less than or equal to the preset memory consumption threshold and the preset function model takes the minimum function value is taken as the optimal solution.

[0189] According to one or more embodiments of the present disclosure, Example 9 provides the method of Example 1, wherein performing node fusion processing on the target computing graph to obtain a fused computing graph including multiple subgraphs includes:

[0190] By presetting a dynamic graph fusion algorithm, with the minimum amount of communication data between the multiple subgraphs and the balanced amount of calculation corresponding to different subgraphs as the optimization goal, and with each subgraph as a closure graph as a constraint, the multiple target subgraphs corresponding to the target calculation graph are determined to obtain the fused calculation graph including the multiple target subgraphs.

[0191] According to one or more embodiments of the present disclosure, Example 10 provides a computing graph processing device, the device comprising:

[0192] An acquisition module is configured to acquire a target computation graph corresponding to the machine learning model to be trained, wherein the target computation graph includes an operator node and a directed edge connecting two operator nodes;

[0193] A graph fusion module is configured to perform node fusion processing on the target computation graph to obtain a fused computation graph including multiple subgraphs;

[0194] A splitting module is configured to effectively split the operator nodes in the fusion computing graph according to a plurality of preset splitting methods for each operator node, so as to obtain a plurality of split data flow graphs corresponding to the fusion computing graph, wherein the split data flow graph is a target computing graph with an effective splitting method;

[0195] The search module is configured to search for the parallel configuration strategy corresponding to the target computational graph according to the partitioned data flow graph, wherein the parallel configuration strategy includes a pipeline parallel set, a data parallel set and a model parallel set, and the parallel configuration strategy is used for parallel training of the machine learning model.

[0196] According to one or more embodiments of the present disclosure, Example 11 provides a computer-readable medium having a computer program stored thereon, characterized in that when the program is executed by a processing device, the steps of the method described in any one of Examples 1-9 above are implemented.

[0197] According to one or more embodiments of the present disclosure, Example 12 provides an electronic device, including:

[0198] a storage device having a computer program stored thereon;

[0199] A processing device is used to execute the computer program in the storage device to implement the steps of the method described in any one of Examples 1-9 above.

[0200] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0201] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0202] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A method for processing a computational graph, characterized in that: Applied to electronic equipment, the method comprises: Obtain a target computation graph corresponding to a machine learning model to be trained, wherein the target computation graph includes an operator node and a directed edge connecting two operator nodes, and the machine learning model is a neural network model for computer vision, natural language processing, or speech recognition; Performing node fusion processing on the target computation graph to obtain a fused computation graph including multiple subgraphs; According to a plurality of preset segmentation methods of each operator node, the operator nodes in the fusion calculation graph are effectively segmented to obtain a plurality of segmentation data flow graphs corresponding to the fusion calculation graph, wherein the segmentation data flow graph is a target calculation graph with an effective segmentation method; Searching for a parallel configuration strategy corresponding to the target computational graph according to the segmented data flow graph, wherein the parallel configuration strategy includes a pipeline parallel set, a data parallel set, and a model parallel set, and the parallel configuration strategy is used to train the machine learning model in parallel; The method of effectively segmenting the operator nodes in the fusion calculation graph according to the multiple preset segmentation modes of each operator node to obtain a segmentation data flow graph corresponding to the fusion calculation graph includes: Representing the input tensor and the output tensor of each operator node in the fusion computation graph as tensor nodes; A tensor-divisible dimension attribute is added to the tensor node, and different connections between tensor nodes are used to represent multiple preset division methods of the operator node to obtain the division data flow graph.

2. The method according to claim 1, characterized in that The adding of a tensor-divisible dimension attribute in the tensor node and using different connections between the tensor nodes to represent a plurality of preset division modes of the operator node to obtain the division data flow graph includes: Use the same annotation method to mark the connections between all tensor nodes associated with the same operator node that is split using the same splitting method; Use different annotation methods to distinguish the connections of each operator node corresponding to different segmentation methods; Determine multiple valid segmentation methods corresponding to the fusion calculation graph according to the lines marked with segmentation methods; Each effective segmentation method corresponding to the fusion calculation graph is used as a segmentation data flow graph to obtain multiple segmentation data flow graphs.

3. The method according to claim 1, characterized in that The parallel configuration strategy corresponding to the target computation graph is searched according to the segmented data flow graph, wherein the parallel configuration strategy includes a pipeline parallel set, a data parallel set and a model parallel set, including: Performing reduction processing on each of the split data flow graphs to obtain a plurality of coarse-grained pending data flow graphs corresponding to the plurality of split data flow graphs; According to the pending data flow graph, a pipeline parallel set, a data parallel set and a model parallel set corresponding to the target computation graph are determined by a dynamic programming algorithm.

4. The method according to claim 3, characterized in that: The reducing process is performed on each of the split data flow graphs to obtain a plurality of coarse-grained pending data flow graphs corresponding to the plurality of split data flow graphs, including: Obtain the computation time attribute corresponding to each operator node in the target computation graph; Determine the consumption time attribute corresponding to each operator node in the split data flow graph according to the calculation time attribute; The operator nodes in the split data flow graph are coarse-grainedly processed according to the consumption time attribute corresponding to each operator node to obtain the pending data flow graph.

5. The method according to claim 4, characterized in that The determining, according to the calculation time attribute, the consumption time attribute corresponding to each operator node in the segmented data flow graph includes: According to the segmentation mode and the computation time attribute of each operator node in the target computation graph, determining the target computation time attribute of the operator node in the segmented data flow graph through a preset mapping relationship; Obtaining the communication time attribute corresponding to each operator node in the split data flow graph; The consumed time attribute is determined based on the target calculation time attribute and the communication time attribute.

6. The method according to claim 3, characterized in that The determining of the pipeline parallel set, the data parallel set and the model parallel set by a dynamic programming algorithm according to the pending data flow graph includes: Constructing a preset function model according to the preset number of splits and the pending data flow graph, wherein the preset function model is used to characterize the correspondence between different segmentation methods and splitting methods and the total time consumption for training the machine learning model; Obtaining the optimal solution of the preset function model to obtain an optimal splitting method and an optimal segmentation method under the optimal splitting method; The target computation graph is split into a plurality of pipeline parallel sets according to the optimal splitting method, and a data parallel set and a model parallel set in each of the pipeline parallel sets are determined according to the optimal splitting method.

7. The method according to claim 6, characterized in that The step of obtaining the optimal solution of the preset function model to obtain an optimal splitting method and an optimal segmentation method under the optimal splitting method includes: Get the memory consumption corresponding to each set of solutions; The target solution when the memory consumption is less than or equal to a preset memory consumption threshold and the preset function model takes a minimum function value is taken as the optimal solution.

8. The method according to claim 1, characterized in that The performing node fusion processing on the target computation graph to obtain a fused computation graph including a plurality of subgraphs includes: By presetting a dynamic graph fusion algorithm, with the minimum amount of communication data between the multiple subgraphs and the balanced amount of calculation corresponding to different subgraphs as the optimization goal, and with each subgraph as a closure graph as a constraint, the multiple target subgraphs corresponding to the target calculation graph are determined to obtain the fused calculation graph including the multiple target subgraphs.

9. A computing graph processing device, characterized in that: Applied to electronic equipment, the device comprises: An acquisition module is configured to acquire a target computation graph corresponding to a machine learning model to be trained, wherein the target computation graph includes an operator node and a directed edge connecting two operator nodes, and the machine learning model is a neural network model for computer vision, natural language processing, or speech recognition; A graph fusion module is configured to perform node fusion processing on the target computation graph to obtain a fused computation graph including multiple subgraphs; A splitting module is configured to effectively split the operator nodes in the fusion computing graph according to a plurality of preset splitting methods for each operator node, so as to obtain a plurality of split data flow graphs corresponding to the fusion computing graph, wherein the split data flow graph is a target computing graph with an effective splitting method; A search module is configured to search for a parallel configuration strategy corresponding to the target computation graph according to the segmented data flow graph, wherein the parallel configuration strategy includes a pipeline parallel set, a data parallel set, and a model parallel set, and the parallel configuration strategy is used to train the machine learning model in parallel; The splitting module is also configured to: represent the input tensor and output tensor of each operator node in the fusion calculation graph as tensor nodes; add tensor splittable dimension attributes in the tensor nodes, and use different connections between tensor nodes to represent multiple preset splitting methods of the operator nodes to obtain a split data flow graph.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 8 are implemented.

11. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 8.

Images