Machine learning model hybrid parallel strategy automatic search method and system

Abstract

The invention provides a machine learning model hybrid parallel strategy automatic search method and system, and the system comprises an inter-operator parallel strategy search module which firstly queries an inter-operator performance overhead model, and then carries out strategy search according to the performance overhead between the operators at the current stage. Wherein the inter-operator performance overhead model calculates the overhead of the corresponding pipeline stage. And the in-operator parallel strategy search module is used for carrying out data parallel and tensor parallel strategy search by adopting a shrinkage point edge elimination dynamic programming algorithm. A performance overhead model in an operator is inquired firstly, and then strategy search is carried out. And the performance overhead model in the operator performs calculation overhead estimation by using an execution-based method, and performs communication overhead estimation by using a calculation-based method at the same time. Compared with an existing working search algorithm, the search algorithm has lower theoretical time complexity, search results can be obtained more quickly, and the quality of strategies obtained through search can be guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of parallel technology for machine learning training, and in particular to a method and system for automatic search of hybrid parallel strategies for machine learning models. Background Art

[0002] Machine learning usually has important applications in many fields. In the process of using machine learning, it is usually necessary to train the machine learning model. The scale of current machine learning models is getting larger and larger. For example, the Llama3 model proposed by Meta contains about 7B to 70B of parameters, which usually requires hundreds of GB of GPU memory. In order to achieve a fast machine learning training process, the machine learning model training process needs to be parallelized. For different models, there are usually different methods suitable for their parallel training. In order to find the optimal parallel strategy for different models, it is necessary to automatically search for machine learning training parallel strategies.

[0003] Existing machine learning models usually include data parallelism, tensor parallelism, and pipeline parallelism. Data parallelism refers to the process of dividing training data samples and transmitting them to different nodes for training. Tensor parallelism refers to the multi-dimensional division of tensors involved in operators, which are assigned to different computing devices, and the calculation results of the operators are obtained by parallel calculations of multiple computing devices. Pipeline parallelism refers to the division of different operators in the neural network of the machine learning model into different stages of the pipeline, turning the machine learning training process into a pipeline form, and improving the throughput of the training process.

[0004] There are two main types of existing automatic search for parallel strategies for machine learning training: automatic search for parallel strategies without considering pipeline parallelism and automatic search for parallel strategies with considering pipeline parallelism. The automatic search for parallel strategies without considering pipeline parallelism first counts the computational overhead of different parallel strategies and the communication overhead within operators for each point in the computation graph, i.e., operator, and counts the communication overhead between operators for each edge in the computation graph, i.e., the flow of tensors. Then, a method based on global dynamic programming is called to select the parallel strategy for each operator in the computation graph. Specifically, try to delete every point or edge in the computational graph, store the computation and communication overhead of the point or edge in the adjacent points; keep shrinking the computational graph, traverse each node on the reduced computational graph, select the parallel strategy of the node, so that the overall overhead of the final computational graph is minimized, and after selecting the parallel strategy of each node, add the originally deleted points and edges back in reverse order according to the order of deletion, and select the appropriate parallel strategy for each added node in the process of adding, and complete the search process of the entire algorithm after restoring to the original graph. This automatic search method of strategy without considering pipeline parallelism has a faster search speed and can find the global optimal parallel strategy, but because it does not consider pipeline parallelism and lacks the dimension of pipeline parallelism, it is difficult to achieve better optimization effects, and because it does not consider the size of GPU memory required by different parallel strategies, the parallel strategy found based on the overhead is likely to be unable to be executed on the corresponding device.

[0005] Existing methods for automatic parallel strategy search that take pipeline parallelism into consideration divide machine learning models into different pipeline stages, and perform data parallel and tensor parallel strategy search in each stage. Related works of this method, such as Alpa, model the pipeline stage overhead as the time required for a batch to pass through the entire pipeline, that is, in the automatic search process of pipeline parallel strategies, the time required for a batch to pass through the entire pipeline should be reduced as much as possible. This method abstracts the automatic search process of data parallel and tensor parallel strategies into an integer linear programming problem, and calls the integer linear programming solver to solve it. This automatic strategy search method that takes pipeline parallelism into consideration combines three different parallel strategies, broadens the optimization space of parallel training, and can find more optimization opportunities. However, this method consumes more search time and cannot find the globally optimal parallel training strategy.

[0006] Although the current parallel training strategy search method can find a parallel training strategy that mixes the three parallel strategies, it takes a long time and cannot find this strategy efficiently.

[0007] To sum up, in response to the above-mentioned problems of the prior art, studying a method and system for automatic search of hybrid parallel strategies for machine learning models, which can quickly find parallel strategies with better quality, has become a key task that needs to be solved urgently. Summary of the invention

[0008] In view of the defects in the prior art, the object of the present invention is to provide a method and system for automatic search of hybrid parallel strategies for machine learning models.

[0009] A method for automatically searching for a hybrid parallel strategy of a machine learning model provided by the present invention comprises the following steps:

[0010] Inter-operator search step: divide the pipeline stages according to the input computation graph and device topology to obtain the divided pipeline stages;

[0011] An inter-operator overhead generation step: using the time of the most time-consuming stage in the divided pipeline stages as a performance indicator to provide an inter-operator performance overhead model for the inter-operator search step;

[0012] Intra-operator search step: perform parallel strategy search combining data parallelism and tensor parallelism on the input computational graph subgraph; use the point reduction and edge elimination algorithm based on the global dynamic programming method to perform strategy search, and obtain the optimal parallel strategy of each operator in the computational graph subgraph and the corresponding node information of the selected parallel strategy, where the computational cost and communication cost corresponding to the strategy of each node are given by the intra-operator cost generation step;

[0013] The step of generating the intra-operator overhead is as follows: for the node information of the selected parallel strategy, the computational overhead, the communication overhead within the operator and the communication overhead between operators are calculated.

[0014] Preferably, in the inter-operator search step, the operators of the computation graph are topologically sorted, and the operators are divided into different pipeline stages using a dynamic programming algorithm.

[0015] Preferably, the dynamic programming algorithm relies on an inter-operator overhead generation step, i.e., calling an inter-operator performance overhead model, dividing different operators into different pipeline stages, and taking a method of reducing the most time-consuming pipeline stage in the pipeline as the overhead model of the dynamic programming algorithm for search, and finally obtaining the divided pipeline stages.

[0016] Preferably, in the inter-operator overhead generation step, the performance overhead of the divided pipeline stage is estimated, and the optimal parallel strategy overhead combining data parallelism and tensor parallelism in a pipeline stage is obtained by calling the intra-operator search step.

[0017] Preferably, in the inter-operator overhead generation step, the most time-consuming stage time in the pipeline is taken as the main consideration factor, and the time consumption of this stage should be reduced as much as possible when applying the search algorithm.

[0018] Preferably, in the intra-operator search step, during the parallel strategy search, data parallel and operator model parallel strategy searches are performed within the operator, and a dynamic programming approach is used to delete points and edges from the computational graph composed of the operators. At the same time, the cost information of the points and edges is saved in the adjacent points and adjacent edges, thereby performing strategy search on a smaller graph.

[0019] Preferably, in the intra-operator search step, when performing point and edge deletion operations, point elimination and edge elimination strategies are applied to any computational graph subgraph, wherein point elimination includes cases of three points, four points, and more than four points, and edge elimination includes cases where there are multiple edges between two points.

[0020] Preferably, in the intra-operator overhead generation step, the computational overhead is obtained by actual testing, and the communication overhead is obtained by dividing the tensor size by the bandwidth.

[0021] The present invention also provides a system for automatic search of hybrid parallel strategies for machine learning models, comprising:

[0022] Inter-operator parallel strategy search module: divides the pipeline stages according to the calculation graph and cluster device conditions;

[0023] Inter-operator performance overhead model: The inter-operator strategy overhead is calculated based on the divided pipeline conditions and the equipment used in the pipeline stage after the division according to the intra-operator parallel strategy search module;

[0024] Intra-operator parallel strategy search module: selects the optimal strategy for each operator based on the computational graph subgraph within the same stage;

[0025] Intra-operator performance overhead model: The corresponding overhead is calculated based on the strategy selected by each operator in the computational graph subgraph within the same stage.

[0026] Preferably, the intra-operator parallel strategy search module includes:

[0027] Point reduction and edge elimination rule register unit: registers the rules available in the point reduction and edge elimination process;

[0028] Point reduction and edge elimination rule selector unit: selects the corresponding point reduction and edge elimination rule according to the current computational graph topology structure;

[0029] Point reduction and edge elimination algorithm executor unit: executes the corresponding dynamic programming algorithm according to the selected point reduction and edge elimination rules.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. This invention proposes and implements a pipeline stage search algorithm based on a new pipeline planning overhead model. This algorithm optimizes the pipeline planning overhead model, accurately evaluates and plans pipeline stages, and significantly reduces the delay of pipeline stage search. Compared with traditional algorithms, it can more efficiently divide pipeline stages according to calculation graphs and equipment conditions.

[0032] 2. Based on the existing point shrinking and edge elimination algorithm, the present invention proposes a new direction-independent point shrinking and edge elimination rule, and proposes a point shrinking and edge elimination rule for large graphs through a heuristic method, which expands the point shrinking and edge elimination capabilities and improves the search efficiency. For large-scale machine learning model calculation graphs, these new rules can more effectively simplify the graph structure and reduce the complexity of the search space. In practical applications, when processing calculation graphs containing a large number of operators and complex connection relationships, the new rules can quickly find a better parallel strategy search path, making the search process more efficient and accurate, and further improving the performance and effect of the automatic search of the entire machine learning model hybrid parallel strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0034] Figure 1 A system and process diagram for automatic search of a hybrid parallel strategy for a machine learning model in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0036] The present invention provides a method and system for automatic search of hybrid parallel strategies for machine learning models, wherein hybrid refers to the use of three parallel strategies, namely data parallelism, tensor parallelism and pipeline parallelism, to perform distributed training on machine learning models. The method divides the computational graph corresponding to the machine learning model into pipeline stages, and then selects strategies for data parallelism and operator model parallelism within each pipeline stage. The inter-operator strategy overhead of the present invention depends on the intra-operator strategy search result, and the subgraph overhead of the intra-operator strategy search result is used as a pipeline stage overhead. The intra-operator strategy overhead depends on the external overhead estimate of the corresponding operator, and the intra-operator strategy search depends on the point reduction and edge elimination algorithm for implementation.

[0037] Embodiment 1:

[0038] Figure 1 A system and process diagram for automatic search of a hybrid parallel strategy for a machine learning model in an embodiment of the present invention.

[0039] like Figure 1 As shown, this embodiment provides a method for automatic search of a hybrid parallel strategy for a machine learning model, comprising the following steps:

[0040] Inter-operator search step: Divide the pipeline stages according to the input computation graph and device topology to obtain the divided pipeline stages.

[0041] The computational graph is a directed acyclic graph composed of different machine learning operators. Each node corresponds to an operator, and each edge corresponds to the inflow and outflow of tensors. The following conditions must be met: each node must exist in a pipeline stage, pipeline stages cannot overlap, different pipeline stages can be calculated in parallel, and each pipeline stage contains at least one device.

[0042] Specifically, the operators of the computation graph are topologically sorted, and the dynamic programming algorithm is used to divide the operators into different pipeline stages.

[0043] More specifically, the dynamic programming algorithm relies on an inter-operator overhead generation step, that is, calling an inter-operator performance overhead model, dividing different operators into different pipeline stages, and taking the approach of reducing the most time-consuming pipeline stage in the pipeline as the overhead model of the dynamic programming algorithm for search, and finally obtaining the divided pipeline stages.

[0044] Inter-operator overhead generation step: using the time of the most time-consuming stage in the divided pipeline stages as a performance indicator to provide an inter-operator performance overhead model for the inter-operator search step.

[0045] Specifically, the performance overhead of the divided pipeline stage is estimated, and the optimal parallel strategy overhead combining data parallelism and tensor parallelism in a pipeline stage is obtained by calling the intra-operator search step.

[0046] The most time-consuming stage in the pipeline should be taken as the main consideration, and the time consumption of this stage should be minimized when applying the search algorithm.

[0047] Intra-operator search step: perform parallel strategy search combining data parallelism and tensor parallelism on the input computational graph subgraph; use the point reduction and edge elimination algorithm based on the global dynamic programming method to perform strategy search to obtain the optimal parallel strategy for each operator in the computational graph subgraph and the corresponding node information of the selected parallel strategy, where the computational overhead and communication overhead corresponding to the strategy of each node are given by the intra-operator overhead generation step.

[0048] Specifically, during parallel strategy search, data parallel and operator model parallel strategy searches are performed within the operator. Dynamic programming is used to delete points and edges in the computational graph composed of operators. At the same time, the cost information of points and edges is saved in adjacent points and adjacent edges, thereby performing strategy search on a smaller graph.

[0049] More specifically, when performing point and edge deletion operations, point and edge elimination strategies are applied to any computational graph subgraph, where point elimination includes three points, four points, and more than four points, and edge elimination includes the case where there are multiple edges between two points.

[0050] The step of generating the intra-operator overhead is as follows: for the node information of the selected parallel strategy, the computational overhead, the communication overhead within the operator and the communication overhead between operators are calculated.

[0051] In this embodiment, the computing overhead is the average delay obtained by executing the operator on a specific device, which is obtained from actual tests, and the communication overhead is obtained by dividing the tensor size by the bandwidth.

[0052] Intra-operator performance overhead model: A model that counts the overhead of parallel strategies within an operator. It mainly uses the overhead model of the existing system to estimate different strategies for different operators.

[0053] Embodiment 2:

[0054] Next, the present invention will be described in more detail.

[0055] This embodiment provides a method for automatically searching for a hybrid parallel strategy for a machine learning model, the method comprising:

[0056] 1) Apply the pipeline dynamic programming partitioning algorithm to the input computational graph to divide the pipeline stages.

[0057] 2) To reduce the most time-consuming stage in the pipeline, overhead generation and strategy search are performed on the pipeline stages.

[0058] 3) Search for parallel strategies for computing graph subgraphs within the pipeline stage through the point reduction and edge elimination algorithm.

[0059] 4) The computational cost of each operator is obtained by actual testing, and the communication cost of each operator is obtained by calculation, which are applied to the parallel strategy search process of the computational graph subgraph.

[0060] Among them, the application of the point reduction and edge elimination algorithm for parallel strategy search includes a predefined series of point reduction and edge elimination rules to delete points and edges from the input calculation graph; these rules not only include rules that can ensure that the optimal strategy is finally searched, but also include some heuristic rules to speed up the search progress.

[0061] The complete implementation process of the present invention is described by the following contents:

[0062] Inter-operator search step: divide the pipeline stages according to the input computation graph and device topology. Assume that for graph G, the operators in the computation graph are ordered according to the topological order to obtain an operator order: O1, O2, ..., O n , we need to divide these operators into K stages, namely s1,s2,...,s K , where each stage is assigned a corresponding number of devices, so there are d1,d2,...,d K Devices, including D is the total number of devices. The current problem is to find a number of stages k such that the slowest stage in the pipeline takes the least time among all possible stages. It can be converted into the formula

[0063]

[0064] This performance overhead model does not take into account the communication between different pipeline stages. This is because during the pipeline calculation process, different pipeline stages can send already calculated data to the next pipeline stage, and the time consumed by the pipeline stage is greater than the time required to transmit data between different pipeline stages. Therefore, the time required for communication between different pipeline stages is hidden and there is no need to consider this communication time in the performance overhead model.

[0065] Specifically, the present invention performs parallel strategy search between operators according to the following dynamic programming algorithm:

[0066] A dynamic programming algorithm is designed based on the inter-operator performance overhead model. Since the number of stages cannot exceed the number of operators, that is, in extreme cases each operator exists in one stage, different numbers of stages can be traversed, and the time consumption of searching for the stage with the longest search time under the determined number of stages is considered.

[0067] Let F(i,j,z) be the time of the longest phase of the first i operators, first j phases, and first z devices, then the problem is transformed into

[0068] t * =min k {F(n,k,D)}

[0069] This process is a dynamic programming process. The specific recursive equation is:

[0070]

[0071] The equation t intra Represents the parallel time within the operator.intra ((o i-x+1 ,...,o i ),d) is to convert the operator o i-x+1 ,...,o i , in parallel on d devices, in parallel in the same stage. Therefore, t intra It is necessary to call the intra-operator parallel module for calculation, and then fill in the three-dimensional table of dynamic programming in the inter-operator parallel module according to the recursive equation. After running the dynamic programming algorithm, according to the obtained three-dimensional table, the inter-operator parallel strategy search algorithm traverses the □ to find a suitable number of stages to minimize the inter-operator performance overhead model.

[0072] The boundary conditions of this dynamic programming are:

[0073] 1. When i=0, the semantics of F(0,j,z) is the time of the longest period among the first 0 operators, the first j stages and the first z devices. Since there are only 0 operators left at this time, the time of the longest period should be 0, so F(0,j,z)=0.

[0074] 2. When j = 1, the semantics of F(i,1,z) is the longest phase time among the first i operators, the first 1 phase, and the first z devices. Since there is only one phase left, all operators and all devices should be parallelized in this phase, so F(i,1,z) = t intra ((o1,...,o i ),1,z).

[0075] 3. When z = 1, the semantics of F(i, j, 1) is the longest time period among the first i operators, the first j stages and the first 1 device. Since there is only one device left, all operators should be run in parallel on 1 stage and 1 device, so F(i, j, 1) = t intra ((o1,...,o i ),1,1).

[0076] The dynamic programming algorithm handles the following equipment situation: the equipment is divided into a grid in the system. For example, (2,3) is a equipment grid, and its semantics is 3 equipment per row, with a total of 2 rows. Assuming that there are D equipment in total, where D = N × M, this system divides the equipment into (1,1), (1,2), (1,4), ..., (1,2 m ) 2 of them m =M, or (2,M), (3,M), ..., (N,M). The device grid is divided in the same way as Alpa, because this division method can ensure that every device in the grid is covered without wasting computing resources.

[0077] Inter-operator overhead generation steps: Apply the overhead model corresponding to the above pipeline stage: t * =min k {F(n,k,D)}. The goal is to minimize the time of the most time-consuming stage in the pipeline. This step will call the operator intra-search step multiple times to obtain an optimal intra-stage time as the cost of this stage.

[0078] Intra-operator search step: perform a parallel strategy search combining data parallelism and tensor parallelism on the input computational graph subgraph. This step uses a point reduction and edge elimination method to perform intra-operator parallel strategy search, where the point reduction and edge elimination rules include:

[0079] Edge elimination rule: If there are multiple edges between any two nodes, these edges are merged into one edge, and the communication overhead of the edges of the parallel strategies corresponding to the two operators is added together to obtain a newly created edge. In this way, while eliminating one edge, the information of the original two edges is retained, simplifying the entire computational graph.

[0080] Point elimination rules: Point elimination situations are divided into three points, four points and more than four points.

[0081] The case of three points is the case where a node has two adjacent edges: in this case, the point and the two adjacent edges are deleted, and the corresponding overhead information is merged into the two adjacent points.

[0082] The case of four points is the case where a node has three adjacent edges: in this case, the point and the three adjacent edges are deleted, and the corresponding overhead information is merged into the three adjacent points. A new edge is generated between any two nodes to store the communication overhead.

[0083] The case where there are more than four points is when a node has more than four adjacent edges: in this case, the point is directly deleted, and a parallel strategy that uses the least memory is selected for the point.

[0084] After shrinking the graph and eliminating the edges to obtain a small graph, enumerate the parallel strategies of the operators on the small graph, and then add the originally deleted operators and edges back in reverse order. When adding them back, select a corresponding parallel strategy for each operator.

[0085] In the above-mentioned rules for reducing points and eliminating edges, the rules corresponding to the case with more than four points cannot search for the optimal parallel strategy. Therefore, in the process of applying rules in the algorithm, the use of such rules will be minimized. The specific method is that if any other rules cannot delete the points or edges, this type of rule will be applied once, and then other rules will continue to be applied.

[0086] Intra-operator overhead generation step: Calculate the computational overhead, communication overhead, and inter-operator communication overhead for the node that has selected the parallel strategy. The intra-operator communication overhead depends on the parallel strategy selected by the operator. Different parallel strategies may require inter-device communication when performing calculations in the same operator. The inter-operator communication overhead depends on the parallel strategies selected by different operators, which may require inter-device communication for the transmission of tensors between operators.

[0087] Embodiment 3:

[0088] The present invention provides a machine learning model hybrid parallel strategy automatic search system, which can be implemented by executing the process steps of the machine learning model hybrid parallel strategy automatic search method, that is, technical personnel in this field can understand the machine learning model hybrid parallel strategy automatic search method as a preferred implementation of the machine learning model hybrid parallel strategy automatic search system.

[0089] The system includes: an inter-operator parallel strategy search module: this module uses a dynamic programming algorithm to divide the input calculation graph into pipeline stages. First, the inter-operator performance overhead model is queried, and then a strategy search is performed based on the performance overhead between operators in the current stage. Among them, the inter-operator performance overhead model will calculate the corresponding pipeline stage overhead based on the number of operators in the divided pipeline stages, the types of operators, and the number of allocated devices. An intra-operator parallel strategy search module: this module uses a dynamic programming algorithm of shrinking points and eliminating edges to search for data parallelism and tensor parallelism strategies for the input calculation graph subgraph. It first queries the intra-operator performance overhead model, and then performs a strategy search based on the performance overhead within the current operator. The intra-operator performance overhead model estimates the computational overhead by using an execution-based method, and at the same time uses a calculation-based method to estimate the communication overhead.

[0090] Figure 1 A system and process diagram for automatic search of a hybrid parallel strategy for a machine learning model in an embodiment of the present invention.

[0091] like Figure 1 Specifically, this embodiment provides a system for automatic search of hybrid parallel strategies for machine learning models, including:

[0092] Inter-operator parallel strategy search module: divides the pipeline stages according to the calculation graph and cluster device conditions;

[0093] Inter-operator performance overhead model: The inter-operator strategy overhead is calculated based on the divided pipeline conditions and the equipment used in the pipeline stage after the division according to the intra-operator parallel strategy search module;

[0094] Intra-operator parallel strategy search module: selects the optimal strategy for each operator based on the computational graph subgraph within the same stage;

[0095] In this embodiment, the intra-operator parallel strategy search module includes:

[0096] Point reduction and edge elimination rule register unit: registers the rules available in the point reduction and edge elimination process;

[0097] Point reduction and edge elimination rule selector unit: selects the corresponding point reduction and edge elimination rule according to the current computational graph topology structure;

[0098] Point reduction and edge elimination algorithm executor unit: executes the corresponding dynamic programming algorithm according to the selected point reduction and edge elimination rules.

[0099] Intra-operator performance overhead model: The corresponding overhead is calculated based on the strategy selected by each operator in the computational graph subgraph within the same stage.

[0100] The present invention verifies the Bert models with different numbers of layers and can achieve up to 2.5 times performance improvement compared to Alpa.

[0101] The embodiment of the present invention provides a method and system for automatic search of hybrid parallel strategies for machine learning models, which combines data parallelism, tensor parallelism and pipeline parallelism in the input calculation graph and the number of devices. Compared with the prior art, the present invention can customize the expansion of point reduction and edge elimination rules, uses a more appropriate cost model, and greatly improves the search efficiency.

[0102] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0103] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for automatic search of hybrid parallel strategies for machine learning models, characterized in that: The steps include: Inter-operator search step: divide the pipeline stages according to the input computation graph and device topology to obtain the divided pipeline stages; An inter-operator overhead generation step: using the time of the most time-consuming stage in the divided pipeline stages as a performance indicator to provide an inter-operator performance overhead model for the inter-operator search step; Intra-operator search step: perform parallel strategy search combining data parallelism and tensor parallelism on the input computational graph subgraph; use the point reduction and edge elimination algorithm based on the global dynamic programming method to perform strategy search, and obtain the optimal parallel strategy of each operator in the computational graph subgraph and the corresponding node information of the selected parallel strategy, where the computational cost and communication cost corresponding to the strategy of each node are given by the intra-operator cost generation step; The step of generating the intra-operator overhead is as follows: for the node information of the selected parallel strategy, the computational overhead, the communication overhead within the operator and the communication overhead between operators are calculated.

2. The method for automatic search of hybrid parallel strategies for machine learning models according to claim 1, characterized in that: In the inter-operator search step, the operators of the computation graph are topologically sorted, and the operators are divided into different pipeline stages using a dynamic programming algorithm.

3. The method for automatic search of hybrid parallel strategies for machine learning models according to claim 2, characterized in that: The dynamic programming algorithm relies on the inter-operator overhead generation step, that is, calling the inter-operator performance overhead model, dividing different operators into different pipeline stages, and taking the method of reducing the most time-consuming pipeline stage in the pipeline as the overhead model of the dynamic programming algorithm for search, and finally obtaining the divided pipeline stages.

4. The method for automatic search of hybrid parallel strategies for machine learning models according to claim 1, characterized in that: In the inter-operator overhead generation step, the performance overhead of the divided pipeline stage is estimated, and the optimal parallel strategy overhead combining data parallelism and tensor parallelism in a pipeline stage is obtained by calling the intra-operator search step.

5. The method for automatic search of hybrid parallel strategies for machine learning models according to claim 4, characterized in that: In the inter-operator overhead generation step, the most time-consuming stage time in the pipeline is taken as the main consideration factor, and the time consumption of this stage should be reduced as much as possible when applying the search algorithm.

6. The method for automatic search of hybrid parallel strategies for machine learning models according to claim 1, characterized in that: In the intra-operator search step, during the parallel strategy search, data parallel and operator model parallel strategy searches are performed within the operator, and the calculation graph composed of the operators is deleted by dynamic programming. At the same time, the cost information of the points and edges is saved in the adjacent points and adjacent edges, so that the strategy search is performed on a smaller graph.

7. The method for automatic search of hybrid parallel strategies for machine learning models according to claim 6, characterized in that: In the intra-operator search step, when performing point and edge deletion operations, the point elimination and edge elimination strategies are applied to any computational graph subgraph, where point elimination includes three points, four points, and more than four points, and edge elimination includes the case where there are multiple edges between two points.

8. The method for automatic search of hybrid parallel strategies for machine learning models according to claim 1, characterized in that: In the intra-operator overhead generation step, the computation overhead is obtained by actual testing, and the communication overhead is obtained by dividing the tensor size by the bandwidth.

9. A system for automatic search of hybrid parallel strategies for machine learning models, characterized in that: include: Inter-operator parallel strategy search module: divides the pipeline stages according to the calculation graph and cluster device conditions; Inter-operator performance overhead model: The inter-operator strategy overhead is calculated based on the divided pipeline conditions and the equipment used in the pipeline stage after the division according to the intra-operator parallel strategy search module; Intra-operator parallel strategy search module: selects the optimal strategy for each operator based on the computational graph subgraph in the same stage; Intra-operator performance overhead model: The corresponding overhead is calculated based on the strategy selected by each operator in the computational graph subgraph within the same stage.

10. The system for automatic search of hybrid parallel strategies for machine learning models according to claim 9, characterized in that: The intra-operator parallel strategy search module includes: Point reduction and edge elimination rule register unit: registers the rules available in the point reduction and edge elimination process; Point reduction and edge elimination rule selector unit: selects the corresponding point reduction and edge elimination rule according to the current computational graph topology structure; Point reduction and edge elimination algorithm executor unit: executes the corresponding dynamic programming algorithm according to the selected point reduction and edge elimination rules.

Images