A deep learning model automatic parallel training method suitable for a heterogeneous cluster

Abstract

The application discloses a kind of deep learning model automatic parallel training methods suitable for heterogeneous cluster, accept the heterogeneous cluster required for training and deep learning model as input;For deep learning model, obtain its execution performance information on training system;For the cluster required for training, obtain the execution performance information of cluster, and generate the maximum homogeneous node set of cluster and node set division;Subsequently traverse each node set division, for all node sets contained in it, solve the maximum throughput of the node set;Subsequently, the maximum throughput of the current node set division is solved by the load balancing algorithm based on integer quadratic programming method, so as to solve the optimal strategy in all node set divisions and output.The application supports solving the optimal parallel strategy of given deep learning model under the setting of heterogeneous cluster.Can be applied to single machine multi-card, multi-machine multi-card and other various distributed training environments composed of homogeneous or heterogeneous devices.High training efficiency, less energy consumption.

Description

Technical Field

[0001] This invention relates to an automatic parallel training method for deep learning models suitable for heterogeneous clusters, and relates to the field of artificial intelligence technology, especially the field of parallel training technology for deep learning models. Background Technology

[0002] With the continuous development of artificial intelligence and deep learning technologies, the parameter scale of deep learning models is also constantly expanding. While these deep learning models achieve optimal performance on most tasks in their respective fields, their training costs are also constantly increasing. The performance improvement of the hardware required for training cannot keep up with the increasing cost of training deep learning models. Therefore, in actual production, multi-GPU and multi-machine environments are often used for distributed training.

[0003] In real-world production environments, heterogeneous clusters (i.e., clusters with not entirely identical hardware devices participating in training) are prevalent in distributed training contexts. For heterogeneous clusters, the computing and storage capabilities of the devices participating in training vary, making it difficult to uniformly configure training parameters and parallel strategies. In practice, configuration often results in either using the lowest-performing device as the standard, leading to underutilization of the computing power of higher-performing devices and thus low computing power utilization efficiency of the heterogeneous cluster, or only utilizing a subset of homogeneous nodes for training, leaving the computing power of other heterogeneous nodes unused and wasting computing resources. Therefore, how to automatically solve for the optimal parallel training strategy algorithm for heterogeneous clusters, ensuring that the computing power of all devices participating in training is fully utilized as much as possible, has become a research direction of significant importance.

[0004] Because heterogeneous clusters contain various computing devices with different performance levels, and parallel strategies for deep learning models also exhibit diverse parallel modes (data parallelism, tensor parallelism, pipelined parallelism, optimizer parallelism, etc.), the parallel strategies for deep learning models have a greater combination space and more constraints on nodes in heterogeneous clusters, resulting in a significantly larger actual strategy solution space compared to homogeneous clusters. Therefore, existing methods for automatically searching parallel training strategies for deep learning models are often only applicable to homogeneous clusters and cannot be extended to heterogeneous clusters, or they can only optimize a limited number or variety of parallel strategies for heterogeneous clusters, leaving considerable room for improvement in practical heterogeneous clusters. How to design a solution that is suitable for heterogeneous clusters and incorporates all existing common parallel strategy algorithms into the solution space to find better parallel strategies has become an important research problem. Summary of the Invention

[0005] Purpose of the Invention: Current methods for automatically searching for parallel training strategies for deep learning models either only support clusters composed of homogeneous nodes and cannot support heterogeneous clusters, or can only consider a limited number of parallel training strategies on heterogeneous clusters. Therefore, the above methods struggle to find more efficient parallel training strategies on heterogeneous clusters, leading to a waste of computational resources. Thus, there is a need to design a method for automatically searching for parallel training strategies for deep learning models that is suitable for heterogeneous clusters and can consider all existing common parallel training strategies.

[0006] Technical Solution: An automatic parallel training method for deep learning models suitable for heterogeneous clusters, the main steps of which are as follows:

[0007] Step S1: Input the deep learning model and heterogeneous cluster information;

[0008] Step S2: For heterogeneous clusters, use the maximum homogeneous node set generation algorithm to generate a list of the maximum homogeneous node sets;

[0009] Step S3: Using the list of the largest isomorphic node sets, generate a set of node set partitioning schemes through a depth-first search algorithm;

[0010] Step S4: For each node set partition within the set of node set partitioning schemes, enumerate all its maximum isomorphic node sets, solve the time spent in each iteration, calculate the TPI of the node set partition using a load balancing algorithm based on integer quadratic programming, and compare and update the current optimal strategy.

[0011] Step S5: Output the optimal parallel strategy for the deep learning model on the heterogeneous cluster.

[0012] Step S2, for a given heterogeneous cluster, generates a list of the largest homogeneous node set, including:

[0013] Using the maximum homogeneous node set generation algorithm, for a given heterogeneous cluster Enumeration yields the set of all the largest homogeneous nodes in the heterogeneous cluster, thus obtaining the set of nodes t. h List of maximum homogeneous node sets

[0014] The algorithm for generating the maximum homogeneous node set includes initializing a hash table hmap with node type ntype as the key and the maximum homogeneous node set as the value. ntype→T′ An empty hash table; traverse the entire heterogeneous cluster. For heterogeneous clusters For each node n in the hash table hmap, ntype→T′If a key-value pair with the node type n.ntype of the current node exists in the hash table, then update the hash table hmap. ntype→T′ Insert node n into the set corresponding to n.ntype. Otherwise, add it to the hash table hmap. ntype→T′ Insert key-value pairs<ntype,{n}> Finally, set the list of the largest isomorphic nodes to be equal to the aforementioned hash table hmap. ntype→T′ Output a list of values ​​and the list of the largest isomorphic node set.

[0015] Step S3, for the list of the largest isomorphic node sets obtained in S2, uses a depth-first search method to enumerate the set of node set partitioning schemes. Specifically, step S3 involves:

[0016] Using the depth-first search method, for the list of the largest isomorphic nodes generated in step S2... T′ Recursively enumerate to generate all valid node sets for partitioning. and a set of node set partitioning schemes The steps include:

[0017] Step S301, Input the list of the largest node set

[0018] Step S302: Initialize the node partitioning scheme size k = 0, and the node partitioning scheme set.

[0019] Step S303: Determine whether the size k of the node partitioning scheme is less than or equal to the length of the list of the largest isomorphic node set |list T′ If so, proceed to step S304; otherwise, proceed to step S307.

[0020] Step S304: For a node set partitioning scheme with k node sets, the algorithm first initializes the current partitioning scheme as a set containing k empty sets.

[0021] Step S305: Recursively search for all partition schemes containing sets of k nodes, and update... The specific steps include:

[0022] We use a recursive process based on depth-first search to enumerate all possible solutions containing k sets of nodes. This recursive process can be represented as a recursive function that accepts a list of the current maximum isomorphic node sets. T′ Current partitioning scheme The current considerations include the index of the largest isomorphic node set list element ele_id, the index of the first empty set in the current partitioning scheme insert_id, and the set of partitioning schemes. Its main function is: if the current ele_id exceeds the maximum isomorphic node set list list. T′ The range indicates that all node sets have been partitioned, and the current partitioning scheme can be applied. Add to the current set of partitioning schemes In the middle. Otherwise, consider the current list of the largest isomorphic node set. T′ The set of nodes ele_id should be placed in the current partitioning scheme. In which partition set should it be inserted either into the current partition scheme? A non-empty set of partitions, either inserted into the current partitioning scheme. The first empty partition set (i.e., the insert_id-th set). Then, recursively enumerate the placement schemes for the next node set (ele_id+1) until all node sets have been placed into the current partition scheme. It is in a certain partition set.

[0023] Step S306, update k = k + 1, proceed to step S303.

[0024] Step S307: Output the generated set of node partitioning schemes.

[0025] Step S4 refers to the set of node partitioning schemes obtained in step S3. Each node set within is divided Enumerate all sets of maximally isomorphic nodes, calculate the time total index (TPI) for each iteration, and use a load balancing algorithm based on integer quadratic programming to calculate the partitioning of the node set. Compare and update the current optimal strategy, including:

[0026] For a set of node set partitioning schemes, enumerate each node set partitioning scheme.

[0027] For the current node set partitioning For each maximally isomorphic node set in the dataset, calculate the time per iteration (TPI) for that maximally isomorphic node set.

[0028] For the current node set partitioning, a load balancing algorithm based on integer quadratic programming is used to solve for the load balancing vector l indicating the batch size allocation result. Then, the time spent on each iteration of the current node set partitioning is calculated. The load balancing algorithm of the integer quadratic programming method specifically includes:

[0029] Establish the objective function: ensure that the indicator vector l of the batch size allocated to each node set is as consistent as possible with the product of the iteration time (TPI) of that node set.

[0030] Establish batch size constraints: Let the mini-batch size during the training process of a deep learning model be B. Regardless of how the batch size is allocated to different node sets, the sum of the batch sizes allocated to all node sets must be B.

[0031] Establish parallelism constraints: ensure that the batch size allocated to each node set after load balancing can be directly divided by the number of working devices in each node set.

[0032] Thus, the indicator vector l, which indicates the batch size allocated to each node set, is solved using integer quadratic programming, thereby achieving load balancing among node sets.

[0033] Step S5, which outputs the optimal parallel strategy for the deep learning model on the heterogeneous cluster, includes:

[0034] Solution results based on the partitioning of each node set Select the optimal set of nodes for partitioning. Output the optimal strategy combination for a deep learning model under a given heterogeneous cluster. The strategy combination includes four strategies: data parallelism, tensor parallelism, pipeline parallelism, and optimizer parallelism, along with their configuration parameters.

[0035] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the automatic parallel training method for deep learning models applicable to heterogeneous clusters as described above.

[0036] A computer-readable storage medium storing a computer program that performs the automatic parallel training method for deep learning models suitable for heterogeneous clusters as described above.

[0037] Beneficial Effects: This invention supports finding the optimal parallel strategy for a given deep learning model in distributed environments such as single-machine multi-GPU, homogeneous clusters, and heterogeneous clusters. The strategy space includes data parallelism, tensor parallelism, pipeline parallelism, and optimizer parallelism strategies, using a hybrid quadratic integer programming method to quickly search for the optimal strategy. It can be applied to the above-mentioned various distributed training environments, providing the optimal parallel optimization strategy for the parallel training process of a given deep learning model. This helps improve the training efficiency of deep learning models, thereby reducing training time and energy consumption. Attached Figure Description

[0038] Figure 1 This is a flowchart of the implementation method of the present invention;

[0039] Figure 2 This is a schematic diagram of the system environment for implementing the method of the present invention;

[0040] Figure 3 This is a flowchart of step S3 of the method for implementing the present invention. Detailed Implementation

[0041] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0042] A method for automatically searching for parallel training strategies for deep learning models supports solving for the optimal parallel training strategies for various machine learning models in distributed environments with single-machine multi-GPU configurations, homogeneous and heterogeneous multi-machine multi-GPU configurations. The solution space includes data parallelism, tensor parallelism, pipeline parallelism, and optimizer parallelism training strategies, and can be applied to the above-mentioned distributed environments for distributed training optimization of deep learning models.

[0043] This embodiment utilizes a heterogeneous training environment with a total of eight GPU devices. The first four GPU devices are V100-SXM2 32GB GPUs with a clock frequency limited to 1215 MHz, a power limit of 150 watts (W), and a streaming multiprocessor frequency limit of 600 MHz. The latter four GPU devices are unrestricted V100-SXM2 32GB GPUs. The first four devices (GPU0 to GPU3) are virtualized as node N0, and the latter four devices (GPU4 to GPU7) are virtualized as node N1.

[0044] To help understand the relationship between the cluster devices in this embodiment, Figure 2 The topology of the training environment in this embodiment is illustrated. The first four GPU devices (GPU0, GPU1, GPU2, GPU3) are connected to CPU0 via a PCIe bus, and the latter four GPU devices (GPU4, GPU5, GPU6, GPU7) are connected to CPU1 via a PCIe bus. CPU0 and CPU1 are connected via a QPI bus. This embodiment applies a method for automatically searching for parallel training strategies for deep learning models in heterogeneous clusters to find the optimal parallel strategy for training the BERT-Huge model in this training environment. The specific workflow of the method for automatically searching for parallel training strategies for deep learning models in this embodiment is as follows:

[0045] like Figure 1As shown, the method for automatically searching for parallel training strategies for deep learning models first inputs the deep learning model and the heterogeneous cluster (step S1).

[0046] In this embodiment, the deep learning model is the BERT-Huge model, and the heterogeneous cluster is... Figure 2 The heterogeneous training environment of the 8-GPU device.

[0047] Then, for the input heterogeneous cluster, generate a list of the largest homogeneous node set. (Step S2)

[0048] In this embodiment, the process of generating the list of the largest isomorphic node set includes: initializing a hash table hmap with node type ntype as the key and the largest isomorphic node set as the value. ntype→T′ An empty hash table; traverse the entire heterogeneous cluster. For heterogeneous clusters For each node n in the hash table hmap, ntype→T′ If a key-value pair with the node type n.ntype of the current node exists in the hash table, then update the hash table hmap. ntype→T′ Insert node n into the set corresponding to n.ntype. Otherwise, add it to the hash table hmap. ntype→T′ Insert key-value pairs<ntype,{n}> Finally, set the list of the largest isomorphic nodes to be equal to the aforementioned hash table hmap. ntype→T′ Output the list of values.

[0049] Then, for the generated list of the largest isomorphic node sets, a depth-first search algorithm is used to generate a set of node partitioning schemes. (Step S3)

[0050] In this embodiment, a set of node partitioning schemes is generated using a depth-first search algorithm. Step S3 includes:

[0051] Using the depth-first search method, for the list of the largest isomorphic nodes generated in step S2... T′ Recursively enumerate to generate all valid node sets for partitioning. And the set of node partitioning schemes Step S3 as follows Figure 3 As shown, the specific steps include:

[0052] Step S301, Input the list of the largest node set

[0053] Step S302: Initialize the node partitioning scheme size k = 0, and the node partitioning scheme set.

[0054] Step S303: Determine whether the size k of the node partitioning scheme is less than or equal to the length of the list of the largest isomorphic node set |list T′ If so, proceed to step S304; otherwise, proceed to step S307.

[0055] Step S304: For a node set partitioning scheme with k node sets, the algorithm first initializes the current partitioning scheme as a set containing k empty sets.

[0056] Step S305: Recursively search for all partition schemes containing sets of k nodes, and update... The specific steps include:

[0057] We use a recursive process based on depth-first search to enumerate all possible solutions containing k sets of nodes. This recursive process can be represented as a recursive function that accepts a list of the current maximum isomorphic node sets. T′ Current partitioning scheme The current considerations include the index of the largest isomorphic node set list element ele_id, the index of the first empty set in the current partitioning scheme insert_id, and the set of partitioning schemes. Its main function is: if the current ele_id exceeds the maximum isomorphic node set list list. T′ The range indicates that all node sets have been partitioned, and the current partitioning scheme can be applied. Add to the current set of partitioning schemes In the middle. Otherwise, consider the current list of the largest isomorphic node set. T′ The set of nodes ele_id should be placed in the current partitioning scheme. In which partition set should it be inserted either into the current partition scheme? A non-empty set of partitions, either inserted into the current partitioning scheme. The first empty partition set (i.e., the insert_id-th set). Then, recursively enumerate the placement schemes for the next node set (ele_id+1) until all node sets have been placed into the current partition scheme. It is in a certain partition set.

[0058] Step S306, update k = k + 1, proceed to step S303.

[0059] Step S307: Output the generated set of node partitioning schemes.

[0060] Subsequently, for each node set partitioning scheme in the set of node set partitioning schemes, enumerate all the largest isomorphic node sets, solve the time spent in each iteration, use the load balancing algorithm based on integer quadratic programming to calculate the time spent in each iteration (TPI) corresponding to the current node set partitioning scheme, compare and update the current optimal strategy (step S4).

[0061] In this embodiment, for each node set partitioning scheme in the set of node set partitioning schemes, all maximum isomorphic node sets are enumerated, and the time taken for each iteration (TPI) is calculated. This includes: for the set of node set partitioning schemes, enumerating each node set partitioning scheme... For the current node set partitioning For each maximally isomorphic node set in the dataset, calculate the time per iteration (TPI) for that maximally isomorphic node set.

[0062] In this embodiment, the time to process (TPI) for each iteration of the current node set partitioning scheme is calculated using a load balancing algorithm based on integer quadratic programming. The optimal strategy is then compared and updated, including:

[0063] For the current node set partitioning, a load balancing algorithm based on integer quadratic programming is used to solve for the load balancing vector l indicating the batch size allocation result. Then, the time spent on each iteration of the current node set partitioning is calculated.

[0064] To aid in understanding the load balancing algorithm based on integer quadratic programming, this example will illustrate an integer quadratic programming mathematical model:

[0065] Objective function: Formally, assume the current node set is partitioned. There are a total Given a set of nodes, with an indicator vector l representing the batch size allocated to each node set, to ensure load balancing across node sets, the product of the batch size allocated to each node set and the iteration time per round (TPI) of that node set should be as consistent as possible. Therefore, it is only necessary to ensure that the dot product of the indicator vector and the row vector tpi formed by the TPI of each node set approximates k times (k1) of the unit vector l. Its expression can be represented by formula (1):

[0066]

[0067] Since the above formula cannot be directly used as the objective function of the integer quadratic programming algorithm, cosine similarity is introduced to perform an equivalent transformation on the above formula. At this time, in order to make the load of each node set balanced, the cosine similarity between vector l and 1 / tpi (taking the reciprocal of vector tpi element by element) is required to be close to 1, that is, formula (2) can achieve the maximum value:

[0068]

[0069] The following section explains the various constraints in conjunction with the formulas:

[0070] Constraint 1 (Batch Size Constraint): Let the mini-batch size during deep learning model training be B. Regardless of how the batch size is allocated to different node sets, the sum of the batch sizes allocated to all node sets must be B. Therefore, this constraint can be modeled as Equation (3-1):

[0071]

[0072] Constraint 2 (Parallelism Constraint): Assume a cluster The CPC has There are 1 working device. The node set is divided into groups of size [size missing]. When all nodes are used in the UniAP algorithm to compute the computing power of the node set, the batch size can be expressed as a constant vector. To ensure that the batch size allocated to each node set after load balancing is directly divisible by the number of working devices in each node set, additional parallelism constraints need to be introduced. Let the number of working devices held by all node sets be a constant vector d. Then, the parallelism constraint can be formalized as the expressions shown in equations (3-2a) to (3-2b):

[0073]

[0074] Equation (3-2a) constrains the batch size allocated to each node set after the load balancing algorithm is executed to be divisible by the number of their respective working devices. In the equation, ⊙ is the element-wise multiplication symbol for vectors. Equation (3-2b) then constrains the upper and lower bounds of the batch size allocated after the load balancing algorithm is executed.

[0075] Subsequently, by using the solution results of the load balancing algorithm based on the above-mentioned integer quadratic programming model, we can obtain the batch size indicator vector l for the current node partitioning scheme, and thus obtain the solution strategy corresponding to the current node partitioning scheme.

[0076] Finally, the optimal parallel strategy for the deep learning model on the heterogeneous cluster is output, including: the solution results based on the partitioning of each node set. Select the optimal set of nodes for partitioning. Output the optimal strategy combination for the deep learning model under a given heterogeneous cluster. The strategy combination includes four strategies: data parallelism, tensor parallelism, pipeline parallelism, and optimizer parallelism, along with their configuration parameters (step S5).

[0077] The method of this invention was experimented on multiple models and systems. During the experiments, the optimal parallel strategy for a specified deep learning model was searched using this method on the server given in the experimental examples. Subsequently, after obtaining the parallel training strategy, the method was tested on the training environment given in the experimental examples to determine the throughput of the deep learning model trained using this parallel strategy under the current environment. Experimental results on four models—BERT-Huge, T5-Large, ViT-Huge, and Swin-Huge—show that compared to manually specifying the parallel method, this method can improve training throughput by up to 11.27 times. Compared to the existing best-performing method for heterogeneous clusters, this invention can improve training throughput by up to approximately 3.65 times.

[0078] Obviously, those skilled in the art should understand that the steps of the method for automatically searching for parallel training strategies of deep learning models in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computing device-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

Claims

1. A method for automatic parallel training of deep learning models suitable for heterogeneous clusters, characterized in that, The steps are as follows: Step S1: Input the deep learning model and heterogeneous cluster information; Step S2: For heterogeneous clusters, use the maximum homogeneous node set generation algorithm to generate a list of the maximum homogeneous node sets; Step S3: Using the list of the largest isomorphic node sets, generate a set of node set partitioning schemes through a depth-first search algorithm; Step S4: For each node set partition within the set of node set partitioning schemes, enumerate all its maximum isomorphic node sets, solve the time spent in each iteration, calculate the TPI of the node set partition using a load balancing algorithm based on integer quadratic programming, and compare and update the current optimal strategy. Step S5: Output the optimal strategy for the deep learning model on the heterogeneous cluster. Step S2, which generates a list of the largest homogeneous node set for a given heterogeneous cluster, includes: Using the maximum homogeneous node set generation algorithm, for a given heterogeneous cluster Enumerate all the largest homogeneous node sets in the heterogeneous cluster, and thus obtain the list of the largest homogeneous node sets. ; The algorithm for generating the maximum isomorphic node set includes initializing a set of nodes based on node type. A hash table where the key is the set of largest isomorphic nodes and the value is the set of largest isomorphic nodes. An empty hash table; traverse the entire heterogeneous cluster. For heterogeneous clusters Each node in If hash table The node type of the current node exists in the table. If the key-value pairs are not found, then update the hash table. middle Insert node into the corresponding set Otherwise, input the hash table. Insert key-value pairs Finally, set the list of the largest isomorphic nodes to be equal to the aforementioned hash table. Output a list of values ​​and the list of the largest isomorphic node set. ; The load balancing algorithm of the integer quadratic programming method specifically includes: Define the objective function: a vector indicating the batch size allocated to each node set. Consistent with the product of the time taken for each iteration of this node set; Establish batch size constraints: Let the mini-batch size during deep learning model training be . Regardless of how the batch size is allocated to different node sets, the sum of the batch sizes allocated to all node sets must be [value missing]. ; Establish parallelism constraints: ensure that the batch size allocated to each node set after load balancing is directly divisible by the number of working devices in each node set; Therefore, the indicator vector indicating the batch size allocated to each node set can be solved using integer quadratic programming. This enables load balancing among node sets.

2. The automatic parallel training method for deep learning models applicable to heterogeneous clusters according to claim 1, characterized in that, In step S3, a depth-first search method is used to find the maximum homogeneous node set list. Recursively enumerate to generate all valid node sets for partitioning. and a set of node set partitioning schemes The steps include: Step S301, Input the list of the largest node set ; Step S302: Initialize the node partitioning scheme size Node partitioning scheme set ; Step S303: Determine the size of the node partitioning scheme. Is it less than or equal to the length of the list of the largest isomorphic node set? If so, proceed to step S304; otherwise, proceed to step S307. Step S304, for a set of nodes with a number of nodes... The algorithm first initializes the current partitioning scheme to contain the node set partitioning scheme. The set of empty sets ; Step S305, recursively search all containing Partitioning scheme for a set of nodes, updating ; Step S306, Update ; Step S307: Output the generated set of node partitioning schemes. .

3. The automatic parallel training method for deep learning models applicable to heterogeneous clusters according to claim 2, characterized in that, Recursive search of all contents Partitioning scheme for a set of nodes, updating The specific steps include: Using a recursive process based on depth-first search, for all possible values ​​containing... The recursive process of enumerating the possible schemes for each set of nodes can be represented as a recursive function that accepts a list of the current largest isomorphic node sets. Current partitioning scheme The index of the element in the list of the largest homogeneous node set currently considered. The index of the first empty set in the current partitioning scheme and a set of partitioning schemes If the current Exceeded the list of maximum isomorphic nodes The range indicates that all node sets have been partitioned, meaning the current partitioning scheme is complete. Add to the current set of partitioning schemes If so, consider the current list of the largest isomorphic nodes; otherwise, consider the list of the largest isomorphic nodes. No. The set of nodes should be placed in the current partitioning scheme. In which partition set should it be inserted either into the current partition scheme? A non-empty set of partitions, either inserted into the current partitioning scheme. The first empty partition set; then recursively enumerate the next node set ( The placement scheme continues until all node sets are placed into the current partitioning scheme. It is in a certain partition set.

4. The automatic parallel training method for deep learning models applicable to heterogeneous clusters according to claim 1, characterized in that, Step S4, for the set of node set partitioning schemes Each node set within is divided Enumerate all sets of its largest isomorphic nodes and calculate the time taken for each iteration. The load balancing algorithm based on integer quadratic programming is used to calculate the partitioning of the node set. Compare and update the current optimal strategy, including: For a set of node set partitioning schemes, enumerate each node set partitioning scheme. ; For the current node set partitioning For each maximally isomorphic set in the dataset, calculate the time taken for each iteration over that maximally isomorphic set. ; For the current node set partitioning, a load balancing algorithm based on integer quadratic programming is used to solve for the load balancing vector indicating the batch size allocation result. Then, the time spent on each iteration of partitioning the current node set can be calculated. .

5. The automatic parallel training method for deep learning models suitable for heterogeneous clusters according to claim 1, characterized in that, Step S5, which outputs the optimal parallel strategy for the deep learning model on the heterogeneous cluster, includes: Solution results based on the partitioning of each node set Select the optimal set of nodes for partitioning. It outputs the optimal strategy combination for a deep learning model under a given heterogeneous cluster. The strategy combination includes four strategies: data parallelism, tensor parallelism, pipeline parallelism, and optimizer parallelism, as well as their configuration parameters.

6. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the automatic parallel training method for deep learning models applicable to heterogeneous clusters as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that executes the automatic parallel training method for deep learning models suitable for heterogeneous clusters as described in any one of claims 1-5.

Images