Deep learning model automatic parallel training method suitable for heterogeneous cluster

Abstract

The invention discloses a deep learning model automatic parallel training method suitable for a heterogeneous cluster. The method comprises the following steps: receiving a heterogeneous cluster and a deep learning model required by training as input; for the deep learning model, obtaining execution performance information of the deep learning model on a training system; for a cluster required by training, acquiring execution performance information of the cluster, and generating a maximum isomorphic node set and node set division of the cluster; traversing division of each node set, and solving the maximum throughput of the node set for all the node sets contained in the node set; and then solving the maximum throughput of current node set division through a load balancing algorithm based on an integer quadratic programming method, thereby solving an optimal strategy in all node set division and outputting the optimal strategy. According to the method, the optimal parallel strategy for solving the given deep learning model under the heterogeneous cluster setting is supported. The method can be applied to various distributed training environments such as a single-machine multi-card environment and a multi-machine multi-card environment composed of isomorphic or heterogeneous equipment. Training efficiency is high and energy consumption is low.

Description

Technical Field

[0001] The present invention relates to an automatic parallel training method for deep learning models applicable to heterogeneous clusters, and relates to the field of artificial intelligence technology, in particular to the field of parallel training technology for deep learning models. Background Art

[0002] With the continuous development of artificial intelligence technology and deep learning technology, the parameter scale of deep learning models has also been continuously expanding. While these deep learning models have achieved the best performance in most tasks in their respective fields, their training costs have also been increasing. However, the performance improvement rate of the hardware equipment required for training is difficult to keep up with the increase in the training cost of deep learning models. Therefore, in actual production, it is often necessary to use a multi-card, multi-machine environment to conduct training in a distributed training manner.

[0003] In actual production environments, heterogeneous clusters (i.e., clusters with different hardware devices involved in training) are widely used in distributed training contexts. For heterogeneous clusters, the computing power and storage capacity of the devices involved in training vary, so it is difficult to uniformly configure training parameters and parallel strategies. In actual use, they can only be configured based on the lowest-performance device, resulting in the inability to fully utilize the computing power of higher-performance devices, which leads to low computing power utilization efficiency of heterogeneous clusters. Or only some of the homogeneous nodes in the heterogeneous cluster are used for training, resulting in the inability to utilize the computing power of other heterogeneous nodes, which also leads to a waste of computing power resources. Therefore, how to automatically solve the optimal parallel training strategy algorithm for heterogeneous clusters so that the computing power of all computing devices involved in training in the heterogeneous cluster can be fully utilized as much as possible has become a research direction with great research significance.

[0004] Since heterogeneous clusters have a variety of computing devices with different performances, there are also a variety of parallel modes for parallel strategies for deep learning models (data parallelism, tensor parallelism, pipeline parallelism, optimizer parallelism, etc.). The parallel strategies of deep learning models have more combination spaces and constraints on the nodes of heterogeneous clusters, resulting in a significant increase in the actual strategy solution space compared to homogeneous clusters. Therefore, the existing methods for automatically searching for parallel training strategies for deep learning models are often only applicable to homogeneous clusters and cannot be extended to heterogeneous clusters for application, or they optimize a limited number or a limited number of parallel strategies for heterogeneous clusters, so that the strategies obtained on actual heterogeneous clusters still have a lot of room for improvement. How to design a method that can be oriented to heterogeneous clusters and incorporate all existing common parallel strategy algorithms into the solution space to solve a better parallel strategy has become an important research issue. Summary of the invention

[0005] Purpose of the invention: At present, the parallel strategy method for automatically searching deep learning models can only support clusters composed of homogeneous nodes and cannot support heterogeneous clusters, or can only consider limited parallel strategies on heterogeneous clusters. Therefore, it is difficult for the above method to find a more efficient parallel training strategy on a heterogeneous cluster, resulting in a waste of computing resources of the heterogeneous cluster. Therefore, it is necessary to design a parallel training strategy method for automatically searching deep learning models that can be oriented to heterogeneous clusters and can take into account all existing common parallel training strategies.

[0006] Technical solution: A method for automatic parallel training of deep learning models suitable for heterogeneous clusters. The main steps are as follows:

[0007] Step S1, input deep learning model and heterogeneous cluster information;

[0008] Step S2, for heterogeneous clusters, using a maximum homogeneous node set generation algorithm to generate a maximum homogeneous node set list;

[0009] Step S3, using the maximum isomorphic node set list to generate a set of node set partitioning schemes through a depth-first search algorithm;

[0010] Step S4, for each node set partition in the node set partitioning scheme set, enumerate all its maximum isomorphic node sets, solve the time spent on each iteration, use the load balancing algorithm based on the integer quadratic programming method to calculate the TPI of the node set partition, compare and update the current optimal strategy;

[0011] Step S5, output the optimal parallel strategy for the deep learning model on the heterogeneous cluster.

[0012] The step S2, for a given heterogeneous cluster, generates a maximum homogeneous node set list, including:

[0013] Using the maximum homogeneous node set generation algorithm, for a given heterogeneous cluster Enumerate all the maximum homogeneous node sets of the heterogeneous cluster, and then obtain the h A list of maximum isomorphic node sets consisting of maximum isomorphic node sets

[0014] The maximum isomorphic node set generation algorithm includes initializing a hash table hmap with node type ntype as key and maximum isomorphic node set as value ntype→T′ It is an empty hash table; it traverses the entire heterogeneous cluster For heterogeneous clusters For each node n in the hash table hmap ntype→T′If there is a key-value pair of the node type n.ntype of the current node in the hash table, update the hash table hmap ntype→T′ Insert node n into the set corresponding to n.ntype. Otherwise, add node n to the hash table hmap. ntype→T′ Insert key-value pairs<ntype,{n}> ; Finally, let the maximum isomorphic node set list be equal to the aforementioned hash table hmap ntype→T′ 's value list and output the maximum isomorphic node set list

[0015] The step S3, for the maximum isomorphic node set list obtained in S2, uses a depth-first search method to enumerate a set of node set partitioning schemes. The step S3 is specifically as follows:

[0016] Using the depth-first search method, for the maximum isomorphic node set list generated in step S2 T′ , recursively enumerate to generate all legal node set partitions And the node set partitioning scheme set The steps include:

[0017] Step S301: input the maximum node set list

[0018] Step S302, initialize the node partitioning scheme size k = 0, the node partitioning scheme set

[0019] Step S303, determine whether the node partitioning scheme size k is less than or equal to the maximum isomorphic node set list length |list T′ |, if yes, go to step S304, otherwise go to step S307.

[0020] Step S304: for a node set partitioning scheme with k node sets, the algorithm first initializes the current partitioning scheme to a set containing k empty sets.

[0021] Step S305, recursively search all partitioning schemes containing k node sets, and update The specific steps include:

[0022] Using a recursive process based on depth-first search, all possible solutions containing k node sets are enumerated. This recursive process can be expressed as a recursive function: accept the current maximum isomorphic node set list list T′ , Current partitioning scheme The element index ele_id of the maximum isomorphic node set currently considered, the first empty set index insert_id of the current partitioning scheme, and the partitioning scheme set Its main function is: if the current ele_id exceeds the maximum isomorphic node set list list T′ range, indicating that all node sets have been divided, and the current division scheme can be Add to the current set of partitioning schemes Otherwise, consider the current maximum isomorphic node set list list T′ The ele_idth node set should be placed in the current partitioning scheme In which partition set should it be inserted into the current partitioning scheme? A non-empty set of partitions of which either is inserted into the current partitioning scheme The first empty partition set (i.e., the insert_id set) is selected. Then the placement scheme of the next node set (ele_id+1) is recursively enumerated until all node sets are placed in the current partition scheme. In a certain partition set.

[0023] Step S306, update k=k+1, and go to step S303.

[0024] Step S307: Output the generated node set partitioning scheme set

[0025] Step S4, for the node set partitioning scheme set obtained in step S3 Each node set in Enumerate all its maximum isomorphic node sets, solve the time TPI spent on each iteration, and use the load balancing algorithm based on integer quadratic programming method to calculate the node set partition Compare and update the current optimal strategy, including:

[0026] For a set of node set partitioning schemes, enumerate each node set partition

[0027] For the current node set partition For each maximum isomorphic node set in , solve the time TPI spent on each iteration of the maximum isomorphic node set.

[0028] For the current node set partition, the load balancing algorithm based on the integer quadratic programming method is used to solve the load balancing vector l indicating the batch size allocation result. Then the time spent on each iteration of the current node set partition is solved. The load balancing algorithm of the integer quadratic programming method specifically includes:

[0029] Establish the objective function: ensure that the product of the batch size indicator vector l assigned to each node set and the time per iteration (TPI) of the node set is as consistent as possible.

[0030] Establish batch size constraints: Let the mini-batch size during deep learning model training be B. No matter how the batch sizes are allocated to different sets of nodes, the sum of the batch sizes allocated to all sets of nodes must be B.

[0031] Establish parallelism constraints: Ensure that the batch size assigned to each node set after load balancing is directly divisible by the number of working devices in each node set.

[0032] Therefore, the indicator vector l indicating the batch size allocated to each node set is solved by the integer quadratic programming method to achieve load balancing between node sets.

[0033] The step S5, outputting the optimal parallel strategy of the deep learning model on the heterogeneous cluster, includes:

[0034] Solution results based on the partitioning of each node set Select the best node set partition Output the optimal strategy combination corresponding to the deep learning model under a given heterogeneous cluster. The strategy combination includes four strategies: data parallelism, tensor parallelism, pipeline parallelism, and optimizer parallelism, as well as their configuration-related parameters.

[0035] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the automatic parallel training method for deep learning models applicable to heterogeneous clusters as described above are implemented.

[0036] A computer-readable storage medium stores a computer program for executing the automatic parallel training method for deep learning models applicable to heterogeneous clusters as described above.

[0037] Beneficial effects: The present invention supports solving the optimal parallel strategy for a given deep learning model in distributed environments such as single machine with multiple cards, homogeneous clusters, and heterogeneous clusters. The strategy space includes data parallelism, tensor parallelism, pipeline parallelism, and optimizer parallelism strategies, and the optimal strategy is quickly searched out using a hybrid quadratic integer programming method. It can be applied in the above-mentioned multiple distributed training environments to provide the optimal parallel optimization strategy for the parallel training process of a given deep learning model. It helps to improve the training efficiency of deep learning models, thereby reducing training time and energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a method for implementing the present invention;

[0039] Figure 2 A schematic diagram of a system environment for implementing the method of the present invention;

[0040] Figure 3 Flow chart of step S3 of the method for implementing the present invention. DETAILED DESCRIPTION

[0041] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0042] A method for automatically searching for parallel training strategies for deep learning models. It supports solving the optimal parallel training strategies for various machine learning models in a distributed environment with multiple cards on a single machine, or multiple machines and multiple cards of the same or heterogeneous structure. The solution space includes data parallelism, tensor parallelism, pipeline parallelism, and optimizer parallel training strategies. It can be applied to the above distributed environment to perform distributed training optimization of deep learning models.

[0043] This embodiment uses a heterogeneous training environment in which a total of 8 GPU devices participate in training - the training environment has eight GPU devices, of which the first four GPU devices are V100-SXM2 32GB GPU devices with a limited clock frequency of 1215 megahertz (MHz), a power limit of 150 watts (W), and a stream multiprocessor frequency limit of 600 megahertz (MHz), and the last four GPU devices are V100-SXM2 32GB GPU devices without any restrictions. The first four devices (GPU0 to GPU3) are virtualized as node N0, and the last four devices (GPU4 to GPU7) are virtualized as node N1.

[0044] To help understand the relationship between the cluster devices in this embodiment, Figure 2 The topological structure corresponding to the training environment of this embodiment is shown. Among them, the first four GPU devices (GPU0, GPU1, GPU2, GPU3) are connected to CPU0 via the PCIe bus, and the last four GPU devices (GPU4, GPU5, GPU6, GPU7) are connected to CPU1 via the PCIe bus. CPU0 and CPU1 are connected via the QPI bus. This embodiment will apply a method for automatically searching for parallel training strategies for deep learning models for heterogeneous clusters to search for the optimal parallel strategy for training the BERT-Huge model in this training environment. In this embodiment, the specific workflow of the method for automatically searching for parallel training strategies for deep learning models is described as follows:

[0045] like Figure 1As shown, the method for automatically searching for a parallel training strategy for a deep learning model first inputs a deep learning model and a heterogeneous cluster (step S1).

[0046] In this embodiment, the deep learning model is the BERT-Huge model, and the heterogeneous cluster is Figure 2 The heterogeneous training environment of the 8GPU device.

[0047] Then, for the input heterogeneous cluster, generate the maximum homogeneous node set list (Step S2).

[0048] In this embodiment, the process of generating the maximum isomorphic node set list includes: initializing a hash table hmap with the node type ntype as the key and the maximum isomorphic node set as the value ntype→T′ It is an empty hash table; it traverses the entire heterogeneous cluster For heterogeneous clusters For each node n in the hash table hmap ntype→T′ If there is a key-value pair of the node type n.ntype of the current node in the hash table, update the hash table hmap ntype→T′ Insert node n into the set corresponding to n.ntype. Otherwise, add node n to the hash table hmap. ntype→T′ Insert key-value pairs<ntype,{n}> ; Finally, let the maximum isomorphic node set list be equal to the aforementioned hash table hmap ntype→T′ and output the list of values.

[0049] Then, for the generated maximum isomorphic node set list, use the depth-first algorithm to generate a set of node partitioning set solutions (Step S3).

[0050] In this embodiment, a depth-first search algorithm is used to generate a node partitioning scheme set Step S3 comprises:

[0051] Using the depth-first search method, for the maximum isomorphic node set list generated in step S2 T′ , recursively enumerate to generate all legal node set partitions and node set partitioning scheme set Step S3 is as follows Figure 3 As shown, the specific steps include:

[0052] Step S301: input the maximum node set list

[0053] Step S302, initialize the node partitioning scheme size k = 0, the node partitioning scheme set

[0054] Step S303, determine whether the node partitioning scheme size k is less than or equal to the maximum isomorphic node set list length |list T′ |, if yes, go to step S304, otherwise go to step S307.

[0055] Step S304: for a node set partitioning scheme with k node sets, the algorithm first initializes the current partitioning scheme to a set containing k empty sets.

[0056] Step S305, recursively search all partitioning schemes containing k node sets, and update The specific steps include:

[0057] Using a recursive process based on depth-first search, all possible solutions containing k node sets are enumerated. This recursive process can be expressed as a recursive function: accept the current maximum isomorphic node set list list T′ , Current partitioning scheme The element index ele_id of the maximum isomorphic node set currently considered, the first empty set index insert_id of the current partitioning scheme, and the partitioning scheme set Its main function is: if the current ele_id exceeds the maximum isomorphic node set list list T′ range, indicating that all node sets have been divided, and the current division scheme can be Add to the current set of partitioning schemes Otherwise, consider the current maximum isomorphic node set list list T′ The ele_idth node set should be placed in the current partitioning scheme In which partition set should it be inserted into the current partitioning scheme? A non-empty set of partitions of which either is inserted into the current partitioning scheme The first empty partition set (i.e., the insert_id set) is selected. Then the placement scheme of the next node set (ele_id+1) is recursively enumerated until all node sets are placed in the current partition scheme. In a certain partition set.

[0058] Step S306, update k=k+1, and go to step S303.

[0059] Step S307: Output the generated node set partitioning scheme set

[0060] Subsequently, for each node set partitioning scheme in the node set partitioning scheme set, all the largest isomorphic node sets therein are enumerated, the time spent on each iteration is solved, and the load balancing algorithm based on the integer quadratic programming method is used to calculate the time spent on each round of iteration (TPI) corresponding to the current node set partitioning scheme, and the current optimal strategy is compared and updated (step S4).

[0061] In this embodiment, for each node set partitioning scheme in the node set partitioning scheme set, all the maximum isomorphic node sets therein are enumerated, and the time spent on each iteration (TPI) is solved, including: for the node set partitioning scheme set, enumerating each node set partition For the current node set partition For each maximum isomorphic node set in , solve the time TPI spent on each iteration of the maximum isomorphic node set.

[0062] In this embodiment, a load balancing algorithm based on an integer quadratic programming method is used to calculate the time spent in each round of iteration (TPI) corresponding to the current node set partitioning scheme, and compare and update the current optimal strategy, including:

[0063] For the current node set partition, the load balancing algorithm based on the integer quadratic programming method is used to solve the load balancing vector l indicating the batch size allocation result. Then the time spent on each iteration of the current node set partition is solved.

[0064] To help understand the load balancing algorithm based on the integer quadratic programming method, this example will list an integer quadratic programming mathematical model:

[0065] Objective function: Formally, assuming that the current node set is divided into A total of node sets, and the indicator vector representing the batch size assigned to each node set is l. To balance the load of each node set, it is necessary to ensure that the product of the batch size assigned to each node set and the time consumed for each round of iteration (TPI) of the node set is as consistent as possible. Therefore, it is only necessary to ensure that the dot product of the indicator vector and the row vector tpi formed by each node set TPI is close to k times the unit vector 1 (k1). Its expression can be expressed as formula (1):

[0066]

[0067] Because the above formula cannot be directly used as the objective function of the integer quadratic programming algorithm, the cosine similarity is introduced to perform an equivalent transformation on the above formula. At this time, in order to balance the load of each node set, the cosine similarity of vector l and 1 / tpi (the inverse of vector tpi element by element) is required to be close to 1, that is, formula (2) can obtain the maximum value:

[0068]

[0069] The following is an introduction to the constraints in combination with the formula:

[0070] Constraint 1 (batch size constraint): Assume that the mini-batch size in the deep learning model training process is B. No matter how the batch sizes of different node sets are allocated, the sum of the batch sizes allocated to all node sets must be B. Therefore, this constraint can be modeled as formula (3-1):

[0071]

[0072] Constraint 2 (parallelism constraint): Set the cluster There are The node set is divided into When all nodes are used to calculate the node set computing power of the UniAP algorithm, the batch size can be expressed as a constant vector In order to ensure that the batch size assigned to each node set after load balancing can be directly divided by the number of working devices of each node set, it is necessary to introduce additional parallelism constraints. Assuming that the number of working devices held by all node sets is a constant vector d, the parallelism constraint can be formalized as the expressions shown in formulas (3-2a) to (3-2b):

[0073]

[0074] Formula (3-2a) constrains the batch size assigned to each node set after executing the load balancing algorithm to be divisible by the number of their respective working devices. In the formula, ⊙ is the element-by-element multiplication symbol of the vector. Formula (3-2b) constrains the upper and lower bounds of the batch size assigned after executing the load balancing algorithm.

[0075] Subsequently, through the solution result of the load balancing algorithm based on the above integer quadratic programming model, the indicator vector l of the batch size for the current node partitioning scheme can be obtained, thereby obtaining the solution strategy corresponding to the current node partitioning scheme.

[0076] Finally, the optimal parallel strategy for deep learning models on heterogeneous clusters is output, including: solution results based on the partitioning of each node set Select the best node set partition Output the optimal strategy combination corresponding to the deep learning model under a given heterogeneous cluster. The strategy combination includes four strategies: data parallelism, tensor parallelism, pipeline parallelism, and optimizer parallelism, as well as their configuration-related parameters (step S5).

[0077] The method of the present invention has been experimented on multiple models and systems. During the experiment, the method was used on the server given in the experimental example to search for the optimal parallel strategy for the specified deep learning model. Subsequently, after the parallel training strategy was searched by the method, it was tested on the training environment given in the experimental example to obtain the throughput of the deep learning model in the current environment when the parallel strategy was used for training. The experimental results on the four models of BERT-Huge, T5-Large, ViT-Huge, and Swin-Huge show that compared with the manually specified parallel method, this method can increase the training throughput by up to 11.27 times. Compared with the existing best-performing method for heterogeneous clusters, the present invention can increase the training throughput by about 3.65 times at most.

[0078] Obviously, those skilled in the art should understand that the various steps of the method for automatically searching for parallel training strategies for deep learning models in the above-mentioned embodiment of the present invention can be implemented with a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented with program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

Claims

1. A method for automatic parallel training of deep learning models suitable for heterogeneous clusters, characterized in that: Here are the steps: Step S1, input deep learning model and heterogeneous cluster information; Step S2, for heterogeneous clusters, using a maximum homogeneous node set generation algorithm to generate a maximum homogeneous node set list; Step S3, using the maximum isomorphic node set list to generate a set of node set partitioning schemes through a depth-first search algorithm; Step S4, for each node set partition in the node set partitioning scheme set, enumerate all its maximum isomorphic node sets, solve the time spent on each iteration, use the load balancing algorithm based on the integer quadratic programming method to calculate the TPI of the node set partition, compare and update the current optimal strategy; Step S5, output the optimal parallel strategy for the deep learning model on the heterogeneous cluster.

2. The automatic parallel training method for deep learning models applicable to heterogeneous clusters according to claim 1, characterized in that: The step S2, for a given heterogeneous cluster, generates a maximum homogeneous node set list, including: Using the maximum homogeneous node set generation algorithm, for a given heterogeneous cluster Enumerate all the maximum homogeneous node sets of the heterogeneous cluster to obtain the maximum homogeneous node set list The maximum isomorphic node set generation algorithm includes initializing a hash table hmap with node type ntype as key and maximum isomorphic node set as value ntype→T′ It is an empty hash table; it traverses the entire heterogeneous cluster For each node n in a heterogeneous cluster c, if the hash table hmap ntype→T′ If there is a key-value pair of the node type n.ntype of the current node in the hash table, update the hash table hmap ntype→T′ Insert node n into the set corresponding to n.ntype. Otherwise, add node n to the hash table hmap. ntype→T′ Insert key-value pairs<ntype,{n}> ; Finally, let the maximum isomorphic node set list be equal to the aforementioned hash table hmap ntype→T′ 's value list and output the maximum isomorphic node set list 3. The automatic parallel training method for deep learning models applicable to heterogeneous clusters according to claim 1, characterized in that: In step S3, a depth-first search method is used to find the maximum isomorphic node set list list T′ , recursively enumerate to generate all legal node set partitions And the node set partitioning scheme set The steps include: Step S301: input the maximum node set list Step S302, initialize the node partitioning scheme size k = 0, the node partitioning scheme set Step S303, determine whether the node partitioning scheme size k is less than or equal to the maximum isomorphic node set list length |list T′ |, if yes, proceed to step S304, otherwise proceed to step S307; Step S304: for a node set partitioning scheme with k node sets, the algorithm first initializes the current partitioning scheme to a set containing k empty sets. Step S305, recursively search all partitioning schemes containing k node sets, and update Step S306, update k=k+1, and go to step S303; Step S307: Output the generated node set partitioning scheme set 4. The automatic parallel training method for deep learning models applicable to heterogeneous clusters according to claim 3 is characterized in that: Recursively search for all partitioning schemes containing k nodes, and update The specific steps include: Using a recursive process based on depth-first search, all possible solutions containing k node sets are enumerated. The recursive process can be expressed as a recursive function: accept the current maximum isomorphic node set list list T′ , Current partitioning scheme The element index ele_id of the maximum isomorphic node set currently considered, the first empty set index insert_id of the current partitioning scheme, and the partitioning scheme set If the current ele_id exceeds the maximum isomorphic node set list list T′ range, indicating that all node sets have been divided, that is, the current division scheme Add to the current set of partitioning schemes Otherwise, consider the current maximum isomorphic node set list list T′ The ele_idth node set should be placed in the current partitioning scheme In which partition set should it be inserted into the current partitioning scheme? A non-empty set of partitions of which either is inserted into the current partitioning scheme The first empty partition set; then recursively enumerate the placement scheme of the next node set (ele_id+1) until all node sets are placed in the current partition scheme In a certain partition set.

5. The automatic parallel training method for deep learning models applicable to heterogeneous clusters according to claim 1, characterized in that: In step S4, for the node set partitioning scheme set Each node set in Enumerate all its maximum isomorphic node sets, solve the time TPI spent on each iteration, and use the load balancing algorithm based on integer quadratic programming method to calculate the node set partition Compare and update the current optimal strategy, including: For a set of node set partitioning schemes, enumerate each node set partition For the current node set partition For each maximum isomorphic node set in , solve the time TPI spent on each iteration of the maximum isomorphic node set; For the current node set partition, use the load balancing algorithm based on the integer quadratic programming method to solve the load balancing vector l indicating the batch size allocation result; then solve the time spent on each iteration of the current node set partition 6. The automatic parallel training method for deep learning models applicable to heterogeneous clusters according to claim 5, characterized in that: The load balancing algorithm of the integer quadratic programming method specifically includes: Establish the objective function: ensure that the indicator vector l of the batch size assigned to each node set is consistent with the product of the time consumed in each round of iteration of the node set; Establish batch size constraints: Let the mini-batch size during deep learning model training be B. No matter how the batch sizes are allocated to different sets of nodes, the sum of the batch sizes allocated to all sets of nodes must be B. Establish parallelism constraints: ensure that the batch size assigned to each node set after load balancing is directly divisible by the number of working devices in each node set; Therefore, the indicator vector l indicating the batch size allocated to each node set is solved by the integer quadratic programming method to achieve load balancing between node sets.

7. The automatic parallel training method for deep learning models applicable to heterogeneous clusters according to claim 1, characterized in that: The step S5, outputting the optimal parallel strategy of the deep learning model on the heterogeneous cluster, includes: Solution results based on the partitioning of each node set Select the best node set partition Output the optimal strategy combination corresponding to the deep learning model under a given heterogeneous cluster; the strategy combination includes four strategies: data parallelism, tensor parallelism, pipeline parallelism, and optimizer parallelism, as well as their configuration-related parameters.

8. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the automatic parallel training method for deep learning models applicable to heterogeneous clusters as described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the automatic parallel training method for deep learning models applicable to heterogeneous clusters as described in any one of claims 1 to 7.

Images