Pipeline training optimization method based on DAG structure deep learning model task placement
By determining the parallel mode in a heterogeneous GPU cluster and optimizing the DAG structure division of the deep learning model, the low training efficiency and load imbalance of the DAG structure deep learning model in a heterogeneous GPU environment are solved, and efficient pipeline training is achieved.
Patent Information
- Application Number
- CN202410820368.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-24
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-06-24
AI Technical Summary
In heterogeneous GPU clusters, traditional deep learning model division methods are difficult to adapt to DAG structures, resulting in low efficiency of pipeline hybrid parallel training. The use of heterogeneous GPUs complicates the task placement problem, affecting the training speed and load of deep learning models balanced.
By determining the GPU configuration of each node, generating parallel modes and selecting the optimal solution, dividing directed acyclic graphs of deep learning models, optimizing sub-graph divisions to reduce communication overhead and improve load balancing, greedy ideas and heuristic methods are adopted The problem of decomposing the parallel mode in heterogeneous GPU clusters is selected to select the optimal parallel solution to achieve minimum cutting.
It improves the distributed training efficiency of large models, ensures load balancing and low communication overhead in heterogeneous GPU environment, and improves the training speed of deep learning models.
Smart Images

Figure CN119005269A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of distributed training, and in particular to a pipeline training optimization method based on DAG structure deep learning model task placement. Background Art
[0002] With the continuous development of deep learning technology, big models have achieved remarkable results in various industries. In the power sector, the power big model released by China Southern Power Grid can provide services for various users: for ordinary power users, they can chat with the energy power big model about power policies and regulations, energy structure composition and "dual carbon" development goals, and can learn about electricity charges, power consumption, fault repair, and electricity bill deduction information from the big model to help with family energy-saving diagnosis; for professional power market users, they can consult power equipment supply and installation knowledge, obtain suggestions on power safety, protection, and energy conservation, and listen to guidance on power facility maintenance and troubleshooting; for power industry employees, they can issue instructions to them to automatically generate data processing results, accurately identify image details of defect scenes, help retrieve and process data during power inspections, identify defects, and automatically generate reports. However, as the scale of deep learning model parameters continues to increase, the requirements for computing resources and training algorithms required for training and fine-tuning big models are getting higher and higher, and distributed training methods need to be used to improve their accuracy and generalization capabilities.
[0003] The most commonly used method in distributed training is the pipeline hybrid parallel training method that combines data parallelism and model parallelism. Data parallelism can accelerate training, but it cannot support the training tasks of large-scale deep learning models that cannot be put into the video memory of a single GPU. In model parallelism, each GPU is only responsible for the calculation of a part of the deep learning model, which can reduce the consumption of video memory. However, due to the computational dependency of neural networks, the devices responsible for the training tasks of different parts of the deep learning model frequently wait for each other, resulting in a large amount of device idle time, which affects the training speed of the deep learning model. In order to improve GPU utilization, the pipeline hybrid parallel strategy was proposed and widely used. In pipeline parallelism, the data of each batch is further divided into several smaller batches (mini-batches). On the basis of model parallelism, each mini-batch is processed in sequence by different parts of the deep learning model on the device group (called worker, or stage), which effectively reduces the idle time of each GPU.
[0004] In order to maximize the efficiency of hybrid parallel training of deep learning model pipelines, the DNN (Deep Neural Network, DNN) model needs to be divided in an appropriate way. Nowadays, the computational data flow graphs of most deep learning models are no longer simple linear stacks of network layers and activation layers, but have become directed acyclic graphs (DAG) structures. The traditional linear deep learning model division method is not applicable to deep learning models with DAG structures. In addition, with the continuous advancement of technology, new GPU devices with stronger computing power are launched almost every two years, and the devices in the cluster are also updated accordingly. However, due to the cost-effectiveness of the equipment and the need to support training tasks with different acceleration requirements, usually only a part of the equipment is updated, which makes the use of heterogeneous GPUs almost inevitable in many cases. Therefore, in order to improve the training efficiency of deep learning models and ensure load balancing in the pipeline stage, it is necessary to design and implement deep learning model task placement algorithms in heterogeneous GPU scenarios to solve the problem of deep learning model division with directed acyclic graph structures.
[0005] In the problem of pipeline stage division of DAG structure deep learning model, the number of subgraphs to be divided must be determined first, and the nature of pipeline hybrid parallelism that can combine multiple devices into one stage causes uncertainty in the number of stages. At the same time, the computing load represented by different DAG nodes and the communication cost represented by different DAG edges are very different. The traditional graph partitioning algorithm method of measuring computing load by the number of nodes and measuring communication cost by the number of cut edges is no longer applicable.
[0006] In the scenario of heterogeneous GPU clusters, due to differences in computing power and bandwidth, when different nodes in the DAG are assigned to different GPU devices, the resulting computing time and communication time are significantly different. At the same time, when multiple GPUs are assigned to the same stage, the computing time and communication time of the nodes in that stage will change, and there are also differences in the impact of devices of different types and different network bandwidths, which greatly increases the complexity of the task placement problem. Summary of the invention
[0007] The present invention aims to solve at least one of the technical problems existing in the prior art.
[0008] The present invention provides the following technical solution: a pipeline training optimization method based on DAG structure deep learning model task placement, the method comprising:
[0009] Determine the possible parallel modes within all types of nodes based on the GPU configuration of each node;
[0010] Select among the parallel modes of all nodes to obtain several parallel schemes between different nodes;
[0011] The directed acyclic graph of the deep learning model is divided into subgraphs in different parallel schemes, the division results are obtained, the overall cost is calculated, and the optimal scheme is selected.
[0012] The possible parallel modes within a node are determined as follows:
[0013] The possible parallel modes of GPUs in each node are solved as unordered integer splits to obtain the parallel mode partitioning scheme within the computing node.
[0014] The process of solving and determining the node parallel mode is:
[0015] Modeling the physical resources of heterogeneous GPU clusters;
[0016] Set the initialization value of the variable during the parallel mode solution process;
[0017] Generate all possible splitting schemes through nested loops and save the final splitting scheme to partitions_list.
[0018] Components with the same computing power and bandwidth are defined as the same type. The heterogeneous GPU cluster used includes two layers of topology, namely: level 1, a node layer based on GPUs, and level 2, a cluster layer based on nodes.
[0019] Initialize an array partition of length n+1 to temporarily store the results of each split, and create an empty list partitions_l to store all the final splitting plans;
[0020] Set the starting variable partition_index to 1, which represents the position of the current operation in the splitting process, and remainder to n-1, which represents the remaining values to be split.
[0021] The inner loop tries different splitting methods based on the current starting value start_value and the remaining value remainder, and the outer loop ensures that all possible splitting solutions are found;
[0022] When generating the split result, when (2×start_value≤remainder), it means that the current start_value is used to further split the remainder. At this time, partition[partition_index] is set to start_value, and remainder and partition_index are updated;
[0023] If start_value≤remainder, add the current start_value and the updated remainder to partition, and copy the current split plan to partitions_list;
[0024] As start_value increases and remainder decreases, new splits continue to be generated until no new splits can be formed.
[0025] The process of determining the parallelization scheme between different nodes includes:
[0026] Analysis of the problem of determining parallel solutions;
[0027] Design a solution algorithm to obtain possible parallel solution selection results.
[0028] Assign n identical nodes to different parallel schemes, each scheme is considered as a group, and each node must choose a parallel scheme.
[0029] Generate all possible combinations based on the generateAllCombinations function, where each combination represents a specific parallel mode allocation scheme.
[0030] Model the computational process of a deep learning model as a directed acyclic graph;
[0031] The subgraph partitioning algorithm is executed on each parallel scheme;
[0032] After the calculation is divided, the training time of the pipeline parallelization is taken as the solution with the shortest training time.
[0033] Each operation in the deep learning model calculation process is counted as a point in the calculation graph, and the dependency between these operations will be used as an edge in the directed acyclic graph, modeling the deep learning model calculation process as a directed acyclic calculation graph.
[0034] Using the greedy idea, the score of node v in the directed acyclic graph corresponding to the deep learning model is calculated in each stage, and finally the node v is placed in the stage with the highest score.
[0035] According to the training time of the stage with the longest execution time in the pipeline hybrid parallel total time, the maximum training time of each stage in each scheme is calculated, and the scheme with the smallest time is the optimal scheme.
[0036] In the work, the present invention models the division of the pipeline stages of the DAG structure deep learning model as a subgraph partitioning problem. Taking each stage of the pipeline during training as a subgraph, the deep learning model division and task placement is to perform appropriate subgraph division to ensure load balancing and the lowest possible communication overhead, that is, the minimum cut. For each node in the graph, the objective function of the computation and communication cost of each subgraph it is divided into is calculated, and finally it is divided into the subgraph with the minimum cost.
[0037] Based on the heuristic idea, the pipeline mixed parallel mode problem of all nodes in the heterogeneous GPU cluster is decomposed into the possible parallel mode problem of GPU in each node and the parallel scheme selection problem of each node. According to the GPU configuration of each node, the possible parallel mode inside the node of this type is determined, that is, the number of division stages inside the node is determined, and then all nodes are allowed to choose from the parallel modes obtained in the previous step to obtain several parallel schemes. Finally, the DAG of the deep learning model is divided into subgraphs in different parallel schemes, the division results are obtained, the overall cost is calculated, and the optimal scheme is selected.
[0038] The present invention can solve the problems of deep learning model partitioning and task placement in the presence of GPU heterogeneity and bandwidth heterogeneity, combining the DAG structure of the deep learning model, and improve the distributed training efficiency of large models. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 A schematic diagram of the division of a directed acyclic graph structure deep learning model in a GPU cluster;
[0040] Figure 2 It is a schematic diagram of a specific implementation of the method of the present invention. DETAILED DESCRIPTION
[0041] The technical solution provided by the present invention will be described in detail below in conjunction with specific embodiments. It should be understood that the following specific implementation methods are only used to illustrate the present invention and are not used to limit the scope of the present invention.
[0042] The present invention provides a pipeline training optimization method based on DAG structure deep learning model task placement, comprising the following steps:
[0043] Step 1: Aiming at the GPU heterogeneity problem in computing clusters, design an algorithm to determine the possible parallel modes within all types of nodes based on the GPU configuration of each node, and then determine the number of stages within the node. GPU clusters are usually expanded by introducing new computing nodes equipped with new GPUs, rather than adding new GPUs to existing nodes. Therefore, in heterogeneous GPU clusters, the heterogeneity of GPUs is generally reflected in the heterogeneity of computing nodes, that is, there are multiple nodes in the cluster, and different types of nodes may usually be equipped with different types of GPUs, while the GPUs within the nodes are usually isomorphic.
[0044] First, the problem of possible parallel modes of GPUs in each node is solved as an unordered integer splitting problem. Assuming that a certain type of node has m homogeneous GPUs, then determining the parallel mode of this node is equivalent to dividing the m GPUs into i stages. Since at least one GPU is required in each stage, the problem of parallel modes within the node is an integer splitting problem at this time. In addition, in the initial case of graph partitioning, there are no points corresponding to the computing operations assigned to the stages corresponding to the GPUs. At the same time, the parameters of the stages corresponding to the homogeneous GPUs in the node are exactly the same (computing power, communication bandwidth between GPUs, external bandwidth, etc.). The stages corresponding to each GPU in the same node are completely equivalent, and the order can be exchanged with each other without any impact when the points are not assigned. Therefore, the problem of possible parallel modes within each node is essentially an unordered integer splitting problem at this time. According to existing research, this problem can be solved in O(n 2 / 2) time complexity.
[0045] Step 2: In pipeline hybrid parallelism, the subgraphs in the graph partitioning algorithm correspond to the hybrid parallel stages, and the graph partitioning algorithm needs to first determine the number of subgraphs and the various parameters of each subgraph (such as computing power, bandwidth, etc.), that is, to determine the number of stages and the GPU configuration corresponding to each stage. Therefore, based on a heuristic idea, the pipeline hybrid parallel mode problem of all nodes in the heterogeneous GPU cluster is decomposed into the possible parallel mode problem of the GPU in each node and the parallel scheme selection problem of each node.
[0046] The possible parallel modes for each node have been determined in step 1. At this time, each node needs to select one of these solutions to execute. Assuming that there are n nodes of a certain type and the number of corresponding parallel solutions is j, then determining the parallel solutions for all nodes of this type is equivalent to dividing the n nodes into j groups. Because the stages corresponding to nodes of the same type are equivalent in the initial state and can be exchanged in order without affecting the training, the parallel solution selection problem for each node is a problem of grouping the same elements, and the number of elements in the group can be zero (the solution can not be executed). This problem can be solved by the partition method at this time, and the time complexity is O(n).
[0047] Step 3: After executing the parallel mode determination and parallel scheme selection in steps 1 and 2, the calculation process of the deep learning model is modeled as a directed acyclic graph (DAG), and then the DAG of the deep learning model is divided into subgraphs in different parallel schemes obtained in step 2, the division results are obtained, the overall cost is calculated, and the optimal scheme is selected. That is, the pipeline stages and their hardware configurations are set first, and then the deep learning model is divided according to the different device topologies represented by these schemes, thereby solving the problem of the uncertain number of subgraphs.
[0048] Specifically, step 1 includes:
[0049] Step 1-1: First, model the physical resources of the heterogeneous GPU cluster to determine the hierarchical structure of the GPU cluster. In a GPU cluster, the GPUs within a node are usually homogeneous, while the GPUs of different nodes may be heterogeneous; in addition, the bandwidth within the node is homogeneous, while the internal bandwidth of different nodes may differ, and the bandwidth between different nodes may also differ, so the GPU is divided into a hierarchical structure of L layers. Levelk is composed of levelk-1 components, and they are used as division units when dividing stages, and level1 uses GPUs as division units. We define components with the same computing power and bandwidth as the same type, and the heterogeneous GPU cluster used has only two layers of topology, namely the node layer level1 with GPUs as units and the cluster layer level2 with nodes as units. To this end, the present invention models the physical resources as follows:
[0050] Table 1 Physical resource modeling in heterogeneous GPU clusters
[0051]
[0052]
[0053] Step 1-2: After modeling the physical resources of the GPU cluster, it is necessary to determine the internal parallel mode for the computing node at level 2, that is, to give a distribution plan for a fixed number of GPUs between different stages. The key points here include the following two points: 1) The GPU parallel mode problem within the node: m homogeneous GPUs are allocated to i different stages, and at least one GPU needs to be allocated to each stage. This problem can be regarded as an integer splitting problem, that is, dividing the integer m into the sum of i positive integers, each positive integer represents the number of GPUs allocated to a stage. 2) Unordered integer splitting: Since all GPUs in the node are homogeneous and are completely equivalent before no specific computing tasks are assigned. This means that the order of division does not affect the result, thereby simplifying the problem to an unordered integer splitting problem.
[0054] This step first sets the initialization value of the parallel mode. Create an array partition of length n+1, and initialize all elements to 0. This array is used to temporarily store the results of each split. At the same time, create an empty list partitions_list to store all the final split plans. partition_index is set to 1, representing the position of the current operation in the split process. remainder is the remaining value to be split, and the initial value is n-1.
[0055] Step 1-3: After setting the initial value in the previous step, this step needs to be solved by nested loops to find all possible splitting solutions. The outer loop ensures that the algorithm continues to run until all possible splitting solutions are found. The inner loop tries different splitting methods based on the current start_value and remainder. start_value represents the value currently considered for inclusion in the splitting solution. It starts from partition[partition_index-1]+1 to ensure the order of the split (that is, each subsequent number is not less than the previous number).
[0056] Then the number of split schemes is generated. When (2×Start_value≤remainder), it means that the current start_value can be used to further split the remainder. At this time, partition[partition_index] is set to start_value, and remainder and partition_index are updated. Then, if start_value≤remainder, the algorithm will add the current start_value and the updated remainder to the partition, and copy the current split scheme to partitions_list. As start_valne increases and remainder decreases, the algorithm continues to generate new split schemes until no new splits can be formed. Finally, all possible split schemes are collected in partitions_list and used as the return value of the algorithm. The above process is shown in Algorithm 1 in Table 2.
[0057]
[0058] Table 2
[0059] This algorithm can quickly derive all possible splitting solutions for an integer n, ensure that each generated splitting solution is unique by increasing the number in the splitting result, and gradually build each possible splitting solution by dynamically adjusting the start_value and remainder. The whole process not only ensures the integrity of the splitting solution, but also ensures the efficiency of the algorithm.
[0060] Furthermore, the process of determining the parallel mode in step 2 includes the following steps:
[0061] Step 2-1: Analysis of the problem of determining parallel solutions. The determination of parallel solutions is defined as assigning n identical nodes to j different parallel solutions. Each solution can be regarded as a group, and each node must choose a parallel solution. Since each node is of the same type, they are interchangeable, which means that different grouping methods will not affect the final calculation results.
[0062] Step 2-2: After analyzing the problem in the previous step, this step needs to provide a problem-solving algorithm to determine the parallel solution for each node. The algorithm flow is shown in Algorithm 2 in Table 4.
[0063]
[0064] Table 4
[0065] The generateplanSelectionResults function is the main function of the algorithm, which is responsible for generating all possible parallel mode selection results, that is, the number of each mode selected, which can be zero. It uses the generateAllCombinations function to generate all possible combinations, where each combination represents a specific parallel mode allocation scheme. The generateAllCombinations function generates all possible combinations, that is, all possible parallel mode allocation methods. It constructs each combination recursively. In the specific implementation, the existing method can be directly called to generate, so it will not be repeated in the pseudo code.
[0066] Furthermore, step 3 of dividing the deep learning model DAG into subgraphs in different parallel schemes and selecting the optimal scheme includes the following steps:
[0067] Step 3-1: First, model the computational process of the deep learning model. The computational graph of the deep learning model will be constructed into a directed acyclic graph (DAG). Each operation in the deep learning model computation process is counted as a point in the computational graph, such as convolution operations, normalization operations, activation functions, and tensor addition calculations, and the dependencies between these operations will be an edge in the directed acyclic graph.
[0068] For deep learning model training in heterogeneous GPU clusters, the parameters of the deep learning model and the information of the intermediate results are fixed after the hyperparameters and the deep learning model structure are given. However, due to the difference in computing time caused by the difference in computing power of different types of GPUs, each combination of GPU and deep learning model needs to be characterized separately. By analyzing the training process of a batch of deep learning models through Profiler, the data corresponding to each point v in the figure can be obtained as shown in Table 3.
[0069] Table 3. Modeling of deep learning model calculation process
[0070]
[0071] Step 3-2: After modeling the calculation process of the deep learning model, it is necessary to perform subgraph partitioning on the directed acyclic graph in each parallel scheme. Specifically, this method adopts the greedy idea. After calculating the score of the node v in the directed acyclic graph corresponding to the deep learning model in each stage, the node v is placed in the stage with the highest score. Among them, time_unit is the minimum time between the calculation time and the communication time of all nodes in the graph, which is used to standardize the time data involved in the calculation; capacity is the load limit of each stage. After obtaining the load of the stage through the getStageTime function, if the load has exceeded the load limit, no more nodes should be added to it to ensure load balancing; α and γ are hyperparameters for the greedy strategy score calculation.
[0072] Existing graph partitioning algorithms usually use the size of the point set to constrain the subgraph load, and use the number of neighbors of the current node v in the current subgraph to constrain the edge cutting. However, this method is based on the actual training characteristics of hybrid parallelism in heterogeneous GPU clusters, and redesigns the score calculation formula of the greedy strategy as follows:
[0073] Score = t comm -α*γ*(t comp ) γ-1 #(1)
[0074] The meaning of this formula is: the communication time t between node v and the existing nodes in the current stage comp Constrain the edge cutting, with v's computation time t in the current stage comp Constrain the stage load, that is, compare the communication cost saved by putting node v into the current stage with the computational overhead saved by not putting it into the current stage, so as to obtain the score of putting node v into the stage.
[0075] Step 3-3: After the division in the previous step, this step needs to calculate the training time of the deep learning model after division under the current parallel scheme. Specifically, this method executes Algorithm 3 (as shown in Table 5) for each scheme obtained by Algorithm 2, and then calculates the maximum training time of each stage in each scheme based on the principle that the total time of pipeline hybrid parallelism is the training time of the stage with the longest execution time. The scheme with the shortest time is the optimal scheme. Let the time of the optimal scheme be T best , d and s are the scheme and stage respectively, D and S are the scheme set and stage set respectively, then:
[0076] T best =min d∈D max s∈S (T S )#(2)
[0077]
[0078]
[0079] Table 5
[0080] The present invention aims at the problem of partitioning deep learning models in a hybrid parallel environment of heterogeneous GPU clusters, and considers the directed acyclic graph structure corresponding to the deep learning framework calculation graph and the deep learning model structure, so as to improve the pipeline training efficiency of the DAG structure deep learning model. This method solves the problem of task placement in the heterogeneous GPU pipeline training of the deep learning model with DAG structure, so as to achieve load balancing in the pipeline stage.
[0081] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.
Claims
1. A pipeline training optimization method based on DAG structure deep learning model task placement, characterized by: The method includes: Determine the possible parallel modes within all types of nodes based on the GPU configuration of each node; Select among the parallel modes of all nodes to obtain several parallel schemes between different nodes; The directed acyclic graph of the deep learning model is divided into subgraphs in different parallel schemes, the division results are obtained, the overall cost is calculated, and the optimal scheme is selected.
2. The pipeline training optimization method based on DAG structure deep learning model task placement according to claim 1 is characterized in that: The possible parallel modes within a node are determined as follows: The possible parallel modes of GPUs in each node are solved as unordered integer splits to obtain the parallel mode partitioning scheme within the computing node.
3. The pipeline training optimization method based on DAG structure deep learning model task placement according to claim 2 is characterized in that: The process of solving and determining the node parallel mode is: Modeling the physical resources of heterogeneous GPU clusters; Set the initialization value of the variable during the parallel mode solution process; Generate all possible splitting schemes through nested loops and save the final splitting scheme to partitions_list.
4. The pipeline training optimization method based on DAG structure deep learning model task placement according to claim 3 is characterized in that: Components with the same computing power and bandwidth are defined as the same type. The heterogeneous GPU cluster used includes two layers of topology, namely: level 1, a node layer based on GPUs, and level 2, a cluster layer based on nodes.
5. The pipeline training optimization method based on DAG structure deep learning model task placement according to claim 3 is characterized in that: Initialize an array partition of length n+1 to temporarily store the results of each split, and create an empty list partitions_list to store all the final split plans; Set the starting variable partition_index to 1, which represents the current position of the operation in the split process, and remainder to n-1, which represents the remaining values to be split.
6. The pipeline training optimization method based on DAG structure deep learning model task placement according to claim 3 is characterized by: The inner loop tries different splitting methods based on the current starting value start_value and the remaining value remainder, and the outer loop ensures that all possible splitting solutions are found; When generating the split result, when (2×start_value≤remainder), it means that the current start_value is used to further split the remainder. At this time, partition[partition_index] is set to start_value, and remainder and partition_index are updated; If start_value≤remainder, add the current start_value and the updated remainder to partition, and copy the current split plan to partitions_list; As start_value increases and remainder decreases, new splits continue to be generated until no new splits can be formed.
7. The pipeline training optimization method based on DAG structure deep learning model task placement according to claim 1 is characterized in that: The process of determining the parallelization scheme between different nodes includes: Analysis of the problem of determining parallel solutions; Design a solution algorithm to obtain possible parallel solution selection results.
8. The pipeline training optimization method based on DAG structure deep learning model task placement according to claim 7 is characterized in that: Assign n identical nodes to j different parallel schemes, each scheme is considered as a group, and each node must choose a parallel scheme.
9. The pipeline training optimization method based on DAG structure deep learning model task placement according to claim 7, characterized in that: Generate all possible combinations based on the generateAllCombinations function, where each combination represents a specific parallel mode allocation scheme.
10. The pipeline training optimization method based on DAG structure deep learning model task placement according to claim 1, characterized in that: Model the computational process of deep learning models as a directed acyclic graph; The subgraph partitioning algorithm is executed on each parallel scheme; After the calculation is divided, the training time of the pipeline parallelization is taken as the solution with the shortest training time.
11. The pipeline training optimization method based on DAG structure deep learning model task placement according to claim 10, characterized in that: Each operation in the deep learning model calculation process is counted as a point in the calculation graph, and the dependency between these operations will be used as an edge in the directed acyclic graph, modeling the deep learning model calculation process as a directed acyclic calculation graph.
12. The pipeline training optimization method based on DAG structure deep learning model task placement according to claim 10, characterized in that: Using the greedy idea, the score of the node v in the directed acyclic graph corresponding to the deep learning model is calculated in each stage, and finally the node v is placed in the stage with the highest score.
13. The pipeline training optimization method based on DAG structure deep learning model task placement according to claim 10, characterized in that: According to the training time of the stage with the longest execution time in the pipeline hybrid parallel total time, the maximum training time of each stage in each scheme is calculated, and the scheme with the smallest time is the optimal scheme.
Citation Information
Patent Citations
Data flow compilation optimization method oriented to multi-core cluster
CN103970580A
Heterogeneous network perception model division and task placement method in pipelined distributed deep learning
CN110533183A
Assembly line parallel training task allocation method based on deep reinforcement learning
CN113312178A
Large model hybrid parallel training acceleration method and system
CN116755876A
Assembly line parallel method for accelerating neural network training in heterogeneous GPU cluster
CN116883229A
Cited By
Deep learning large model training method and system for heterogeneous equipment
CN119557113A
Assembly line check point operation method and operation system thereof
CN119623585A
Graph structure-based machine learning Pipeline generation and deployment method and system
CN121303272A
AI model automatic parallel segmentation method oriented to multiple GPU servers
CN121436211A
An AI model automatic parallel cutting method for multi-GPU server
CN121436211B
