A heterogeneous-aware data parallel training method for consumer-grade GPU clusters

Abstract

The application belongs to the field of data training, and particularly relates to a heterogeneous perception data parallel training method for a consumer-level GPU cluster. Resource self-adaptive sub-models are generated for heterogeneous consumer-level GPUs through model pruning to decouple layer-by-layer dependency, and then the gradients of the overlapping areas are synchronized across GPUs. Specifically, a sub-model generator and a cluster-based reduction mechanism are designed to quickly generate resource self-adaptive sub-models and realize efficient gradient synchronization. An efficient offline analyzer is designed to immediately determine the optimal configuration (micro-batch size and sub-model size) of parallel training, so as to balance memory utilization and computing efficiency and realize load balancing.

Description

Technical Field

[0001] This application belongs to the field of data training, specifically involving a parallel training method for heterogeneous perceptual data for consumer-grade GPU clusters. Background Technology

[0002] With the rapid popularization of large-scale deep learning models in language, vision and multimodal tasks, training costs and resource thresholds continue to rise. Due to the price and high resource consumption of data center-grade GPUs, many academic research and industrial practices have begun to shift to training using heterogeneous consumer-grade GPU clusters composed of different generations, heterogeneous memory capacity and computing power. This type of scenario is based on multi-card collaboration, but the inconsistency of memory, computing power and link bandwidth between devices makes it difficult for traditional data parallelism (DP), tensor parallelism (TP), pipeline parallelism (PP) and its hybrid form (HP) to fully utilize memory and computing power at the same time, thus restricting throughput and convergence efficiency. In a heterogeneous environment, the mainstream parallel paradigm generally faces the following bottlenecks: (1) computational idleness caused by hierarchical dependence. After TP / PP is split into operators or hierarchical levels, forward-backward propagation requires frequent all-gather, reduce-scatter and other synchronous operations. Weak computing power or slow link nodes become "shortcuts", causing strong computing power nodes to wait significantly; (2) nonlinear matching of memory and computing power. The computing power increase of different GPU models is not proportional to the memory capacity, making it difficult to balance "segmentation based on memory compatibility" and "segmentation based on computing power balance", ultimately reducing overall efficiency. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention proposes a heterogeneous perceptual data parallel training method for consumer-grade GPU clusters. Specifically, the core design of the framework involves generating resource-adaptive sub-models for heterogeneous consumer-grade GPUs through model pruning to decouple layer-by-layer dependencies, and then synchronizing gradients across overlapping regions on the GPU. Specifically, a sub-model generator and a cluster-based reduction mechanism are designed to quickly generate resource-adaptive sub-models and achieve efficient gradient synchronization. An efficient offline analyzer is designed to immediately determine the optimal configuration for parallel training (micro-batch size and sub-model size), thereby balancing memory utilization and computational efficiency to achieve load balancing.

[0004] The technical solution adopted by this invention to solve its technical problem is: a parallel training method for heterogeneous perceptual data for consumer-grade GPU clusters, which includes the following steps:

[0005] A parallel training method for heterogeneous sensing data for consumer-grade GPU clusters, characterized by the following steps:

[0006] S1: Construct a consumption model to characterize the consumption of a single round of training;

[0007] S2: Analyze and estimate the key factors of the constructed consumption model and design a fast search algorithm to search for the optimal distributed training configuration;

[0008] S3: Generate structured masks for large models based on training configurations;

[0009] S4: Heterogeneous worker nodes generate heterogeneous-aware sub-models based on resource adaptive masks and train them on unique micro-batches;

[0010] S5: Design a cluster-based communication protocol mechanism for gradient aggregation.

[0011] Preferably, a cost model is constructed to characterize the cost of a single training round of the model. This cost to be optimized represents the time required for training with the same parameters in a single training round. :

[0012] ;

[0013] in, Represents the training data, and Indicates the size of the training data. , and These represent the communication time, computation time, and video memory usage in a single iteration, respectively. For micro-batch size; Size of the sub-model; For training nodes Computational power; This represents the total number of GPUs.

[0014] Preferably, the key factors of the constructed consumption model are analyzed and estimated:

[0015] estimate :

[0016] ;

[0017] in, , and Subgroups The total communication scale, the slowest link bandwidth, and the number of training nodes with the slowest link. For the first A set of sub-communication groups for each cluster class. Indicates the total number of communication clusters;

[0018] estimate :

[0019] ;

[0020] Estimation: Estimation is performed using a linear model.

[0021] The preferred search algorithm flow is as follows:

[0022] S201. Determine the complete model as follows and has Each layer, heterogeneous cluster, has Each GPU has a video memory capacity of [number] GPUs. and computing power Let the current optimal time be . ;

[0023] S202. From Start by iterating through the global batch size and selecting the optimal training configuration by continuously comparing the model time consumption of different configurations;

[0024] S203. Solve for the optimal training configuration under the current global batch size by maximizing memory usage and balancing heterogeneous computing load. ;

[0025] S204. Calculate the current training time and compare it with the optimal time, and finally iterate to select the optimal value;

[0026] S205. After the node, the training configuration obtained from the search is sent to all heterogeneous training nodes for sub-model pruning to obtain the optimal sub-model size.

[0027] Preferably, S3 designs a uniform structured pruning algorithm and generates the optimal structured mask based on the optimal sub-model size obtained in S2. The steps are as follows:

[0028] S301. Determine the complete model as and has Each layer has a 2D matrix (cnt) used to count the number of times each parameter region in each layer of the model is used, and a 2D matrix (Score) used to count the importance score of each parameter region in each layer of the model. For each node... Let its mask be . and , representing the columns and rows that need to be retained, respectively;

[0029] S302. Traversal Each layer is allocated according to its layer;

[0030] S303. Determine whether the currently divided layer is a linear layer or a multi-head attention layer. If it is a linear layer, proceed to the next step; otherwise, proceed to S305.

[0031] S304. Record the current parameter range size. The output dimension of the linear layer, i.e. ;

[0032] S305. Record the current parameter range size. The number of attention heads, i.e. ;

[0033] S306. Statistical analysis of assigned levels The entire parameter range ,for Subparameter region within Its fraction is: ;

[0034] S307. Scores based on statistics In ascending order Sort;

[0035] S308. Traverse each node and record the current allocation index as... ;

[0036] S309. For parameter range ,remember And the row of the next level is equal to the column of the current level, that is ;

[0037] S310 Update ;Finish.

[0038] Preferably, in step S4, for multi-head attention layers, the obtained mask determines which head needs to be pruned. Specifically, for linear layers, the obtained mask determines which output layers need to be pruned, and the input layers also need to be pruned accordingly to generate a complete sub-model. Based on the sub-model, a corresponding optimizer is constructed for training. A data extractor is constructed using the optimal micro-batch size obtained in step S1. Nodes use the data extractor, sub-model, and corresponding optimizer for a single round of training to generate the gradient of the sub-model.

[0039] Preferably, step S5 designs a gradient reduction mechanism based on graph coloring: the master training node first counts the parameter regions of all training nodes, and allows training nodes with the same parameter region to directly perform gradient reduction. Different combinations of training nodes do not affect each other. A graph coloring algorithm is used to adjust the reduction order, setting training nodes with the same parameter region as a point. Points with common training nodes have edges. In this way, a graph is formed and the graph coloring algorithm is used to ensure that combinations of training nodes in the same cluster do not have common training nodes. Thus, all combinations of training nodes in a cluster can be reduced simultaneously.

[0040] Preferably, S5's cluster reduction mechanism based on the greedy graph coloring algorithm includes the following steps:

[0041] S501. Construct a graph, traverse all sub-communication groups, set sub-communication groups with the same parameter region as a point, and add an edge between sub-communication groups with a common training node.

[0042] S502. Set an empty color set to store the sub-communication groups that have been colored, and initialize color k to 0;

[0043] S503. Loop through all vertices until all vertices are colored, first take the next color k=k++;

[0044] S504. Consider all vertices in turn; if a vertex has already been colored, continue to consider the next vertex; if vertex i is colored with color k without conflict, then color it.

[0045] S505. Group the data into different clusters according to the same color, and finally output the sub-communication groups of all clusters and their corresponding communication parameter areas.

[0046] Compared with the prior art, the beneficial effects of this application are as follows:

[0047] This invention addresses the inefficiency of distributed training on consumer-grade server clusters by employing a heterogeneous sensing data parallel training framework for consumer-grade GPU clusters. Building upon data parallelism, this invention introduces structured model pruning, proposing a heterogeneous sensing data parallel training framework for consumer-grade GPU clusters.

[0048] This invention proposes Helix, a heterogeneous perceptual data parallel variant for resource-efficient large-scale model training on heterogeneous consumer GPUs. The core design concept of Helix is ​​to generate resource-adaptive sub-models for heterogeneous consumer GPUs through model pruning, thereby decoupling hierarchical dependencies and synchronizing gradients in overlapping regions across GPUs.

[0049] Helix employs a sub-model generator and a cluster-based Reduce mechanism to rapidly generate resource-adaptive sub-models and achieve efficient gradient synchronization. The offline analyzer quickly determines the optimal parallel training configuration by constructing a consumption model tailored to the designed parallel mechanism and using a fast search algorithm to balance the load while considering both memory utilization and computational efficiency. Attached Figure Description

[0050] Figure 1 This is a schematic flowchart of the method of the present invention;

[0051] Figure 2 This is a schematic diagram of the uniform structured pruning algorithm in the Helix framework;

[0052] Figure 3 This is a schematic diagram of the cluster-based communication protocol mechanism in the Helix framework. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0054] Example 1 This example provides a parallel training method for heterogeneous perceptual data for consumer-grade GPU clusters. The core design concept of this parallel framework is to generate resource-adaptive sub-models for heterogeneous consumer-grade GPUs through model pruning, thereby decoupling hierarchical dependencies and synchronizing gradients in overlapping regions across GPUs. The process is as follows: Figure 1 As shown, the main steps include:

[0055] S1. Construct a consumption model to characterize the single-round training of the model, and analyze and estimate the important factors.

[0056] In this step, a cost model is constructed to characterize the cost of a single round of training. This cost to be optimized represents the time required for training with the same parameters in a single round. As shown below.

[0057] .

[0058] in, , and These represent the communication time, computation time, and GPU memory usage in a single iteration, respectively, which are factors affecting the training cost of the model. The three key factors. Furthermore, these three factors are further influenced by the size of the sub-model. and micro-batch size And this is determined by the device's own computing power. Additionally, Represents the training data, and Indicates the size of the training data. This represents the total number of iterations in each training round.

[0059] S2. Estimate the three key factors mentioned above.

[0060] for In this regard, we designed a novel cluster-class reduction communication mechanism based on the full reduction mechanism, therefore we estimate... As shown below:

[0061] ;

[0062] in, For the first A set of sub-communication groups for each cluster class. This indicates the total number of communication clusters. , and Subgroups The total communication scale, the slowest link bandwidth, and the number of training nodes with the slowest link. Specifically, Subgroup The structured mask corresponding to the parameter region is determined by the mask generated using the proposed pruning module. The decision will be described in detail in step three.

[0063] for In other words, it is determined by the current workload and computing power. Furthermore, for and We can use a linear model to estimate it.

[0064] Next, a fast search algorithm was designed that compresses the search space using two dimensions: maximizing memory usage and balancing heterogeneous computing load. Based on this, the optimal micro-batch size and sub-model size are determined for heterogeneous nodes through continuous iterative search. The specific search algorithm flow is as follows:

[0065] S201. Determine the complete model as follows and has Each layer, heterogeneous cluster has Each GPU has a video memory capacity of [number] GPUs. and computing power Let the current optimal time be... ;

[0066] S202. From Start by iterating through the global batch size and selecting the optimal training configuration by continuously comparing the model time consumption of different configurations;

[0067] S203. First, the optimal training configuration under the current global batch size is solved by maximizing memory usage and balancing heterogeneous computing load. ;

[0068] S204. Calculate the current training time and compare it with the optimal time, and finally iterate to select the optimal value;

[0069] S205. After the master node, the obtained training configuration is sent to all heterogeneous training nodes for sub-model pruning. End.

[0070] S3. Considering that complex pruning algorithms reduce computational efficiency, while simple pruning algorithms struggle to guarantee high convergence efficiency, such as... Figure 2 As shown, the steps for designing a uniform structured pruning algorithm and generating the optimal structured mask based on the optimal sub-model size obtained in step two are as follows:

[0071] S301. Determine the complete model as and has Each layer, the nodes obtained in step two above. The corresponding sub-model size is A two-dimensional matrix `cnt` is set up to count the number of times each parameter region in each layer of the model is used, and a two-dimensional matrix `Score` is set up to count the importance score of each parameter region in each layer of the model. For each node... Let its mask be . and , representing the columns and rows that need to be retained, respectively;

[0072] S302. Traversal Each layer is allocated according to its layer;

[0073] S303. Determine whether the currently divided layer is a linear layer or a multi-head attention layer. If it is a linear layer, proceed to the next step; otherwise, proceed to step (5).

[0074] S304. Record the current parameter range size. The output dimension of the linear layer, i.e. ;

[0075] S305. Record the current parameter range size. The number of attention heads, i.e. ;

[0076] S306. Statistical analysis of assigned levels The entire parameter range ,for Subparameter region within Its score is ; The model represents the first The first layer Parameters in each parameter region

[0077] S307. Scores based on statistics In ascending order Sort;

[0078] S308. Traverse each node and record the current allocation index as... ;

[0079] S309. For parameter range ,remember And the row of the next level is equal to the column of the current level, that is ;

[0080] S310 Update = ;

[0081] S311. End.

[0082] S4. Heterogeneous worker nodes generate resource-adaptive sub-models using the structured mask obtained in step two. For multi-head attention layers, the obtained mask determines which heads need pruning. Specifically, for linear layers, the obtained mask determines which output layers are pruned. Furthermore, since it is structured pruning, pruning the current output layer will reduce the output dimension, so the input layer needs to be pruned simultaneously to generate a complete sub-model. An optimizer is then constructed based on the sub-model for training. Additionally, a data extractor is constructed using the optimal micro-batch size obtained in step one. The node uses the data extractor, the sub-model, and the corresponding optimizer for a single training epoch to generate the gradient of the sub-model.

[0083] S5. Considering that each heterogeneous training node has sub-models with different training parameter regions, the existing full reduction mechanism leads to redundant communication. Therefore, an efficient graph coloring-based gradient reduction mechanism needs to be designed, such as... Figure 3 As shown. The master training node first calculates the parameter regions of all training nodes and allows training nodes with the same parameter regions to directly perform gradient reduction; different combinations of training nodes do not affect each other. Furthermore, to further accelerate the parallelism of communication, we designed a cluster reduction mechanism based on a greedy graph coloring algorithm, as follows: Figure 3 As shown. It mainly includes the following steps:

[0084] S501. Construct a graph. Traverse all sub-communication groups, set sub-communication groups with the same parameter region as a node, and add an edge between sub-communication groups with a common training node;

[0085] S502. Set an empty color set to store the sub-communication groups that have been colored, and initialize color k to 0;

[0086] S503. Loop through all vertices until all vertices are colored. First, choose a color k = k++;

[0087] S504. Consider all vertices in turn. If a vertex is already colored, continue to consider the next vertex. If vertex i is colored with color k without conflict, then color it.

[0088] S505. Group the data according to the same color into different clusters. Finally, output the sub-communication groups of all clusters and their corresponding communication parameter areas.

[0089] The algorithm described above ensures that no combination of training nodes within the same cluster has a common training node. This allows all combinations of training nodes in a cluster to be reduced simultaneously, greatly accelerating communication efficiency.

Claims

1. A parallel training method for heterogeneous sensing data for consumer-grade GPU clusters, characterized in that, Includes the following steps: S1: Construct a cost model to characterize the cost of a single round of training; this cost to be optimized represents the time required for training with the same parameters in a single round. : ； in, Represents the training data, and Indicates the size of the training data. , and These represent the communication time, computation time, and memory usage in a single iteration, respectively, and are the three key factors affecting the training cost of model T. For micro-batch size; Size of the sub-model; For training nodes Computational power; Total number of GPUs; S2: Analyze and estimate the three key factors of the constructed consumption model and use a search algorithm to search for the optimal distributed training configuration; the optimal distributed training configuration includes the optimal micro-batch size and sub-model size; S3: The uniform structured pruning algorithm is used to generate the optimal structured mask based on the optimal sub-model size obtained in S2; S4: Heterogeneous working nodes generate a heterogeneous sensing sub-model based on the optimal structured mask generated in S3, construct a corresponding optimizer based on the sub-model, construct a data extractor using the optimal micro-batch size obtained in S2, and train the data extractor, sub-model and corresponding optimizer. S5: Gradient aggregation is performed using a cluster-based communication protocol mechanism.

2. The method for parallel training of heterogeneous sensing data for consumer-grade GPU clusters according to claim 1, characterized in that, The key factors of the constructed consumption model are analyzed and estimated, including: estimate : ； in, , and Subgroups The total communication scale, the slowest link bandwidth, and the number of training nodes with the slowest link. For the first A set of sub-communication groups for each cluster class. Indicates the total number of communication clusters; estimate : ； Estimation: Estimation is performed using a linear model.

3. The method for parallel training of heterogeneous sensing data for consumer-grade GPU clusters according to claim 1, characterized in that, The search algorithm process is as follows: S201. Determine the complete model as follows and has Each layer, heterogeneous cluster has Each GPU has a video memory capacity of [number] GPUs. and computing power Let the current optimal time be . ; S202. From Start by iterating through the global batch size and selecting the optimal training configuration by continuously comparing the model time consumption of different configurations; S203. Solve for the optimal training configuration under the current global batch size by maximizing memory usage and balancing heterogeneous computing load. ; S204. Calculate the current training time and compare it with the optimal time, and finally iterate to select the optimal value; S205. After the node, the training configuration obtained from the search is sent to all heterogeneous training nodes for sub-model pruning to obtain the optimal sub-model size.

4. The method for parallel training of heterogeneous sensing data for consumer-grade GPU clusters according to claim 1, characterized in that, The specific steps for S3 are as follows: S301. Determine the complete model as and has Each layer has a 2D matrix (cnt) used to count the number of times each parameter region in each layer of the model is used, and a 2D matrix (Score) used to count the importance score of each parameter region in each layer of the model. For each node... Let its mask be . and , representing the columns and rows that need to be retained, respectively; S302. Traversal Each layer is allocated according to its layer; S303. Determine whether the currently divided layer is a linear layer or a multi-head attention layer. If it is a linear layer, proceed to the next step; otherwise, proceed to S305. S304. Record the current parameter range size. The output dimension of the linear layer, i.e. ; S305. Record the current parameter range size. The number of attention heads, i.e. ; S306. Statistically analyze the entire parameter range of layer l. ,for Subparameter region within Its score is ; S307. Scores based on statistics In ascending order Sort; S308. Traverse each node and record the current allocation index as... ; S309. For parameter range ,remember And the row of the next level is equal to the column of the current level, that is ; S310 Update = ;Finish.

5. The method for parallel training of heterogeneous sensing data for consumer-grade GPU clusters according to claim 1, characterized in that... Step S4: For multi-head attention layers, the obtained mask determines which head needs to be pruned. Specifically, for linear layers, the obtained mask determines which output layers need to be pruned, and the input layers also need to be pruned accordingly to generate a complete sub-model.

6. The method for parallel training of heterogeneous sensing data for consumer-grade GPU clusters according to claim 1, characterized in that... Step S5: Design a gradient reduction mechanism based on graph coloring. The master training node first counts the parameter regions of all training nodes and allows training nodes with the same parameter region to directly perform gradient reduction. Different combinations of training nodes do not affect each other. A graph coloring algorithm is used to adjust the reduction order, setting training nodes with the same parameter region as a single point. Points with common training nodes have edges. In this way, a graph is formed and the graph coloring algorithm is used to ensure that combinations of training nodes in the same cluster do not have common training nodes. Thus, all combinations of training nodes in a cluster can be reduced simultaneously.

7. The method for parallel training of heterogeneous sensing data for consumer-grade GPU clusters according to claim 1, characterized in that... The S5 cluster-based communication reduction mechanism is specifically a cluster reduction mechanism based on a greedy graph coloring algorithm, which includes the following steps: S501. Construct a graph, traverse all sub-communication groups, set sub-communication groups with the same parameter region as a point, and add an edge between sub-communication groups with a common training node. S502. Set an empty color set to store the sub-communication groups that have been colored, and initialize color k to 0; S503. Loop until all vertices are colored, first take a color k=k++; S504. Consider all vertices in turn; if a vertex has already been colored, continue to consider the next vertex; if vertex i is colored with color k without conflict, then color it. S505. Group the data into different clusters according to the same color, and finally output the sub-communication groups of all clusters and their corresponding communication parameter areas.

Images