Model training method, device and storage medium based on heterogeneous GPU cluster

Abstract

The present disclosure relates to a model training method, device and storage medium based on a heterogeneous GPU cluster. The method includes: obtaining hardware indicator data of each GPU in the heterogeneous GPU cluster; obtaining multiple operation types corresponding to the deep learning algorithm in the model to be trained, and measuring the operation performance data of each GPU executing the operation corresponding to each operation type; obtaining the multidimensional characteristics of the communication bandwidth and the load state perception strategy of each GPU; constructing a multidimensional operation performance matrix for each GPU based on the hardware indicator data, the operation performance data, the multidimensional characteristics of the communication bandwidth and the load state perception strategy corresponding to each GPU; and allocating a GPU to each structural layer in the model to be trained for model training to obtain a target model based on the multidimensional operation performance matrix corresponding to each GPU. The present disclosure improves the training efficiency of the model and shortens the training time of the model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a model training method, device, and storage medium based on a heterogeneous GPU cluster. Background Art

[0002] Training large language models requires a hybrid parallel strategy on GPU clusters, combining data parallelism, model parallelism, and pipeline parallelism to leverage thousands or even tens of thousands of GPUs for training. The "slow node" problem has become a major bottleneck in large-scale distributed training; a slowdown in even a single GPU compute node can slow down the entire job. Even a small number of slow nodes in a single job can force tens of thousands of GPUs to wait for training. Furthermore, slow nodes aren't always caused by hardware failures; they can often be the result of a combination of complex factors.

[0003] In addition, due to the high cost of high-end GPUs, it is difficult to obtain a large number of GPUs of the same model at one time; and users often expand computing power resources by accumulating GPUs of different models year by year; therefore, heterogeneous GPU resource environments are becoming increasingly common; however, different GPU nodes have different computing power and video memory capacity. If tasks are still evenly distributed according to traditional assumptions, the slowest GPU will become the new bottleneck; GPUs with fast data processing speeds will be idle while waiting for the slowest GPU, resulting in reduced overall efficiency and thus prolonged model training time. Summary of the Invention

[0004] The present disclosure provides a model training method, device, and storage medium based on a heterogeneous GPU cluster to at least address the problem of long model training time in related technologies. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a model training method based on a heterogeneous GPU cluster is provided, comprising:

[0006] Obtaining hardware indicator data of each GPU in a heterogeneous GPU cluster; the heterogeneous GPU cluster includes at least two heterogeneous GPUs;

[0007] Obtain multiple operation types corresponding to the deep learning algorithm in the model to be trained, and measure the operation performance data of each GPU executing the operation corresponding to each operation type;

[0008] Obtaining multi-dimensional characteristics of the communication bandwidth of each GPU and a load state perception strategy; the load state perception strategy characterizes the correlation between the load and performance of each GPU;

[0009] Constructing a multidimensional operation performance matrix for each GPU based on the hardware indicator data, the operation performance data, the multidimensional characteristics of the communication bandwidth, and the load state perception strategy corresponding to each GPU;

[0010] According to the multi-dimensional operation performance matrix corresponding to each GPU, a GPU is allocated to each structural layer in the model to be trained to perform model training to obtain the target model.

[0011] In an exemplary embodiment, obtaining the multi-dimensional characteristics of the communication bandwidth of each GPU and the load state perception strategy includes:

[0012] Determine the bandwidth and latency characteristics of each GPU at multiple message size levels;

[0013] Determine the performance data of each GPU executing multi-process collective operations;

[0014] Determining the effective bandwidth of each GPU using a preset network technology under multiple load modes; the preset network technology includes at least one of a wireless bandwidth technology and an Ethernet technology;

[0015] Determine the multi-dimensional characteristics of the communication bandwidth of each GPU based on the bandwidth, latency characteristics, performance data, and effective bandwidth corresponding to each GPU;

[0016] Determine the load state awareness strategy for each GPU based on the performance of each GPU under different loads.

[0017] In an exemplary embodiment, determining the load state awareness strategy of each GPU based on the performance of each GPU under different loads includes:

[0018] Analyze the performance decay curves of each GPU executing multiple operation types at different utilization rates to obtain the load-performance correlation relationship;

[0019] Determine the correlation between modeling memory usage and computing performance of each GPU;

[0020] Determine a performance stability analysis result of each GPU under a target load within a target period; the cumulative duration of the target period is greater than or equal to a preset duration, and the target load is greater than or equal to a preset load threshold;

[0021] A load state perception strategy for each GPU is determined based on the load-performance association relationship, the usage-performance association relationship, and the performance stability analysis result corresponding to each GPU.

[0022] In an exemplary embodiment, obtaining multiple operation types corresponding to the deep learning algorithm in the model to be trained and measuring the operation performance data of each GPU executing the operation corresponding to each operation type includes:

[0023] Obtaining an attention mechanism operation type, a multi-layer perceptron operation type, and a preset operation type corresponding to the deep learning algorithm in the model to be trained; the preset operation type includes at least one of a normalization operation type, an embedding operation type, and an activation operation type;

[0024] Measuring first operation performance data of each GPU executing an operation corresponding to the attention mechanism operation type;

[0025] Measuring second operation performance data of each GPU executing an operation corresponding to the multilayer perceptron operation type;

[0026] Measuring third operation performance data of each GPU executing an operation corresponding to the preset operation type;

[0027] The operation performance data corresponding to each GPU is determined according to the first operation performance data, the second operation performance data, and the third operation performance data corresponding to each GPU.

[0028] In an exemplary embodiment, allocating a GPU to each structural layer in the to-be-trained model for model training to obtain a target model according to the multi-dimensional operation performance matrix corresponding to each GPU includes:

[0029] Constructing a parallel pipeline group according to multiple structural layers in the model to be trained;

[0030] Determining a target position of each GPU in the parallel pipeline group according to a multi-dimensional operation performance matrix corresponding to each GPU;

[0031] According to the target position of each GPU in the heterogeneous GPU cluster in the parallel pipeline group, a GPU is allocated to each structural layer in the model to be trained to perform model training to obtain the target model.

[0032] In an exemplary embodiment, the constructing of a parallel pipeline group according to multiple structural layers in the model to be trained includes:

[0033] Sequentially number the multiple structural layers in the model to be trained to construct a model layer set;

[0034] Constructing a plurality of pipeline stage sets in sequence according to the number of each structural layer in the model layer set; each pipeline stage set includes at least two consecutively numbered structural layers;

[0035] The multiple pipeline stage sets are grouped to obtain a target number of the pipeline parallel groups.

[0036] In an exemplary embodiment, determining the target position of each GPU in the parallel pipeline group according to the multi-dimensional operation performance matrix corresponding to each GPU includes:

[0037] Traversing the target number of parallel pipeline groups in sequence, and determining, for a current parallel pipeline group traversed, an activation probability of the current parallel pipeline group;

[0038] Building a data parallel configuration strategy according to the target number of the pipeline parallel groups and the activation probability of the current parallel pipeline group;

[0039] Determining any structural layer in the to-be-trained model as the current structural layer, and determining any pipeline stage in the current parallel pipeline group as the current pipeline stage;

[0040] Constructing a layer stage mapping strategy according to the pipeline stage constraints of the current structural layer, the allocation probability of the current structural layer being divided into the current pipeline stage, and the mapping probability of the current pipeline stage being mapped to each GPU in the heterogeneous GPU cluster;

[0041] A target position of each GPU in the current parallel pipeline group is determined according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, and the layer stage mapping strategy.

[0042] In an exemplary embodiment, determining a target position of each GPU in the current parallel pipeline group according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, and the layer stage mapping strategy includes:

[0043] Get the batch size and sequence length of sample data;

[0044] Constructing a batch configuration strategy based on the batch size and sequence length of the sample data;

[0045] Constructing a training time prediction function according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer-stage mapping strategy, and the batch configuration strategy;

[0046] Determine a target position of each GPU in the current parallel pipeline group according to the training time prediction function.

[0047] In an exemplary embodiment, constructing a training time prediction function according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer-stage mapping strategy, and the batch configuration strategy includes:

[0048] Determining a stage combination delay function for each pipeline stage in each pipeline parallel group according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer stage mapping strategy, and the batch configuration strategy;

[0049] Determining a steady-state critical pipeline stage in each pipeline parallel group according to a stage combination delay function of each pipeline stage in each pipeline parallel group, and determining the stage combination delay of the steady-state critical pipeline stage as a critical path time function;

[0050] Get the gradient synchronization time function corresponding to each pipeline parallel group;

[0051] Determine the complete iteration time function of the model to be trained according to the critical path time function and the gradient synchronization time function corresponding to each pipeline parallel group;

[0052] The training time prediction function is constructed according to the complete iteration time function and the scheduling management time function.

[0053] In an exemplary embodiment, determining the stage combination delay function of each pipeline stage in each pipeline parallel group according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer stage mapping strategy, and the batch configuration strategy includes:

[0054] Determining the normalized performance of each GPU for each operation type based on the multidimensional operation performance matrix, the data parallel configuration strategy, the layer stage mapping strategy, and the batch configuration strategy;

[0055] Determine, for each pipeline parallel group, a comprehensive load of the current pipeline stage; the comprehensive load includes a forward comprehensive load of each operation type during the forward propagation process and a reverse comprehensive load during the reverse propagation process;

[0056] determining a forward delay function of the current pipeline stage according to a ratio between the forward integrated load of each operation type in the current pipeline stage and the normalized performance corresponding to each operation type;

[0057] determining a reverse delay function for the current pipeline stage according to a ratio between the reverse integrated load of each operation type in the current pipeline stage and the normalized performance of each operation type;

[0058] Determine a stage combination delay function of the current pipeline stage according to the forward delay function and the reverse delay function corresponding to the current pipeline stage.

[0059] In an exemplary embodiment, determining the comprehensive load of the current pipeline stage includes:

[0060] Obtaining a first load corresponding to the attention mechanism operation type in the current pipeline stage;

[0061] Obtaining a second load corresponding to the multilayer perceptron operation type in the current pipeline stage;

[0062] Obtaining a third load corresponding to a preset operation type in the current pipeline stage; the preset operation type includes at least one of a normalization operation type, an embedding operation type, and an activation operation type;

[0063] The comprehensive load of the current pipeline stage is determined according to the first load, the second load, and the third load corresponding to the current pipeline stage.

[0064] In an exemplary embodiment, determining the stage combined delay function of the current pipeline stage according to the forward delay function and the reverse delay function corresponding to the current pipeline stage includes:

[0065] Determining a non-overlapping communication residual time function between the current pipeline stage and an adjacent pipeline stage based on the data volume of each GPU in the current pipeline stage and the ratio of the bandwidth between each GPU and the communication GPU;

[0066] Determine the stage combined delay function of the current pipeline stage according to the forward delay function, the reverse delay function, and the communication residual time function corresponding to the current pipeline stage.

[0067] In an exemplary embodiment, determining a target position of each GPU in the current parallel pipeline group according to the training time prediction function includes:

[0068] Obtain GPU mapping uniqueness constraints, video memory capacity constraints, computing performance constraints, communication bandwidth constraints, and critical path constraints to obtain strategy constraints;

[0069] Obtaining, based on the strategy constraints, the training time prediction function, and the hybrid parallel simulator, the training time corresponding to different positions of each GPU in the current parallel pipeline group;

[0070] The position where the training time corresponding to each GPU meets a preset condition is determined as the target position of each GPU in the current parallel pipeline group.

[0071] In an exemplary embodiment, the method for constructing the video memory capacity constraint condition includes:

[0072] For the current pipeline stage in each pipeline parallel group, obtain the model parameter memory of the current structure layer and the activation data memory generated during the forward propagation process;

[0073] Determining a target required parameter memory of the current pipeline stage according to an allocation probability of each current structural layer in the to-be-trained model being divided into the current pipeline stage and a model parameter memory of each current structural layer;

[0074] Determining a target demand activation memory function of the current pipeline stage according to an allocation probability of each current structural layer in the to-be-trained model being divided into the current pipeline stage and an activation data memory corresponding to each current structural layer;

[0075] Activate the memory function according to the target demand and construct the video memory capacity constraint condition.

[0076] In an exemplary embodiment, determining a target position of each GPU in the current parallel pipeline group according to the training time prediction function includes:

[0077] Obtaining, according to the stage combination delay function of the current pipeline stage, stage combination delay data corresponding to different positions of each GPU in the current parallel pipeline group;

[0078] Calculating the pipeline parallel voiding rate and the data parallel voiding rate corresponding to different positions of each GPU in the current parallel pipeline group;

[0079] Determine the training time of each GPU at different positions in the current parallel pipeline group based on the stage combination delay data corresponding to each GPU at different positions in the current parallel pipeline group, the pipeline parallel voiding rate, the data parallel voiding rate, and the training time prediction function;

[0080] Determine a target position of each GPU in the current parallel pipeline group according to the training time of each GPU at different positions in the current parallel pipeline group.

[0081] In an exemplary embodiment, allocating a GPU to each structural layer in the to-be-trained model for model training to obtain a target model according to the multi-dimensional operation performance matrix corresponding to each GPU includes:

[0082] Obtaining the weight coefficient corresponding to each operation type in the model to be trained;

[0083] Obtaining the normalized performance of each GPU for each operation type in the model to be trained, the impact factor of the load of each GPU on the performance of the model to be trained, and the memory adaptability factor of each GPU and the model to be trained;

[0084] Determine a comprehensive fitness score for each GPU and the model to be trained based on the weight coefficient, the normalized performance corresponding to each GPU, the impact factor, and the memory fitness factor;

[0085] Determining multiple screening GPUs based on the comprehensive fitness score of each GPU and the model to be trained;

[0086] According to the multi-dimensional operation performance matrix corresponding to each screened GPU, a GPU is allocated to each structural layer in the model to be trained to perform model training to obtain a target model.

[0087] In an exemplary embodiment, allocating a GPU to each structural layer in the to-be-trained model for model training to obtain a target model according to the multi-dimensional operation performance matrix corresponding to each GPU includes:

[0088] Allocate a GPU to each structural layer in the to-be-trained model according to the multi-dimensional operation performance matrix corresponding to each GPU to obtain a GPU scheduling strategy;

[0089] Determine the sample computation cost of each sample data according to the sequence length, floating-point operation amount, or historical execution time corresponding to similar sample data of each sample data in the sample data set;

[0090] Determine the performance data of each GPU based on the current load status of each GPU;

[0091] According to the sample calculation cost of each sample data and the performance data of each GPU, the GPU corresponding to each sample data is determined to obtain the sample allocation strategy;

[0092] According to the GPU scheduling strategy and the sample allocation strategy, the model to be trained is trained to obtain a target model.

[0093] In an exemplary embodiment, determining the GPU corresponding to each sample data according to the sample calculation cost of each sample data and the performance data of each GPU to obtain the sample allocation strategy includes:

[0094] Get the sample allocation probability of each sample data being assigned to each GPU;

[0095] Constructing a sample allocation constraint condition based on the sample allocation probability of each GPU;

[0096] Constructing a load balancing constraint condition according to the sample allocation probability corresponding to each GPU and the performance data;

[0097] Determining a comprehensive constraint condition according to the sample allocation constraint condition and the load balancing constraint condition;

[0098] Construct a target time function based on the sample allocation probability and performance data corresponding to each GPU;

[0099] According to the comprehensive constraint conditions and the target time function, the GPU corresponding to each sample data is determined to obtain a sample allocation strategy.

[0100] In an exemplary embodiment, determining the comprehensive constraint condition according to the sample distribution constraint condition and the load balancing constraint condition includes:

[0101] Determine sample memory requirements based on the model dimensions and architecture coefficients of the model to be trained;

[0102] Constructing a target video memory capacity constraint condition based on the sample allocation probability, video memory capacity, and sample memory requirement corresponding to each GPU;

[0103] Constructing a batch quantity constraint condition for the sample data set;

[0104] The comprehensive constraint condition is determined according to the sample allocation constraint condition, the load balancing constraint condition, the target video memory capacity constraint condition and the batch quantity constraint condition.

[0105] In an exemplary embodiment, after determining the GPU corresponding to each sample data based on the sample calculation cost of each sample data and the performance data of each GPU and obtaining the sample allocation strategy, the method further includes:

[0106] Based on the sample allocation strategy, the current sample data is input into the model to be trained to obtain the real-time data processing progress of the current GPU corresponding to the current sample data;

[0107] According to the real-time data processing progress of the current GPU, the sample allocation strategy of the sample data located after the current sample data is dynamically adjusted.

[0108] According to a second aspect of an embodiment of the present disclosure, a model training device based on a heterogeneous GPU cluster is provided, comprising:

[0109] A hardware data acquisition module is configured to acquire hardware indicator data of each GPU in a heterogeneous GPU cluster; the heterogeneous GPU cluster includes at least two heterogeneous GPUs;

[0110] An operation performance data acquisition module is configured to acquire multiple operation types corresponding to the deep learning algorithm in the model to be trained and measure the operation performance data of each GPU executing the operation corresponding to each operation type;

[0111] A load strategy acquisition module is configured to execute acquisition of multi-dimensional characteristics of the communication bandwidth of each GPU and a load state perception strategy; the load state perception strategy characterizes the correlation between the load and performance of each GPU;

[0112] a performance matrix construction module configured to construct a multidimensional operation performance matrix for each GPU based on the hardware indicator data, the operation performance data, the multidimensional characteristics of the communication bandwidth, and the load state perception strategy corresponding to each GPU;

[0113] The model training module is configured to execute the multi-dimensional operation performance matrix corresponding to each GPU, allocate a GPU to each structural layer in the model to be trained to perform model training to obtain a target model.

[0114] In an exemplary embodiment, the load strategy acquisition module includes:

[0115] a delay characteristic determination unit configured to determine bandwidth and delay characteristics of each GPU under messages of multiple data volume levels;

[0116] A performance data determining unit is configured to determine performance data of each GPU executing a multi-process aggregate operation;

[0117] An effective bandwidth determination unit is configured to determine the effective bandwidth of each GPU under multiple load modes using a preset network technology; the preset network technology includes at least one of a wireless bandwidth technology and an Ethernet technology;

[0118] a communication bandwidth multi-dimensional feature determination unit configured to determine the communication bandwidth multi-dimensional feature of each GPU based on the bandwidth, latency feature, performance data, and effective bandwidth corresponding to each GPU;

[0119] The strategy determination unit is configured to determine a load state awareness strategy for each GPU based on the performance of each GPU under different loads.

[0120] In an exemplary embodiment, the policy determination unit includes:

[0121] The first relationship determination subunit is configured to analyze the performance decay curves corresponding to each GPU executing multiple operation types at different utilization rates to obtain a load-performance correlation relationship;

[0122] The second relationship determination subunit is configured to determine a usage-performance correlation relationship between the modeling memory usage and the computing performance of each GPU;

[0123] a stability analysis result determination subunit configured to determine a performance stability analysis result of each GPU under a target load within a target period; the cumulative duration of the target period is greater than or equal to a preset duration, and the target load is greater than or equal to a preset load threshold;

[0124] The strategy determination subunit is configured to determine the load state perception strategy of each GPU according to the load-performance association relationship, the usage-performance association relationship and the performance stability analysis result corresponding to each GPU.

[0125] In an exemplary embodiment, the operation performance data acquisition module includes:

[0126] An operation type acquisition unit is configured to execute and acquire an attention mechanism operation type, a multi-layer perceptron operation type, and a preset operation type corresponding to the deep learning algorithm in the to-be-trained model; the preset operation type includes at least one of a normalization operation type, an embedding operation type, and an activation operation type;

[0127] A first data measuring unit is configured to measure first operation performance data of each GPU executing an operation corresponding to the attention mechanism operation type;

[0128] A second data measuring unit is configured to measure second operation performance data of each GPU executing an operation corresponding to the multilayer perceptron operation type;

[0129] A third data measuring unit is configured to measure third operation performance data of each GPU executing an operation corresponding to the preset operation type;

[0130] The operation data determining unit is configured to determine the operation performance data corresponding to each GPU according to the first operation performance data, the second operation performance data, and the third operation performance data corresponding to each GPU.

[0131] In an exemplary embodiment, the model training module includes:

[0132] A pipeline group construction unit is configured to construct a parallel pipeline group according to multiple structural layers in the model to be trained;

[0133] a target position determination unit, configured to determine a target position of each GPU in the parallel pipeline group according to a multi-dimensional operation performance matrix corresponding to each GPU;

[0134] The model training unit is configured to perform model training by allocating a GPU to each structural layer in the model to be trained according to the target position of each GPU in the heterogeneous GPU cluster in the parallel pipeline group to obtain the target model.

[0135] In an exemplary embodiment, the pipeline group construction unit is further configured to sequentially number the multiple structural layers in the model to be trained to construct a model layer set; construct multiple pipeline stage sets in sequence according to the number of each structural layer in the model layer set; each pipeline stage set includes at least two consecutively numbered structural layers; and group the multiple pipeline stage sets to obtain a target number of pipeline parallel groups.

[0136] In an exemplary embodiment, the model training module includes:

[0137] An activation probability determination unit is configured to sequentially traverse the target number of parallel pipeline groups and determine an activation probability of a current parallel pipeline group that has been traversed;

[0138] a parallel configuration strategy building unit, configured to build a data parallel configuration strategy according to the target number of pipeline parallel groups and the activation probability of the current parallel pipeline group;

[0139] a current structural layer determining unit configured to determine any structural layer in the to-be-trained model as the current structural layer, and to determine any pipeline stage in the current parallel pipeline group as the current pipeline stage;

[0140] an allocation probability determining unit configured to construct a layer stage mapping strategy according to the pipeline stage constraint of the current structural layer, the allocation probability of the current structural layer being divided into the current pipeline stage, and the mapping probability of the current pipeline stage being mapped to each GPU in the heterogeneous GPU cluster;

[0141] The position determination unit is configured to determine the target position of each GPU in the current parallel pipeline group according to the multi-dimensional operation performance matrix, the data parallel configuration strategy and the layer stage mapping strategy.

[0142] In an exemplary embodiment, the position determination unit includes:

[0143] A sequence length acquisition subunit is configured to execute the acquisition of the batch number and sequence length of the sample data;

[0144] A batch configuration strategy construction subunit is configured to execute the construction of a batch configuration strategy according to the batch quantity and sequence length of the sample data;

[0145] a time prediction function construction subunit, configured to construct a training time prediction function according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer-stage mapping strategy, and the batch configuration strategy;

[0146] The position determination subunit is configured to execute the training time prediction function to determine the target position of each GPU in the current parallel pipeline group.

[0147] In an exemplary embodiment, the time prediction function construction subunit includes:

[0148] a stage combination delay function determination subunit, configured to determine the stage combination delay function of each pipeline stage in each pipeline parallel group according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer-stage mapping strategy, and the batch configuration strategy;

[0149] a critical path time function determination subunit configured to determine a steady-state critical pipeline stage in each pipeline parallel group according to a stage combination delay function of each pipeline stage in each pipeline parallel group, and determine the stage combination delay of the steady-state critical pipeline stage as the critical path time function;

[0150] A synchronization time function acquisition subunit is configured to execute and acquire the gradient synchronization time function corresponding to each pipeline parallel group;

[0151] a complete iteration time function determination subunit, configured to determine the complete iteration time function of the to-be-trained model according to the critical path time function and the gradient synchronization time function corresponding to each pipeline parallel group;

[0152] The training time prediction function construction subunit is configured to execute the construction of the training time prediction function according to the complete iteration time function and the scheduling management time function.

[0153] In an exemplary embodiment, the stage combination delay function determination subunit includes:

[0154] a normalized performance determination subunit, configured to determine the normalized performance of each GPU for each operation type based on the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer stage mapping strategy, and the batch configuration strategy;

[0155] a comprehensive load determination subunit configured to determine the comprehensive load of the current pipeline stage in each pipeline parallel group; the comprehensive load includes the forward comprehensive load of each operation type during the forward propagation process and the reverse comprehensive load during the reverse propagation process;

[0156] a forward delay function determination subunit, configured to determine the forward delay function of the current pipeline stage according to a ratio between the forward comprehensive load of each operation type in the current pipeline stage and the normalized performance corresponding to each operation type;

[0157] a reverse delay function determining subunit, configured to determine a reverse delay function of the current pipeline stage according to a ratio between the reverse comprehensive load of each operation type in the current pipeline stage and the normalized performance of each operation type;

[0158] The combined delay function determining subunit is configured to determine the stage combined delay function of the current pipeline stage according to the forward delay function and the reverse delay function corresponding to the current pipeline stage.

[0159] In an exemplary embodiment, the comprehensive load determination subunit is further configured to execute the steps of obtaining a first load corresponding to the attention mechanism operation type in the current pipeline stage; obtaining a second load corresponding to the multi-layer perceptron operation type in the current pipeline stage; obtaining a third load corresponding to a preset operation type in the current pipeline stage; the preset operation type includes at least one of a normalization operation type, an embedding operation type, and an activation operation type; and determining the comprehensive load of the current pipeline stage based on the first load, the second load, and the third load corresponding to the current pipeline stage.

[0160] In an exemplary embodiment, the combined delay function determination subunit is further configured to determine the non-overlapping communication residual time function between the current pipeline stage and the adjacent pipeline stage based on the data volume of each GPU in the current pipeline stage and the ratio of the bandwidth between each GPU and the communication GPU; and determine the stage combined delay function of the current pipeline stage based on the forward delay function, reverse delay function and the communication residual time function corresponding to the current pipeline stage.

[0161] In an exemplary embodiment, the position determination subunit is further configured to execute acquisition of GPU mapping uniqueness constraints, video memory capacity constraints, computing performance constraints, communication bandwidth constraints, and critical path constraints to obtain strategy constraints; based on the strategy constraints, the training time prediction function, and the hybrid parallel simulator, obtain the training time corresponding to different positions of each GPU in the current parallel pipeline group; and determine the position where the training time corresponding to each GPU meets the preset conditions as the target position of each GPU in the current parallel pipeline group.

[0162] In an exemplary embodiment, the apparatus further comprises:

[0163] an activation data memory acquisition module, configured to execute, for a current pipeline stage in each pipeline parallel group, acquiring a model parameter memory of the current structural layer and an activation data memory generated during a forward propagation process;

[0164] a target requirement parameter memory acquisition module configured to determine the target requirement parameter memory of the current pipeline stage according to the allocation probability of each current structural layer in the to-be-trained model being divided into the current pipeline stage and the model parameter memory of each current structural layer;

[0165] a target demand activation memory function determination module, configured to determine the target demand activation memory function of the current pipeline stage according to the allocation probability of each current structural layer in the to-be-trained model being divided into the current pipeline stage and the activation data memory corresponding to each current structural layer;

[0166] The video memory capacity constraint condition construction module is configured to execute the memory activation function according to the target demand and construct the video memory capacity constraint condition.

[0167] In an exemplary embodiment, the position determination subunit is further configured to execute a stage combination delay function according to the current pipeline stage to obtain stage combination delay data corresponding to different positions of each GPU in the current parallel pipeline group; calculate the pipeline parallel voiding rate and data parallel voiding rate corresponding to different positions of each GPU in the current parallel pipeline group; determine the training time of each GPU at different positions in the current parallel pipeline group according to the stage combination delay data, pipeline parallel voiding rate, data parallel voiding rate and the training time prediction function corresponding to different positions of each GPU in the current parallel pipeline group; and determine the target position of each GPU in the current parallel pipeline group according to the training time of each GPU at different positions in the current parallel pipeline group.

[0168] In an exemplary embodiment, the model training module includes:

[0169] A weight coefficient acquisition unit is configured to acquire a weight coefficient corresponding to each operation type in the to-be-trained model;

[0170] an impact factor acquisition unit, configured to acquire the normalized performance of each GPU for each operation type in the to-be-trained model, an impact factor of the load of each GPU on the performance of the to-be-trained model, and a memory compatibility factor between each GPU and the to-be-trained model;

[0171] a fitness score determining unit configured to determine a comprehensive fitness score between each GPU and the model to be trained based on the weight coefficient, the normalized performance corresponding to each GPU, the impact factor, and the memory fitness factor;

[0172] A screening unit is configured to determine a plurality of screening GPUs based on a comprehensive fitness score between each GPU and the model to be trained;

[0173] The model determination unit is configured to execute the multi-dimensional operation performance matrix corresponding to each screened GPU, and allocate a GPU to each structural layer in the model to be trained to perform model training to obtain a target model.

[0174] In an exemplary embodiment, the model training module includes:

[0175] a scheduling strategy determining unit configured to allocate a GPU to each structural layer in the to-be-trained model according to a multi-dimensional operation performance matrix corresponding to each GPU, thereby obtaining a GPU scheduling strategy;

[0176] a sample calculation cost determination unit configured to determine the sample calculation cost of each sample data according to the sequence length, floating-point operation amount, or historical execution time corresponding to similar sample data of each sample data in the sample data set;

[0177] a performance data determining unit, configured to determine performance data of each GPU based on a current load state of each GPU;

[0178] a sample allocation strategy determination unit configured to determine the GPU corresponding to each sample data according to the sample calculation cost of each sample data and the performance data of each GPU, and obtain a sample allocation strategy;

[0179] The model determination unit is configured to execute training on the model to be trained according to the GPU scheduling strategy and the sample allocation strategy to obtain a target model.

[0180] In an exemplary embodiment, the sample allocation strategy determination unit includes:

[0181] The sample allocation probability acquisition subunit is configured to obtain the sample allocation probability of each sample data being allocated to each GPU;

[0182] a sample allocation constraint condition construction subunit, configured to construct a sample allocation constraint condition according to the sample allocation probability of each GPU;

[0183] A load balancing constraint condition construction subunit is configured to construct a load balancing constraint condition according to the sample allocation probability corresponding to each GPU and the performance data;

[0184] a comprehensive constraint condition determination subunit, configured to determine a comprehensive constraint condition according to the sample allocation constraint condition and the load balancing constraint condition;

[0185] The target time function construction subunit is configured to execute the construction of the target time function according to the sample allocation probability and performance data corresponding to each GPU;

[0186] The sample allocation strategy determination subunit is configured to determine the GPU corresponding to each sample data according to the comprehensive constraint condition and the target time function, and obtain the sample allocation strategy.

[0187] In an exemplary embodiment, the comprehensive constraint determination subunit is further configured to determine the sample memory requirement based on the model dimension and architecture coefficient of the model to be trained; construct a target memory capacity constraint based on the sample allocation probability, memory capacity and sample memory requirement corresponding to each GPU; construct a batch quantity constraint for the sample data set; and determine the comprehensive constraint based on the sample allocation constraint, the load balancing constraint, the target memory capacity constraint and the batch quantity constraint.

[0188] In an exemplary embodiment, the apparatus further comprises:

[0189] a data processing progress determination module configured to execute, based on the sample allocation strategy, input current sample data into the to-be-trained model, and obtain the real-time data processing progress of the current GPU corresponding to the current sample data;

[0190] The sample allocation strategy adjustment module is configured to dynamically adjust the sample allocation strategy of the sample data located after the current sample data according to the real-time data processing progress of the current GPU.

[0191] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0192] processor;

[0193] a memory for storing instructions executable by the processor;

[0194] The processor is configured to execute the instructions to implement the model training method based on heterogeneous GPU cluster as described above.

[0195] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by an electronic device processor, the electronic device is enabled to execute the model training method based on a heterogeneous GPU cluster as described above.

[0196] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the model training method based on a heterogeneous GPU cluster as described above.

[0197] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0198] The present invention obtains hardware indicator data of each GPU in a heterogeneous GPU cluster; the heterogeneous GPU cluster includes at least two heterogeneous GPUs; obtains multiple operation types corresponding to the deep learning algorithm in the model to be trained, and measures the operation performance data of each GPU executing the operation corresponding to each operation type; obtains the multidimensional characteristics of the communication bandwidth and the load state perception strategy of each GPU; the load state perception strategy characterizes the correlation between the load and performance of each GPU; constructs a multidimensional operation performance matrix for each GPU based on the hardware indicator data, the operation performance data, the multidimensional characteristics of the communication bandwidth and the load state perception strategy corresponding to each GPU; thereby, the GPU static hardware indicators and dynamic runtime characteristics are integrated to obtain a multidimensional operation performance matrix for each GPU, so as to accurately predict the performance of each GPU; then, based on the multidimensional operation performance matrix corresponding to each GPU, a GPU is allocated to each structural layer in the model to be trained for model training to obtain a target model, thereby improving the adaptability of the GPU performance and the structural layer in the model, thereby improving the training efficiency of the model and shortening the training time of the model.

[0199] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0200] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0201] Figure 1 This is a diagram of an application environment of a model training method based on a heterogeneous GPU cluster according to an exemplary embodiment.

[0202] Figure 2 This is a flowchart of a model training method based on a heterogeneous GPU cluster according to an exemplary embodiment.

[0203] Figure 3 The present invention is a flowchart showing a method for determining a load state awareness strategy for each GPU based on the performance of each GPU under different loads, according to an exemplary embodiment.

[0204] Figure 4 This is a flowchart of a method for obtaining multiple operation types corresponding to a deep learning algorithm in a model to be trained and measuring the operation performance data of each GPU performing the operation corresponding to each operation type according to an exemplary embodiment.

[0205] Figure 5 It is a flowchart of a method for determining the stage combination delay function of each pipeline stage in each pipeline parallel group based on the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer stage mapping strategy and the batch configuration strategy according to an exemplary embodiment.

[0206] Figure 6 The present invention is a flowchart showing a method for determining the comprehensive load of the current pipeline stage according to an exemplary embodiment.

[0207] Figure 7 It is a flowchart of a model training method of a model training system based on a heterogeneous GPU cluster according to an exemplary embodiment.

[0208] Figure 8 This is a block diagram of a model training device based on a heterogeneous GPU cluster according to an exemplary embodiment.

[0209] Figure 9 The figure is a block diagram of a server according to an exemplary embodiment.

[0210] Figure 10 It is a block diagram of an electronic device for model training based on a heterogeneous GPU cluster according to an exemplary embodiment. DETAILED DESCRIPTION

[0211] In order to enable ordinary people in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0212] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0213] See also Figure 1 , which shows an application environment diagram of a model training method based on a heterogeneous GPU cluster according to an exemplary embodiment. The application environment may include a server 01 and a client 02.

[0214] Specifically, in the embodiments of this specification, the server 01 may include a standalone server, a distributed server, or a server cluster consisting of multiple servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 01 may include a network communication unit, a processor, and memory, among other things. Specifically, server 01 may be configured to construct a multidimensional operational performance matrix for each GPU in a heterogeneous GPU cluster based on each GPU's corresponding hardware indicator data, operational performance data, multidimensional communication bandwidth characteristics, and load status awareness strategy. Based on each GPU's corresponding multidimensional operational performance matrix, server 01 may allocate a GPU to each structural layer in the model to be trained for model training to obtain a target model. The server 01 may also transmit the hardware indicator data, operational performance data, multidimensional communication bandwidth characteristics, and load status awareness strategy corresponding to each GPU to client 02.

[0215] Specifically, in the embodiments of this specification, client 02 may include physical devices such as smartphones, desktop computers, tablet computers, laptops, digital assistants, smart wearable devices, and in-vehicle terminals. It may also include software running on physical devices, such as web pages provided by service providers to users, or applications provided by these service providers to users. Specifically, client 02 may be used to display hardware indicator data, operational performance data, multi-dimensional communication bandwidth characteristics, and load status awareness strategies corresponding to each GPU.

[0216] Figure 2FIG is a flow chart of a model training method based on a heterogeneous GPU cluster according to an exemplary embodiment. Figure 2 As shown, this method can be applied to Figure 1 The server 01 shown includes the following steps.

[0217] In step S201, hardware indicator data of each GPU in a heterogeneous GPU cluster is obtained; the heterogeneous GPU cluster includes at least two heterogeneous GPUs.

[0218] In the disclosed embodiments, a graphics processing unit (GPU), also known as a display core, includes at least one of a theoretical peak value, video memory specifications, interconnect bandwidth, and architectural features. The theoretical peak value includes at least one of FLOPs, the number of CUDA cores, and tensor cores. FLOPs (Floating-Point Operations Per Second) refers to the number of floating-point operations per second, and the number of CUDA (Compute Unified Device Architecture) cores refers to the number of processor cores used for parallel computing.

[0219] Video memory specifications include at least one of video memory capacity, video memory bandwidth, and video memory type (GDDR6, HBM, etc.);

[0220] Interconnect bandwidth includes at least one of the following: PCIe generation and lane count, NVLink version and bandwidth, and InfiniBand / Ethernet network specifications. PCIe (Peripheral Component Interconnect Express) is a high-speed serial bus standard used to connect computer motherboards and expansion devices. NVLink is a direct interconnect technology between GPUs. InfiniBand (literally, "infinite bandwidth" technology, abbreviated as IB) is a computer network communication standard for high-performance computing. It offers extremely high throughput and low latency, and is used for computer-to-computer data interconnection. Ethernet is a bus-based network topology that uses a distributed arbitration mechanism to resolve conflicts.

[0221] Architectural features include at least one of the GPU architecture generation (including Volta, Turing, Ampere, etc.) and supported data types (FP16 / BF16 / FP8, etc.); among them, FP16, FP8, and BF16 are three different low-precision floating-point formats.

[0222] In step S203, multiple operation types corresponding to the deep learning algorithm in the model to be trained are obtained, and the operation performance data of each GPU executing the operation corresponding to each operation type is measured.

[0223] In the disclosed embodiment, the model to be trained may include but is not limited to a large language model (LLM); the multiple operation types may include but are not limited to at least two of an attention mechanism (Attention) operation type, a multi-layer perceptron (MLP) operation type, a normalization (layerNorm) operation type, an embedding (Embedding) operation type, and an activation (Activation) operation type; then, through an operation-level benchmark test, the operation performance data of each GPU executing the operation corresponding to each operation type is measured.

[0224] In step S205 , a multi-dimensional feature of the communication bandwidth of each GPU and a load state perception strategy are obtained; the load state perception strategy characterizes the correlation between the load and performance of each GPU.

[0225] In the disclosed embodiment, the multi-dimensional characteristics of communication bandwidth may include: (1) point-to-point communication: inter-GPU bandwidth and latency characteristics under different message sizes; (2) collective communication: the performance of collective operations such as All-Reduce and All-Gather at different scales; All-Reduce is a method for synchronizing data between multiple processes. It first performs some form of "reduce" operation (such as sum, maximum, minimum, etc.) on the data of all processes, and then broadcasts the result back to all processes. The All-Gather operation is used to collect the data of each process into all processes. Each process sends its own data to all other processes, so that each process eventually has a copy of the data of all other processes. (3) Cross-node network: the effective bandwidth of InfiniBand / Ethernet under different load modes. The load state awareness strategy can be determined based on load state awareness modeling. The load state awareness strategy is used to characterize the correlation between load and performance.

[0226] In step S207 , a multi-dimensional operation performance matrix of each GPU is constructed according to the hardware indicator data, the operation performance data, the multi-dimensional characteristics of the communication bandwidth, and the load state perception strategy corresponding to each GPU.

[0227] In the disclosed embodiments, the operational performance data, the multidimensional characteristics of the communication bandwidth, and the load state perception strategy are all dynamic runtime indicators. During the simulation training phase, the operational performance data, the multidimensional characteristics of the communication bandwidth, and the load state perception strategy, among other indicators, can be obtained for each GPU during its dynamic operation. A memory hierarchy performance profile can then be constructed using the hardware indicator data, an operational-level performance matrix can be determined based on the operational performance data, a communication performance characteristic matrix can be determined based on the multidimensional characteristics of the communication bandwidth, and a load-perceived performance model can be determined based on the load state perception strategy. This can then lead to the construction of a multidimensional operational performance matrix for each GPU.

[0228] An exemplary, multi-dimensional operational performance matrix is ​​shown below:

[0229]

[0230] in, is the operation-level performance matrix of the i-th GPU, is the communication performance characteristic matrix of the i-th GPU, is the load-aware performance model of the i-th GPU, Profile the memory hierarchy performance of the i-th GPU.

[0231] In step S209, according to the multi-dimensional operation performance matrix corresponding to each GPU, a GPU is allocated to each structural layer in the model to be trained to perform model training to obtain a target model.

[0232] In the disclosed embodiment, the performance of each GPU can be predicted based on the multi-dimensional operation performance matrix corresponding to each GPU, and then a GPU can be allocated to each structural layer in the model to be trained, so as to train the model to be trained and obtain a target model.

[0233] The present invention obtains hardware indicator data of each GPU in a heterogeneous GPU cluster; the heterogeneous GPU cluster includes at least two heterogeneous GPUs; obtains multiple operation types corresponding to the deep learning algorithm in the model to be trained, and measures the operation performance data of each GPU executing the operation corresponding to each operation type; obtains the multidimensional characteristics of the communication bandwidth and the load state perception strategy of each GPU; the load state perception strategy characterizes the correlation between the load and performance of each GPU; constructs a multidimensional operation performance matrix for each GPU based on the hardware indicator data, the operation performance data, the multidimensional characteristics of the communication bandwidth and the load state perception strategy corresponding to each GPU; thereby, the GPU static hardware indicators and dynamic runtime characteristics are integrated to obtain a multidimensional operation performance matrix for each GPU, so as to accurately predict the performance of each GPU; then, based on the multidimensional operation performance matrix corresponding to each GPU, a GPU is allocated to each structural layer in the model to be trained for model training to obtain a target model, thereby improving the adaptability of the GPU performance and the structural layer in the model, thereby improving the training efficiency of the model and shortening the training time of the model.

[0234] In some embodiments, obtaining the multi-dimensional characteristics of the communication bandwidth of each GPU and the load state perception strategy includes:

[0235] Determine the bandwidth and latency characteristics of each GPU at multiple message size levels;

[0236] Determine the performance data of each GPU executing multi-process collective operations;

[0237] Determining the effective bandwidth of each GPU using a preset network technology under multiple load modes; the preset network technology includes at least one of a wireless bandwidth technology and an Ethernet technology;

[0238] Determine the multi-dimensional characteristics of the communication bandwidth of each GPU based on the bandwidth, latency characteristics, performance data, and effective bandwidth corresponding to each GPU;

[0239] Determine the load state awareness strategy for each GPU based on the performance of each GPU under different loads.

[0240] In the disclosed embodiment, effective bandwidth refers to the width of the transmission band at 1 / 2 of the maximum transmittance value; the multi-dimensional characteristics of communication bandwidth may include the bandwidth of each GPU under messages of multiple data volume levels, latency characteristics (inter-GPU bandwidth and latency characteristics under different message sizes), performance data (performance of collective operations such as All-Reduce and All-Gather at different scales), and effective bandwidth (effective bandwidth of InfiniBand / Ethernet under different load modes); thereby enriching the multi-dimensional characteristics of communication bandwidth; and then determining the load status perception strategy of each GPU based on the performance of each GPU under different loads.

[0241] In some embodiments, as Figure 3 As shown, the load state perception strategy of each GPU is determined based on the performance of each GPU under different loads, including:

[0242] S301: Analyze the performance decay curves corresponding to each GPU executing multiple operation types at different utilization rates to obtain a load-performance correlation relationship;

[0243] S303: Determine a correlation between the usage rate of the modeling memory and the computing performance of each GPU;

[0244] S305: Determine a performance stability analysis result of each GPU under a target load within a target period; the cumulative duration of the target period is greater than or equal to a preset duration, and the target load is greater than or equal to a preset load threshold;

[0245] S307: Determine a load status perception strategy for each GPU based on the load-performance correlation, the usage-performance correlation, and the performance stability analysis result corresponding to each GPU.

[0246] In the disclosed embodiments, load state awareness modeling can be performed, and then the performance decay curves corresponding to each GPU executing multiple operation types at different utilization rates can be analyzed to obtain a load-performance correlation relationship. The utilization-performance correlation relationship between the modeled video memory utilization rate and computing performance of each GPU can be determined to obtain the impact of memory on performance. The performance stability analysis results of each GPU under a target load within a target time period can also be determined, thereby performing a performance stability analysis under long-term high load. The preset duration and preset load threshold can both be set according to actual conditions. This embodiment can obtain multiple correlations between the load and performance of each GPU during operation, thereby improving the accuracy of the load state awareness strategy for each GPU.

[0247] In some embodiments, as Figure 4As shown, the method of obtaining multiple operation types corresponding to the deep learning algorithm in the model to be trained and measuring the operation performance data of each GPU executing the operation corresponding to each operation type includes:

[0248] S401: Obtaining an attention mechanism operation type, a multi-layer perceptron operation type, and a preset operation type corresponding to the deep learning algorithm in the model to be trained; the preset operation type includes at least one of a normalization operation type, an embedding operation type, and an activation operation type;

[0249] S403: Measure first operation performance data of each GPU executing an operation corresponding to the attention mechanism operation type;

[0250] S405: Measure second operation performance data of each GPU executing an operation corresponding to the multilayer perceptron operation type;

[0251] S407: Measure third operation performance data of each GPU executing the operation corresponding to the preset operation type;

[0252] S409: Determine the operation performance data corresponding to each GPU according to the first operation performance data, the second operation performance data, and the third operation performance data corresponding to each GPU.

[0253] In the disclosed embodiments, first operational performance data can be measured for each GPU performing an operation corresponding to the attention mechanism operation type. Specifically, the actual throughput and efficiency of sample data of different sequence lengths (512, 1024, 2048, 4096, etc.) at the layer corresponding to the attention mechanism operation type can be measured as the first operational performance data. Second operational performance data based on general matrix multiplication (GEMM) can be measured for multi-layer perceptron layers at multiple data volume levels. Specifically, the GEMM operational performance of MLP layers of different sizes (including the optimization effect of different matrix shapes) can be measured as the second operational performance data. GEMM (General Matrix Multiplication) refers to general matrix multiplication, a fundamental and core operation in linear algebra. Third operational performance data can also be measured for each GPU in the model performing an operation corresponding to the preset operation type. Finally, the first, second, and third operational performance data corresponding to each GPU are determined as the operational performance data corresponding to each GPU, thereby improving the diversity and accuracy of the operational performance data corresponding to each GPU.

[0254] Among them, the operation-level performance matrix model can be obtained based on the operation performance data corresponding to each GPU as follows:

[0255]

[0256] in, is the operation-level performance matrix corresponding to the i-th GPU, 、 、 、 The operational performance data obtained by inputting sample data with sequence lengths of 512, 1024, 2048, and 4096 into the attention mechanism layer; 、 、 、 The GEMM operation performance of MLP layers of different sizes are shown respectively; 、 、 、 They are the operational performance of the normalization layer (layerNorm), embedding layer (Embedding), activation layer (Activation), and other layers in the model. The operational performance can include actual throughput, theoretical peak, etc.

[0257] In some embodiments, allocating a GPU to each structural layer in the to-be-trained model for model training to obtain a target model according to the multi-dimensional operation performance matrix corresponding to each GPU includes:

[0258] Obtaining the weight coefficient corresponding to each operation type in the model to be trained;

[0259] Obtaining the normalized performance of each GPU for each operation type in the model to be trained, the impact factor of the load of each GPU on the performance of the model to be trained, and the memory adaptability factor of each GPU and the model to be trained;

[0260] Determine a comprehensive fitness score for each GPU and the model to be trained based on the weight coefficient, the normalized performance corresponding to each GPU, the impact factor, and the memory fitness factor;

[0261] Determining multiple screening GPUs based on the comprehensive fitness score of each GPU and the model to be trained;

[0262] According to the multi-dimensional operation performance matrix corresponding to each screened GPU, a GPU is allocated to each structural layer in the model to be trained to perform model training to obtain a target model.

[0263] In the embodiment of the present disclosure, the weight coefficient corresponding to each operation type in the model to be trained can be obtained during the simulation operation; the operation performance of each GPU for each operation type in the model to be trained can be normalized to obtain the normalized performance of each GPU for each operation type in the model to be trained; the influence factor of the current load of each GPU in the running state on the performance of the model to be trained is obtained; and the memory adaptability factor of each GPU and the model to be trained is determined based on the hardware indicator data of each GPU; the comprehensive adaptability score of each GPU and the model to be trained is determined based on the weight coefficient, the normalized performance corresponding to each GPU, the influence factor and the memory adaptability factor. Thus, based on the comprehensive adaptability score of each GPU and the model to be trained, multiple screening GPUs that are adapted to the model are determined; finally, based on the multi-dimensional operation performance matrix corresponding to each screening GPU, a GPU is allocated to each structural layer in the model to be trained for model training to obtain a target model, thereby improving the adaptability of the GPU and the model to be trained.

[0264] Exemplarily, a comprehensive fitness score between each GPU and a structural layer of any operation type in the model to be trained can be calculated as the comprehensive fitness score between each GPU and the model to be trained M; wherein the calculation formula for the comprehensive fitness score between the i-th GPU and the model to be trained is as follows:

[0265]

[0266] in, is the comprehensive fitness score between the i-th GPU and model M, is the weight coefficient corresponding to a certain operation type in the model to be trained, is the normalized performance of the GPU for the operation type, is the impact factor of the current load on the performance of the model to be trained, is the memory compatibility factor between the GPU and the model to be trained.

[0267] In some embodiments, a load impact function can be constructed based on the load status perception strategy of each GPU in the GPU cluster, and a memory pressure impact function can be constructed by performing data fitting processing based on the hardware indicator data of each GPU in the GPU cluster, thereby constructing a dynamic performance prediction model for each GPU; so as to be used to predict the dynamic performance of the training time of each GPU.

[0268] For example, the function corresponding to the dynamic performance prediction model of the i-th GPU is as follows:

[0269] in, To determine the basic training duration, it can be determined through historical training data. is the load impact function, It is the memory pressure impact function; To predict the training time.

[0270] In some embodiments, allocating a GPU to each structural layer in the to-be-trained model for model training to obtain a target model according to the multi-dimensional operation performance matrix corresponding to each GPU includes:

[0271] Constructing a parallel pipeline group according to multiple structural layers in the model to be trained;

[0272] Determining a target position of each GPU in the parallel pipeline group according to a multi-dimensional operation performance matrix corresponding to each GPU;

[0273] According to the target position of each GPU in the heterogeneous GPU cluster in the parallel pipeline group, a GPU is allocated to each structural layer in the model to be trained to perform model training to obtain the target model.

[0274] In the disclosed embodiments, multiple structural layers in the model to be trained can be grouped to construct multiple parallel pipeline groups. The number of structural layers in each group can be the same. The target position of each GPU in the corresponding parallel pipeline group is then determined based on the multi-dimensional operational performance matrix corresponding to each GPU. A GPU is then assigned to each structural layer in the model to be trained to perform model training, thereby obtaining the target model. This allows for accurate modeling of the GPU's operational-level performance profile, enabling model-aware optimal task placement.

[0275] In some embodiments, constructing a parallel pipeline group according to multiple structural layers in the model to be trained includes:

[0276] Sequentially number the multiple structural layers in the model to be trained to construct a model layer set;

[0277] Constructing a plurality of pipeline stage sets in sequence according to the number of each structural layer in the model layer set; each pipeline stage set includes at least two consecutively numbered structural layers;

[0278] The multiple pipeline stage sets are grouped to obtain a target number of the pipeline parallel groups.

[0279] In an embodiment of the present disclosure, a set of structural layers in the model to be trained can be obtained, and each structural layer is numbered in sequence according to the connection order; then multiple consecutively numbered structural layers are combined into a pipeline stage set; the number of structural layers in each pipeline stage set can be set to the same number; if it is impossible to distribute them evenly, the same number of pipeline stage sets can be constructed first according to the numbering, and finally the remaining structural layers can be combined into a pipeline stage set.

[0280] Exemplarily, the structure layer set is:

[0281] , where 1 to L in this set correspond to different structural layer numbers;

[0282] The target number (S) of pipeline stages is:

[0283] , where 1 to S in the set correspond to the numbers of different pipeline stage sets; each pipeline stage set includes a continuous structure layer, for example, stage i includes the first Layer to layer.

[0284] In some embodiments, determining the target position of each GPU in the parallel pipeline group according to the multi-dimensional operation performance matrix corresponding to each GPU includes:

[0285] Traversing the target number of parallel pipeline groups in sequence, and determining, for a current parallel pipeline group traversed, an activation probability of the current parallel pipeline group;

[0286] Building a data parallel configuration strategy according to the target number of the pipeline parallel groups and the activation probability of the current parallel pipeline group;

[0287] Determining any structural layer in the to-be-trained model as the current structural layer, and determining any pipeline stage in the current parallel pipeline group as the current pipeline stage;

[0288] Constructing a layer stage mapping strategy according to the pipeline stage constraints of the current structural layer, the allocation probability of the current structural layer being divided into the current pipeline stage, and the mapping probability of the current pipeline stage being mapped to each GPU in the heterogeneous GPU cluster;

[0289] A target position of each GPU in the current parallel pipeline group is determined according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, and the layer stage mapping strategy.

[0290] In the disclosed embodiment, each GPU corresponds to a GPU device; the device set corresponding to the heterogeneous GPU cluster is as follows:

[0291]

[0292] Each device d corresponds to a multi-dimensional operation performance matrix , hardware indicator data: such as video memory capacity Md, communication bandwidth multi-dimensional characteristics: such as the bandwidth between the device and other devices , thus the comprehensive fitness score of device d to model M can be obtained.

[0293] An upper limit on the number of pipeline parallel groups can be set. For example, a target number (G) of pipeline parallel groups can be constructed, where the target number is less than or equal to a preset number threshold. The constructed data parallel copies are as follows:

[0294]

[0295] The activation probability of the current parallel pipeline group is determined as follows:

[0296]

[0297] The layer assignment variables are as follows:

[0298] The probability of the current structural layer being assigned to the current pipeline stage is as follows:

[0299] , which represents whether the model layer l is divided into stage i;

[0300] The corresponding pipeline stage constraints are as follows:

[0301] , which represents that each layer must and can only belong to one stage;

[0302] in, Represents the probability that the lth layer of the model belongs to stage i.

[0303] The mapping probability of the current pipeline stage to each GPU in the heterogeneous GPU cluster is as follows:

[0304] , which represents whether the i-th pipeline stage in the current parallel pipeline group g is mapped to the physical GPU device d.

[0305] This embodiment constructs a data parallel configuration strategy based on the target number of pipeline parallel groups and the activation probability of the current parallel pipeline group; constructs a layer stage mapping strategy based on the pipeline stage constraints of the current structural layer, the allocation probability of the current structural layer being divided into the current pipeline stage, and the mapping probability of the current pipeline stage to each GPU in the heterogeneous GPU cluster; finally, determines the target position of each GPU in the current parallel pipeline group based on the multi-dimensional operation performance matrix, the data parallel configuration strategy and the layer stage mapping strategy.

[0306] In some embodiments, determining a target position of each GPU in the current parallel pipeline group according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, and the layer stage mapping strategy includes:

[0307] Get the batch size and sequence length of sample data;

[0308] Constructing a batch configuration strategy based on the batch size and sequence length of the sample data;

[0309] Constructing a training time prediction function according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer-stage mapping strategy, and the batch configuration strategy;

[0310] Determine a target position of each GPU in the current parallel pipeline group according to the training time prediction function.

[0311] In an embodiment of the present disclosure, a batch configuration strategy is constructed based on the batch quantity and sequence length of the sample data; then, a training time prediction function is accurately constructed by combining the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer stage mapping strategy and the batch configuration strategy; thereby, the target position of each GPU in the current parallel pipeline group can be accurately determined based on the training time prediction function.

[0312] In some embodiments, constructing a training time prediction function according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer-stage mapping strategy, and the batch configuration strategy includes:

[0313] Determining a stage combination delay function for each pipeline stage in each pipeline parallel group according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer stage mapping strategy, and the batch configuration strategy;

[0314] Determining a steady-state critical pipeline stage in each pipeline parallel group according to a stage combination delay function of each pipeline stage in each pipeline parallel group, and determining the stage combination delay of the steady-state critical pipeline stage as a critical path time function;

[0315] Get the gradient synchronization time function corresponding to each pipeline parallel group;

[0316] Determine the complete iteration time function of the model to be trained according to the critical path time function and the gradient synchronization time function corresponding to each pipeline parallel group;

[0317] The training time prediction function is constructed according to the complete iteration time function and the scheduling management time function.

[0318] In some embodiments, the actual training time can be minimized by using a hybrid parallel simulator; the objective function of the simulator is as follows:

[0319]

[0320] in, The actual time of training one iteration (including multiple fixed-length sample data) output by the simulator; is a multi-dimensional operational performance matrix, is the probability of assigning the mapping scheme of the structural layer to the pipeline stage in the model, Configure parameters for data parallelism, Configure parameters for the batch, which can include batch size for fixed-length samples, sequence length, etc.

[0321] The optimization objective decomposition yields the following training time prediction function:

[0322]

[0323] in, is the synchronization time of gradient All-Reduce, is the execution time of the critical path stage, S is the number of pipeline stage sets, and B is the number of micro-batches of sample data; It is a scheduling management time function that can be used to determine the time required for operations such as scheduling and memory management; thereby improving the accuracy of the training time prediction function, thereby improving the prediction accuracy of the training time.

[0324] In some embodiments, as Figure 5 As shown, determining the stage combination delay function of each pipeline stage in each pipeline parallel group according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer stage mapping strategy, and the batch configuration strategy includes:

[0325] S501: Determine the normalized performance of each GPU for each operation type according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer stage mapping strategy, and the batch configuration strategy;

[0326] S503: determining a comprehensive load of the current pipeline stage in each pipeline parallel group; the comprehensive load includes a forward comprehensive load of each operation type during the forward propagation process and a reverse comprehensive load during the reverse propagation process;

[0327] S505: Determine a forward delay function of the current pipeline stage according to a ratio between the forward integrated load of each operation type in the current pipeline stage and the normalized performance corresponding to each operation type;

[0328] S507: Determine a reverse delay function of the current pipeline stage according to a ratio between the reverse comprehensive load of each operation type in the current pipeline stage and the normalized performance of each operation type;

[0329] S509: Determine a stage combination delay function of the current pipeline stage according to the forward delay function and the reverse delay function corresponding to the current pipeline stage.

[0330] In an embodiment of the present disclosure, the forward comprehensive load of each operation type in the current pipeline stage and the normalized performance corresponding to each operation type can be obtained, and then the ratio between the two can be calculated to obtain the forward delay function of the current pipeline stage; and the reverse delay function of the current pipeline stage can be further determined; finally, the stage combination delay function of the current pipeline stage is determined based on the sum of the forward delay function and the reverse delay function corresponding to the current pipeline stage.

[0331] For example, are the forward delay function and the reverse delay function of stage i, respectively. The corresponding formulas of the two are as follows:

[0332]

[0333] in, is the forward integrated load, is the reverse integrated load, is the normalized performance.

[0334] In some embodiments, as Figure 6 As shown, determining the comprehensive load of the current pipeline stage includes:

[0335] S601: Obtain a first load corresponding to the attention mechanism operation type in the current pipeline stage;

[0336] S603: Obtain a second load corresponding to the multilayer perceptron operation type in the current pipeline stage;

[0337] S605: Obtain a third load corresponding to a preset operation type in the current pipeline stage; the preset operation type includes at least one of a normalization operation type, an embedding operation type, and an activation operation type;

[0338] S607: Determine the comprehensive load of the current pipeline stage according to the first load, the second load, and the third load corresponding to the current pipeline stage.

[0339] In an embodiment of the present disclosure, obtaining the third load corresponding to the preset operation type in the current pipeline stage may include obtaining the load corresponding to the normalized operation type and the load corresponding to other operation types in the current pipeline stage; illustratively, the calculation formula for the comprehensive load is as follows:

[0340]

[0341] is the computational load of the operation type op in phase i, which may include the first load , the second load , the load corresponding to the normalized operation type and the corresponding loads for other operation types .

[0342] For example, for the current pipeline stage ( ), the calculation formula for each load obtained by decomposing the typical layer of the Transformer model is as follows:

[0343]

[0344] in, Represents the probability that the lth layer of the model belongs to stage i.

[0345] In some embodiments, determining the stage combined delay function of the current pipeline stage according to the forward delay function and the reverse delay function corresponding to the current pipeline stage includes:

[0346] Determining a non-overlapping communication residual time function between the current pipeline stage and an adjacent pipeline stage based on the data volume of each GPU in the current pipeline stage and the ratio of the bandwidth between each GPU and the communication GPU;

[0347] Determine the stage combined delay function of the current pipeline stage according to the forward delay function, the reverse delay function, and the communication residual time function corresponding to the current pipeline stage.

[0348] In the embodiment of the present disclosure, the stage combination delay function of the current pipeline stage corresponds to the following:

[0349]

[0350] in, are the forward delay function and the reverse delay function of stage i, is the residual communication time between phase i and the adjacent phase that does not completely overlap, and its corresponding calculation formula is as follows:

[0351]

[0352] in, is the output activation data amount or gradient data amount of stage i, For two devices The bandwidth between.

[0353] For example, the maximum stage combination delay can be screened out according to the stage combination delay function of each pipeline stage as follows:

[0354]

[0355] For example, the pipeline steady-state critical path stage The determination method is as follows:

[0356]

[0357] The complete iteration time can be calculated as follows:

[0358]

[0359] in, The time it takes for the pipeline stage to process B micro-batches from filling to flushing; It is the synchronization time of gradient All-Reduce.

[0360] In some embodiments, determining a target position of each GPU in the current parallel pipeline group according to the training time prediction function includes:

[0361] Obtain GPU mapping uniqueness constraints, video memory capacity constraints, computing performance constraints, communication bandwidth constraints, and critical path constraints to obtain strategy constraints;

[0362] Obtaining, based on the strategy constraints, the training time prediction function, and the hybrid parallel simulator, the training time corresponding to different positions of each GPU in the current parallel pipeline group;

[0363] The position where the training time corresponding to each GPU meets a preset condition is determined as the target position of each GPU in the current parallel pipeline group.

[0364] In the embodiment of the present disclosure, the layer partitioning constraint conditions may be constructed first:

[0365] , that is, each layer must and can only belong to one stage;

[0366] If layer l and layer l+1 belong to different stages, then:

[0367] , we can get .

[0368] The GPU mapping uniqueness constraint can be:

[0369]

[0370] Computational performance constraints can be:

[0371]

[0372] in, is the comprehensive fitness score between the i-th GPU and model M;

[0373]

[0374] in, is the computational load of the operation type op in phase i, which may include the first load , the second load , the load corresponding to the normalized operation type and the corresponding loads for other operation types ; is the normalized performance of the GPU for the operation type, is the impact factor of the current load on the performance of the model to be trained, is the memory compatibility factor between the GPU and the model to be trained.

[0375] The communication bandwidth constraints are as follows:

[0376] For adjacent phases i and i+1, if they are mapped on different devices d and d′, then:

[0377]

[0378] is the amount of output activation data or gradient data at stage i.

[0379] The critical path constraints are as follows:

[0380] .

[0381] In some embodiments, the method for constructing the video memory capacity constraint condition includes:

[0382] For the current pipeline stage in each pipeline parallel group, obtain the model parameter memory of the current structure layer and the activation data memory generated during the forward propagation process;

[0383] Determining a target required parameter memory of the current pipeline stage according to an allocation probability of each current structural layer in the to-be-trained model being divided into the current pipeline stage and a model parameter memory of each current structural layer;

[0384] Determining a target demand activation memory function of the current pipeline stage according to an allocation probability of each current structural layer in the to-be-trained model being divided into the current pipeline stage and an activation data memory corresponding to each current structural layer;

[0385] Activate the memory function according to the target demand and construct the video memory capacity constraint condition.

[0386] In the embodiment of the present disclosure, the method for constructing the video memory capacity constraint condition is as follows:

[0387] Get the required parameter memory of the current pipeline stage i:

[0388] ,in, is the model parameter memory of the lth layer;

[0389] Get the demand activation data memory for the current pipeline stage i:

[0390] ,in, Memory for activation data generated by the forward pass of layer l;

[0391] If B micro-batch sample data are used for gradient accumulation, the formula for the demand activation memory function of the current pipeline stage i is as follows:

[0392]

[0393] The video memory capacity constraints are as follows:

[0394]

[0395] in, is the video memory capacity of GPU device d.

[0396] In some embodiments, determining a target position of each GPU in the current parallel pipeline group according to the training time prediction function includes:

[0397] Obtaining, according to the stage combination delay function of the current pipeline stage, stage combination delay data corresponding to different positions of each GPU in the current parallel pipeline group;

[0398] Calculating the pipeline parallel voiding rate and the data parallel voiding rate corresponding to different positions of each GPU in the current parallel pipeline group;

[0399] Determine the training time of each GPU at different positions in the current parallel pipeline group based on the stage combination delay data corresponding to each GPU at different positions in the current parallel pipeline group, the pipeline parallel voiding rate, the data parallel voiding rate, and the training time prediction function;

[0400] Determine a target position of each GPU in the current parallel pipeline group according to the training time of each GPU at different positions in the current parallel pipeline group.

[0401] For example, 、 (the probability of the current structural layer being divided into the current pipeline stage), the target number of pipeline parallel groups G, the micro-batch B of sample data, and the bandwidth function , model structure layer parameters (which may include , ) and variable-length data features (including sequence length of sample data, FLOPs, etc.) as the input of the hybrid parallel simulator. is the model parameter memory of the lth layer; It is the activation data memory generated by the forward direction of the lth layer; the final output is , parallel cavitation rate of pipeline , data parallel cavitation rate and the complete iteration time .in, For an empty slot, The term "idle slots" refers to the waiting time for asynchronous requests. In hybrid parallel simulators, "idle slots" describe underutilized computing resource units within the hybrid computing architecture. These typically refer to "idle computing power" caused by uneven task distribution, dynamic load fluctuations, or communication latency. In large-scale, high-load two-phase flow simulations such as cavitation fraction calculations, managing idle slots directly impacts simulation efficiency. Optimizing the reuse of these "idle resources" can significantly reduce computation time and costs.

[0402] This embodiment aims to determine the optimal position arrangement (target position) of heterogeneous GPUs in a pipeline parallel group. The decision variable is the different positions (ranks) of each GPU in the pipeline parallel group; the key constraints include:

[0403] (1) Memory constraint: The peak memory requirements of different pipeline parallel groups are different; different ranks correspond to the activation values ​​of multiple micro-batches; then the memory requirement parameter of the r-th position is:

[0404]

[0405] in, Memory parameters required for video memory, Memory occupied by activations for a single mini-batch.

[0406] (2) Performance matching: GPU performance matches the computational workload of the structural layer it is responsible for; examples are as follows:

[0407] Four heterogeneous GPUs: 80GB-A100, 48GB-V100, 24GB-RT×4090, 48GB-A40;

[0408] Its feasible arrangement needs to meet the following conditions: A100 can be placed in any position (sufficient video memory), RT×4090 can only be placed in the back position (smaller video memory), and video memory constraints and performance optimization need to be comprehensively considered.

[0409] In some embodiments, to avoid exponential brute force enumeration, a two-phase approach of constrained pruning and simulator-driven search can be adopted:

[0410] 1. Constraint pruning stage:

[0411] The peak memory requirements of each GPU at different positions (PP-ranks) in the pipeline parallel group are pre-calculated; only GPU-ranks that meet the memory requirements are retained in the candidate pool. For each candidate solution, the communication lower bound of five types of parallelism (PP, TP, DP, UP, and CP) is quickly estimated based on the communication topology, filtering out solutions that are significantly worse than the current optimal upper bound. PP stands for pipeline parallelism, TP for tensor parallelism, DP for data parallelism, UP for structured parallelism, and CP for control parallelism. After double pruning, the candidate size is typically less than 100 ranks, significantly reducing the data size compared to a complete enumeration of G pipeline parallel groups.

[0412] 2. Simulator-driven search phase

[0413] Beam Search (width b = 4-8) is used to perform local neighborhood exploration on the remaining candidates; the move operation only exchanges GPU positions locally to keep the GPU memory available. Beam Search is a heuristic search algorithm that is mainly used to find the optimal or near-optimal output sequence in sequence generation tasks. It is an improvement on Greedy Search, balancing the search space and computational efficiency by maintaining the k best candidates at each step. At each step, a fast model is used to give a heuristic score, and only a small number of solutions with the highest scores are called on the hybrid parallel simulator to obtain accurate training time. If the new plan If the result is better than the current optimal one, then a beam search is performed; if there is no improvement for L consecutive steps or the time budget is reached, the process is terminated. This allows the target position of each GPU in the parallel pipeline group to be accurately determined.

[0414] This embodiment proposes a data scheduling strategy that leverages hardware heterogeneity for load compensation, transforming hardware performance imbalances into a scheduling advantage rather than a burden. The sample cost-aware scheduling of this embodiment achieves precise allocation based on accurate modeling of sample computational complexity. The online adaptive rescheduling of this embodiment dynamically adjusts the allocation strategy based on real-time load status. The performance-weighted LPT algorithm of this embodiment adopts an improved longest job first algorithm that takes into account GPU performance differences. The performance-weighted LPT algorithm (Performance-Weighted Longest Processing Time First Scheduling Algorithm) is an extension of the traditional longest processing time first (LPT) scheduling algorithm. By introducing task performance weights (such as computational intensity, resource requirements, deadlines, etc.), it optimizes task allocation in multi-machine, multi-node environments, with the goal of meeting the differentiated performance requirements of tasks while reducing the maximum completion time.

[0415] This embodiment uses sample feature extraction and modeling, and adopts a dynamic load balancing algorithm for real-time detection and adaptive rescheduling mechanism for multi-dimensional sample complexity quantification (sequence length, FLOPS, memory requirements).

[0416] Related technologies pursue uniform sample distribution and ignore hardware performance differences; however, this embodiment matches sample distribution with hardware performance to achieve heterogeneous load compensation; related technologies only focus on model layer division and give insufficient consideration to data scheduling; however, this embodiment proposes sample-level heterogeneous perception scheduling for the first time to achieve end-to-end optimization.

[0417] In some embodiments, allocating a GPU to each structural layer in the to-be-trained model for model training to obtain a target model according to the multi-dimensional operation performance matrix corresponding to each GPU includes:

[0418] Allocate a GPU to each structural layer in the to-be-trained model according to the multi-dimensional operation performance matrix corresponding to each GPU to obtain a GPU scheduling strategy;

[0419] Determine the sample computation cost of each sample data according to the sequence length, floating-point operation amount, or historical execution time corresponding to similar sample data of each sample data in the sample data set;

[0420] Determine the performance data of each GPU based on the current load status of each GPU;

[0421] According to the sample calculation cost of each sample data and the performance data of each GPU, the GPU corresponding to each sample data is determined to obtain the sample allocation strategy;

[0422] According to the GPU scheduling strategy and the sample allocation strategy, the model to be trained is trained to obtain a target model.

[0423] In the embodiment of the present disclosure, for each sample data i, multiple methods can be used to determine the sample calculation cost.

[0424] The sequence length method can be used to obtain the sample calculation cost, and the calculation formula is as follows:

[0425]

[0426] in, is the sample sequence length, is a preset constant;

[0427] In addition, the FLOPs estimation method can also be used to obtain the sample calculation cost; the sample calculation cost can also be predicted based on the historical execution time corresponding to similar sample data of each sample data.

[0428] In some embodiments, the sample required memory can be calculated using the following formula:

[0429]

[0430] in, is the model dimension, are coefficients determined according to the model structure.

[0431] In some embodiments, sample data with similar sequence lengths in the sample data set may be spliced ​​to obtain sample spliced ​​data; the sequence length of the sample spliced ​​data is less than or equal to the target length;

[0432] The splicing strategies can include: (1) shortest fit priority method: prioritize splicing samples of similar length together; (2) total length balance method: control the total sequence length of each spliced ​​batch to be close to the target value; (3) memory alignment method: consider the GPU memory page alignment and optimize the memory layout after splicing according to the memory pages of each GPU.

[0433] In some embodiments, differentiated scheduling can be performed based on the actual performance capabilities of the GPU nodes, and the corresponding weights can be adjusted based on the real-time load of the GPUs;

[0434]

[0435] in, is the weight of the current GPU device, is the actual load of the current GPU device, is the sum of the actual loads of all devices in the GPU cluster, The availability factor of the current GPU device can be determined based on the current load status of the current GPU device. The value range is 0-1, where 0 indicates unavailable and 1 indicates available.

[0436] The load compensation mechanism used in the embodiment of the present disclosure is as follows:

[0437] Prioritize lightweight sample allocation: samples with lower computational cost are sent to nodes with weaker performance or on the critical path first; centralize heavy sample processing: samples with intensive computation are allocated to nodes with strong computing power; dynamic load adjustment: the allocation strategy of subsequent samples is dynamically adjusted according to the real-time processing progress.

[0438] In some embodiments, determining the GPU corresponding to each sample data according to the sample calculation cost of each sample data and the performance data of each GPU to obtain the sample allocation strategy includes:

[0439] Get the sample allocation probability of each sample data being assigned to each GPU;

[0440] Constructing a sample allocation constraint condition based on the sample allocation probability of each GPU;

[0441] Constructing a load balancing constraint condition according to the sample allocation probability corresponding to each GPU and the performance data;

[0442] Determining a comprehensive constraint condition according to the sample allocation constraint condition and the load balancing constraint condition;

[0443] Construct a target time function based on the sample allocation probability and performance data corresponding to each GPU;

[0444] According to the comprehensive constraint conditions and the target time function, the GPU corresponding to each sample data is determined to obtain a sample allocation strategy.

[0445] In the embodiment of the present disclosure, it is assumed that the sample data set , the sample calculation cost of each sample data is , the sample memory requirement is , the GPU set is , each With performance coefficient and video memory capacity ; The decision variable is the probability of assigning each sample data to each GPU: , which represents whether the sample data is assigned to ; Then construct the target time function , , to minimize the maximum completion time.

[0446] The sample allocation constraints are as follows:

[0447]

[0448] The load balancing constraints are as follows:

[0449]

[0450] The video memory capacity constraints are as follows:

[0451]

[0452] If the minimum batch size is set in advance , maximum batch , then the batch quantity constraints are as follows:

[0453]

[0454] In some embodiments, determining the comprehensive constraint condition according to the sample distribution constraint condition and the load balancing constraint condition includes:

[0455] Determine sample memory requirements based on the model dimensions and architecture coefficients of the model to be trained;

[0456] Constructing a target video memory capacity constraint condition based on the sample allocation probability, video memory capacity, and sample memory requirement corresponding to each GPU;

[0457] Constructing a batch quantity constraint condition for the sample data set;

[0458] The comprehensive constraint condition is determined according to the sample allocation constraint condition, the load balancing constraint condition, the target video memory capacity constraint condition and the batch quantity constraint condition.

[0459] In an embodiment of the present disclosure, a longest job first (LPT) heuristic algorithm may be used as a sample allocation algorithm; input data includes a sample data set, a GPU set, and the normalized performance of each operation type in the model; the sample data set is sorted, and then greedy allocation is performed to select the GPU with the lightest load for the sample data; video memory constraints are checked to find a suboptimal GPU or split the sample; and load balancing measurement is performed simultaneously.

[0460] For small-scale problems, dynamic programming can be used to obtain exact solutions.

[0461] The states are defined as follows:

[0462] After characterizing the first k sample data distribution, the cumulative load of each GPU is The optimal solution when

[0463] State transition satisfies:

[0464]

[0465] In some embodiments, after determining the GPU corresponding to each sample data based on the sample calculation cost of each sample data and the performance data of each GPU and obtaining the sample allocation strategy, the method further includes:

[0466] Based on the sample allocation strategy, the current sample data is input into the model to be trained to obtain the real-time data processing progress of the current GPU corresponding to the current sample data;

[0467] According to the real-time data processing progress of the current GPU, the sample allocation strategy of the sample data located after the current sample data is dynamically adjusted.

[0468] During the actual training process, an online adaptive scheduling algorithm can be used for scheduling, and sample allocation can be dynamically adjusted according to the real-time load of the GPU.

[0469] The performance data of each GPU can also be dynamically adjusted based on the dynamic performance adjustment model. If the GPU's real-time load is greater than a first load threshold and less than a second load threshold, the performance data will decrease less than the first load threshold. If the first load threshold is less than the second load threshold, the performance data will decrease more than the first load threshold. In other words, performance will decrease slightly under high load and significantly under overload. A load rebalancing strategy can also be constructed to calculate the expected distribution ratio and then distribute sample data according to this ratio.

[0470] Through the above complete mathematical modeling and algorithm design, the sample scheduling strategy of this embodiment can fully utilize hardware heterogeneity, transform hardware performance imbalance into the advantage of load distribution, and achieve dynamic balancing of computing load in heterogeneous clusters.

[0471] The disclosed embodiments adopt multi-dimensional heterogeneous performance portrait modeling technology and propose a multi-dimensional quantitative modeling framework for GPU heterogeneous performance, breaking through the traditional simple assumption based on theoretical peak values. It adopts static-dynamic fusion modeling + multi-level normalization system + real-time performance prediction, providing accurate performance basis for heterogeneous environment scheduling, serving as the data foundation for the entire optimization system, and constructing a complete dynamic performance portrait for the first time.

[0472] This disclosed embodiment employs a constraint-aware joint task placement optimization algorithm, combining simulator solving with heuristic strategies for the first time to solve the NP-hard task placement problem in heterogeneous environments. NP refers to non-deterministic polynomials (NP), and NP-hard refers to problems for which all NP problems can be reduced to polynomial time complexity. A combination of mixed integer programming modeling, greedy heuristics, and local optimization of bottleneck areas achieves globally optimal partitioning of pipelines for significant improvements in training throughput. Global optimization is guaranteed based on mathematical optimization theory.

[0473] The disclosed embodiments adopt a load-compensated heterogeneous-aware data scheduling strategy, overturning the traditional concept of uniform distribution and proposing for the first time a scheduling mechanism that utilizes hardware heterogeneity for load compensation. The performance-weighted LPT algorithm, sample complexity modeling, and online adaptive rescheduling are used to transform hardware performance imbalances into scheduling advantages, achieving "heterogeneity with heterogeneity" and realizing sample-level heterogeneous-aware optimization for the first time.

[0474] The embodiments of the present disclosure have the following beneficial effects:

[0475] (1) Shortened training time

[0476] The actual training time can be shortened by up to 15% (based on accurate evaluation using a simulator). The time required to complete the same training task is significantly reduced under the same hardware configuration. This significantly accelerates and saves costs for long-term training operations on very large models.

[0477] (2) Reduce the impact of slow GPU nodes on the training process

[0478] Significantly reduce the drag of slow nodes on overall training, effectively prevent the "barrel effect", and reduce the situation where high-speed GPUs are waiting for low-speed GPUs.

[0479] (3) Improved the efficiency of the assembly line

[0480] Significantly improve the efficiency and stability of pipeline parallelism; critical path optimization makes each stage of the pipeline more balanced, reducing pipeline bubbles and waiting time.

[0481] (4) Enhanced fault tolerance

[0482] Improve the robustness of the training process, dynamically adjust sample allocation when node performance fluctuates, and enhance the fault tolerance and stability of large-scale training.

[0483] (5) Fully utilize the computing power of each GPU in the GPU collection

[0484] Each GPU in the GPU set can play a more full role, reducing idle and wasted computing power, and increasing the proportion of effective running time of training jobs.

[0485] (6) Improved communication efficiency

[0486] Considering topology bandwidth, data transmission delay is reduced, model gradient synchronization is more efficient, and upstream and downstream data transmission are optimized.

[0487] In some embodiments, as Figure 7 As shown, Figure 7 The following is a flowchart of a model training method for a model training system based on a heterogeneous GPU cluster, including:

[0488] Obtain a heterogeneous GPU cluster and a large model structure; then divide the model training into three stages.

[0489] Phase 1: Building a machine performance profile; this mainly includes three steps: collecting static hardware metrics, measuring dynamic runtime metrics, and mathematically modeling the performance profile.

[0490] Phase 2: Task scheduling and GPU placement; implemented through MLP exact solution, greedy heuristic strategy, and hybrid optimization strategy;

[0491] Phase 3: Sample data scheduling strategy; including sample feature extraction and modeling, sample packaging and sorting, and heterogeneous perception differentiated scheduling; during the scheduling process, the LPT heuristic algorithm, online adaptive scheduling, and optimized training scheduling scheme are successively adopted to minimize training time and improve training efficiency.

[0492] Figure 8 This is a block diagram of a model training device based on a heterogeneous GPU cluster according to an exemplary embodiment. Figure 8 , the device comprises:

[0493] The hardware data acquisition module 810 is configured to acquire hardware indicator data of each GPU in a heterogeneous GPU cluster; the heterogeneous GPU cluster includes at least two heterogeneous GPUs;

[0494] The operation performance data acquisition module 820 is configured to obtain multiple operation types corresponding to the deep learning algorithm in the training model and measure the operation performance data of each GPU executing the operation corresponding to each operation type;

[0495] The load strategy acquisition module 830 is configured to execute the acquisition of the multi-dimensional characteristics of the communication bandwidth of each GPU and the load state perception strategy; the load state perception strategy characterizes the correlation between the load and performance of each GPU;

[0496] A performance matrix construction module 840 is configured to construct a multi-dimensional operation performance matrix for each GPU based on the hardware indicator data, the operation performance data, the multi-dimensional characteristics of the communication bandwidth, and the load state perception strategy corresponding to each GPU;

[0497] The model training module 850 is configured to execute the multi-dimensional operation performance matrix corresponding to each GPU, and allocate a GPU to each structural layer in the model to be trained to perform model training to obtain a target model.

[0498] In an exemplary embodiment, the load strategy acquisition module includes:

[0499] a delay characteristic determination unit configured to determine bandwidth and delay characteristics of each GPU under messages of multiple data volume levels;

[0500] A performance data determining unit is configured to determine performance data of each GPU executing a multi-process aggregate operation;

[0501] An effective bandwidth determination unit is configured to determine the effective bandwidth of each GPU under multiple load modes using a preset network technology; the preset network technology includes at least one of a wireless bandwidth technology and an Ethernet technology;

[0502] a communication bandwidth multi-dimensional feature determination unit configured to determine the communication bandwidth multi-dimensional feature of each GPU based on the bandwidth, latency feature, performance data, and effective bandwidth corresponding to each GPU;

[0503] The strategy determination unit is configured to determine a load state awareness strategy for each GPU based on the performance of each GPU under different loads.

[0504] In an exemplary embodiment, the policy determination unit includes:

[0505] The first relationship determination subunit is configured to analyze the performance decay curves corresponding to each GPU executing multiple operation types at different utilization rates to obtain a load-performance correlation relationship;

[0506] The second relationship determination subunit is configured to determine a usage-performance correlation relationship between the modeling memory usage and the computing performance of each GPU;

[0507] a stability analysis result determination subunit configured to determine a performance stability analysis result of each GPU under a target load within a target period; the cumulative duration of the target period is greater than or equal to a preset duration, and the target load is greater than or equal to a preset load threshold;

[0508] The strategy determination subunit is configured to determine the load state perception strategy of each GPU according to the load-performance association relationship, the usage-performance association relationship and the performance stability analysis result corresponding to each GPU.

[0509] In an exemplary embodiment, the operation performance data acquisition module includes:

[0510] An operation type acquisition unit is configured to execute and acquire an attention mechanism operation type, a multi-layer perceptron operation type, and a preset operation type corresponding to the deep learning algorithm in the to-be-trained model; the preset operation type includes at least one of a normalization operation type, an embedding operation type, and an activation operation type;

[0511] A first data measuring unit is configured to measure first operation performance data of each GPU executing an operation corresponding to the attention mechanism operation type;

[0512] A second data measuring unit is configured to measure second operation performance data of each GPU executing an operation corresponding to the multilayer perceptron operation type;

[0513] A third data measuring unit is configured to measure third operation performance data of each GPU executing an operation corresponding to the preset operation type;

[0514] The operation data determining unit is configured to determine the operation performance data corresponding to each GPU according to the first operation performance data, the second operation performance data, and the third operation performance data corresponding to each GPU.

[0515] In an exemplary embodiment, the model training module includes:

[0516] A pipeline group construction unit is configured to construct a parallel pipeline group according to multiple structural layers in the model to be trained;

[0517] a target position determination unit, configured to determine a target position of each GPU in the parallel pipeline group according to a multi-dimensional operation performance matrix corresponding to each GPU;

[0518] The model training unit is configured to perform model training by allocating a GPU to each structural layer in the model to be trained according to the target position of each GPU in the heterogeneous GPU cluster in the parallel pipeline group to obtain the target model.

[0519] In an exemplary embodiment, the pipeline group construction unit is further configured to sequentially number the multiple structural layers in the model to be trained to construct a model layer set; construct multiple pipeline stage sets in sequence according to the number of each structural layer in the model layer set; each pipeline stage set includes at least two consecutively numbered structural layers; and group the multiple pipeline stage sets to obtain a target number of pipeline parallel groups.

[0520] In an exemplary embodiment, the model training module includes:

[0521] An activation probability determination unit is configured to sequentially traverse the target number of parallel pipeline groups and determine an activation probability of a current parallel pipeline group that has been traversed;

[0522] a parallel configuration strategy building unit, configured to build a data parallel configuration strategy according to the target number of pipeline parallel groups and the activation probability of the current parallel pipeline group;

[0523] a current structural layer determining unit configured to determine any structural layer in the to-be-trained model as the current structural layer, and to determine any pipeline stage in the current parallel pipeline group as the current pipeline stage;

[0524] an allocation probability determining unit configured to construct a layer stage mapping strategy according to the pipeline stage constraint of the current structural layer, the allocation probability of the current structural layer being divided into the current pipeline stage, and the mapping probability of the current pipeline stage being mapped to each GPU in the heterogeneous GPU cluster;

[0525] The position determination unit is configured to determine the target position of each GPU in the current parallel pipeline group according to the multi-dimensional operation performance matrix, the data parallel configuration strategy and the layer stage mapping strategy.

[0526] In an exemplary embodiment, the position determination unit includes:

[0527] A sequence length acquisition subunit is configured to execute the acquisition of the batch number and sequence length of the sample data;

[0528] A batch configuration strategy construction subunit is configured to execute the construction of a batch configuration strategy according to the batch quantity and sequence length of the sample data;

[0529] a time prediction function construction subunit, configured to construct a training time prediction function according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer-stage mapping strategy, and the batch configuration strategy;

[0530] The position determination subunit is configured to execute the training time prediction function to determine the target position of each GPU in the current parallel pipeline group.

[0531] In an exemplary embodiment, the time prediction function construction subunit includes:

[0532] a stage combination delay function determination subunit, configured to determine the stage combination delay function of each pipeline stage in each pipeline parallel group according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer-stage mapping strategy, and the batch configuration strategy;

[0533] a critical path time function determination subunit configured to determine a steady-state critical pipeline stage in each pipeline parallel group according to a stage combination delay function of each pipeline stage in each pipeline parallel group, and determine the stage combination delay of the steady-state critical pipeline stage as the critical path time function;

[0534] A synchronization time function acquisition subunit is configured to execute and acquire the gradient synchronization time function corresponding to each pipeline parallel group;

[0535] a complete iteration time function determination subunit, configured to determine the complete iteration time function of the to-be-trained model according to the critical path time function and the gradient synchronization time function corresponding to each pipeline parallel group;

[0536] The training time prediction function construction subunit is configured to execute the construction of the training time prediction function according to the complete iteration time function and the scheduling management time function.

[0537] In an exemplary embodiment, the stage combination delay function determination subunit includes:

[0538] a normalized performance determination subunit, configured to determine the normalized performance of each GPU for each operation type based on the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer stage mapping strategy, and the batch configuration strategy;

[0539] a comprehensive load determination subunit configured to determine the comprehensive load of the current pipeline stage in each pipeline parallel group; the comprehensive load includes the forward comprehensive load of each operation type during the forward propagation process and the reverse comprehensive load during the reverse propagation process;

[0540] a forward delay function determination subunit, configured to determine the forward delay function of the current pipeline stage according to a ratio between the forward comprehensive load of each operation type in the current pipeline stage and the normalized performance corresponding to each operation type;

[0541] a reverse delay function determining subunit, configured to determine a reverse delay function of the current pipeline stage according to a ratio between the reverse comprehensive load of each operation type in the current pipeline stage and the normalized performance of each operation type;

[0542] The combined delay function determining subunit is configured to determine the stage combined delay function of the current pipeline stage according to the forward delay function and the reverse delay function corresponding to the current pipeline stage.

[0543] In an exemplary embodiment, the comprehensive load determination subunit is further configured to execute the steps of obtaining a first load corresponding to the attention mechanism operation type in the current pipeline stage; obtaining a second load corresponding to the multi-layer perceptron operation type in the current pipeline stage; obtaining a third load corresponding to a preset operation type in the current pipeline stage; the preset operation type includes at least one of a normalization operation type, an embedding operation type, and an activation operation type; and determining the comprehensive load of the current pipeline stage based on the first load, the second load, and the third load corresponding to the current pipeline stage.

[0544] In an exemplary embodiment, the combined delay function determination subunit is further configured to determine the non-overlapping communication residual time function between the current pipeline stage and the adjacent pipeline stage based on the data volume of each GPU in the current pipeline stage and the ratio of the bandwidth between each GPU and the communication GPU; and determine the stage combined delay function of the current pipeline stage based on the forward delay function, reverse delay function and the communication residual time function corresponding to the current pipeline stage.

[0545] In an exemplary embodiment, the position determination subunit is further configured to execute acquisition of GPU mapping uniqueness constraints, video memory capacity constraints, computing performance constraints, communication bandwidth constraints, and critical path constraints to obtain strategy constraints; based on the strategy constraints, the training time prediction function, and the hybrid parallel simulator, obtain the training time corresponding to different positions of each GPU in the current parallel pipeline group; and determine the position where the training time corresponding to each GPU meets the preset conditions as the target position of each GPU in the current parallel pipeline group.

[0546] In an exemplary embodiment, the apparatus further comprises:

[0547] an activation data memory acquisition module, configured to execute, for a current pipeline stage in each pipeline parallel group, acquiring a model parameter memory of the current structural layer and an activation data memory generated during a forward propagation process;

[0548] a target requirement parameter memory acquisition module configured to determine the target requirement parameter memory of the current pipeline stage according to the allocation probability of each current structural layer in the to-be-trained model being divided into the current pipeline stage and the model parameter memory of each current structural layer;

[0549] a target demand activation memory function determination module, configured to determine the target demand activation memory function of the current pipeline stage according to the allocation probability of each current structural layer in the to-be-trained model being divided into the current pipeline stage and the activation data memory corresponding to each current structural layer;

[0550] The video memory capacity constraint condition construction module is configured to execute the memory activation function according to the target demand and construct the video memory capacity constraint condition.

[0551] In an exemplary embodiment, the position determination subunit is further configured to execute a stage combination delay function according to the current pipeline stage to obtain stage combination delay data corresponding to different positions of each GPU in the current parallel pipeline group; calculate the pipeline parallel voiding rate and data parallel voiding rate corresponding to different positions of each GPU in the current parallel pipeline group; determine the training time of each GPU at different positions in the current parallel pipeline group according to the stage combination delay data, pipeline parallel voiding rate, data parallel voiding rate and the training time prediction function corresponding to different positions of each GPU in the current parallel pipeline group; and determine the target position of each GPU in the current parallel pipeline group according to the training time of each GPU at different positions in the current parallel pipeline group.

[0552] In an exemplary embodiment, the model training module includes:

[0553] A weight coefficient acquisition unit is configured to acquire a weight coefficient corresponding to each operation type in the to-be-trained model;

[0554] an impact factor acquisition unit, configured to acquire the normalized performance of each GPU for each operation type in the to-be-trained model, an impact factor of the load of each GPU on the performance of the to-be-trained model, and a memory compatibility factor between each GPU and the to-be-trained model;

[0555] a fitness score determining unit configured to determine a comprehensive fitness score between each GPU and the model to be trained based on the weight coefficient, the normalized performance corresponding to each GPU, the impact factor, and the memory fitness factor;

[0556] A screening unit is configured to determine a plurality of screening GPUs based on a comprehensive fitness score between each GPU and the model to be trained;

[0557] The model determination unit is configured to execute the multi-dimensional operation performance matrix corresponding to each screened GPU, and allocate a GPU to each structural layer in the model to be trained to perform model training to obtain a target model.

[0558] In an exemplary embodiment, the model training module includes:

[0559] a scheduling strategy determining unit configured to allocate a GPU to each structural layer in the to-be-trained model according to a multi-dimensional operation performance matrix corresponding to each GPU, thereby obtaining a GPU scheduling strategy;

[0560] a sample calculation cost determination unit configured to determine the sample calculation cost of each sample data according to the sequence length, floating-point operation amount, or historical execution time corresponding to similar sample data of each sample data in the sample data set;

[0561] a performance data determining unit, configured to determine performance data of each GPU based on a current load state of each GPU;

[0562] a sample allocation strategy determination unit configured to determine the GPU corresponding to each sample data according to the sample calculation cost of each sample data and the performance data of each GPU, and obtain a sample allocation strategy;

[0563] The model determination unit is configured to execute training on the model to be trained according to the GPU scheduling strategy and the sample allocation strategy to obtain a target model.

[0564] In an exemplary embodiment, the sample allocation strategy determination unit includes:

[0565] The sample allocation probability acquisition subunit is configured to obtain the sample allocation probability of each sample data being allocated to each GPU;

[0566] a sample allocation constraint condition construction subunit, configured to construct a sample allocation constraint condition according to the sample allocation probability of each GPU;

[0567] A load balancing constraint condition construction subunit is configured to construct a load balancing constraint condition according to the sample allocation probability corresponding to each GPU and the performance data;

[0568] a comprehensive constraint condition determination subunit, configured to determine a comprehensive constraint condition according to the sample allocation constraint condition and the load balancing constraint condition;

[0569] The target time function construction subunit is configured to execute the construction of the target time function according to the sample allocation probability and performance data corresponding to each GPU;

[0570] The sample allocation strategy determination subunit is configured to determine the GPU corresponding to each sample data according to the comprehensive constraint condition and the target time function, and obtain the sample allocation strategy.

[0571] In an exemplary embodiment, the comprehensive constraint determination subunit is further configured to determine the sample memory requirement based on the model dimension and architecture coefficient of the model to be trained; construct a target memory capacity constraint based on the sample allocation probability, memory capacity and sample memory requirement corresponding to each GPU; construct a batch quantity constraint for the sample data set; and determine the comprehensive constraint based on the sample allocation constraint, the load balancing constraint, the target memory capacity constraint and the batch quantity constraint.

[0572] In an exemplary embodiment, the apparatus further comprises:

[0573] a data processing progress determination module configured to execute, based on the sample allocation strategy, input current sample data into the to-be-trained model, and obtain the real-time data processing progress of the current GPU corresponding to the current sample data;

[0574] The sample allocation strategy adjustment module is configured to dynamically adjust the sample allocation strategy of the sample data located after the current sample data according to the real-time data processing progress of the current GPU.

[0575] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0576] In an exemplary embodiment, an electronic device is also provided, including a processor; a memory for storing processor-executable instructions; wherein, when the processor is configured to execute the instructions stored in the memory, it implements the model training method based on a heterogeneous GPU cluster provided in any of the above embodiments.

[0577] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Figure 9 is a block diagram of a server according to an exemplary embodiment. Figure 9As shown, the server 900 may vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 910 (processor 910 may include, but is not limited to, a processing device such as a microprocessor (MCU) or a programmable logic device (FPGA), a memory 930 for storing data, and one or more storage media 920 (e.g., one or more mass storage devices) for storing application programs 923 or data 922. The memory 930 and storage media 920 may be either transient or persistent storage. The program stored in the storage medium 920 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the CPU 910 may be configured to communicate with the storage medium 920 to execute the series of instruction operations in the storage medium 920 on the server 900. The server 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input and output interfaces 940, and / or one or more operating systems 921, such as Windows Server™, Mac OSX™, Unix™, Linux™, FreeBSD™, etc.

[0578] The input / output interface 940 can be used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the server 900. In one embodiment, the input / output interface 940 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the input / output interface 940 may be a radio frequency (RF) module for wireless communication with the Internet.

[0579] It can be understood by those skilled in the art that Figure 9 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 9 More or fewer components than shown, or with Figure 9 Different configurations shown.

[0580] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 930 including instructions. The instructions may be executed by the processor 910 of the apparatus 900 to perform the above method. Alternatively, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0581] In an exemplary embodiment, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the model training method based on a heterogeneous GPU cluster provided in any of the above embodiments.

[0582] Figure 10 This is a block diagram of an electronic device for model training based on a heterogeneous GPU cluster according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 10 As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a model training method based on a heterogeneous GPU cluster is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0583] Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0584] The present invention obtains hardware indicator data of each GPU in a heterogeneous GPU cluster; the heterogeneous GPU cluster includes at least two heterogeneous GPUs; obtains multiple operation types corresponding to the deep learning algorithm in the model to be trained, and measures the operation performance data of each GPU executing the operation corresponding to each operation type; obtains the multidimensional characteristics of the communication bandwidth and the load state perception strategy of each GPU; the load state perception strategy characterizes the correlation between the load and performance of each GPU; constructs a multidimensional operation performance matrix for each GPU based on the hardware indicator data, the operation performance data, the multidimensional characteristics of the communication bandwidth and the load state perception strategy corresponding to each GPU; thereby, the GPU static hardware indicators and dynamic runtime characteristics are integrated to obtain a multidimensional operation performance matrix for each GPU, so as to accurately predict the performance of each GPU; then, based on the multidimensional operation performance matrix corresponding to each GPU, a GPU is allocated to each structural layer in the model to be trained for model training to obtain a target model, thereby improving the adaptability of the GPU performance and the structural layer in the model, thereby improving the training efficiency of the model and shortening the training time of the model.

[0585] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, which can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0586] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0587] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A model training method based on a heterogeneous GPU cluster, characterized in that: include: Obtaining hardware indicator data of each GPU in a heterogeneous GPU cluster; the heterogeneous GPU cluster includes at least two heterogeneous GPUs; Obtain multiple operation types corresponding to the deep learning algorithm in the model to be trained, and measure the operation performance data of each GPU executing the operation corresponding to each operation type; Obtain multi-dimensional characteristics of communication bandwidth and load status perception strategy for each GPU; The load state perception strategy characterizes the correlation between the load and performance of each GPU; Constructing a multidimensional operation performance matrix for each GPU based on the hardware indicator data, the operation performance data, the multidimensional characteristics of the communication bandwidth, and the load state perception strategy corresponding to each GPU; Constructing a parallel pipeline group according to multiple structural layers in the to-be-trained model, sequentially traversing a target number of the parallel pipeline groups, and determining an activation probability of the current parallel pipeline group for the traversed current parallel pipeline group; Building a data parallel configuration strategy according to the target number of parallel pipeline groups and the activation probability of the current parallel pipeline group; Determining any structural layer in the to-be-trained model as the current structural layer, and determining any pipeline stage in the current parallel pipeline group as the current pipeline stage; Constructing a layer stage mapping strategy according to the pipeline stage constraints of the current structural layer, the allocation probability of the current structural layer being divided into the current pipeline stage, and the mapping probability of the current pipeline stage being mapped to each GPU in the heterogeneous GPU cluster; According to the multi-dimensional operation performance matrix, the data parallel configuration strategy and the layer stage mapping strategy, the target position of each GPU in the current parallel pipeline group is determined, and a GPU is allocated to each structural layer in the model to be trained for model training to obtain the target model.

2. The method according to claim 1, characterized in that The method of obtaining the multi-dimensional characteristics of the communication bandwidth of each GPU and the load state perception strategy includes: Determine the bandwidth and latency characteristics of each GPU at multiple message size levels; Determine the performance data of each GPU executing multi-process collective operations; Determining the effective bandwidth of each GPU using a preset network technology under multiple load modes; the preset network technology includes at least one of a wireless bandwidth technology and an Ethernet technology; Determine the multi-dimensional characteristics of the communication bandwidth of each GPU based on the bandwidth, latency characteristics, performance data, and effective bandwidth corresponding to each GPU; Determine the load state awareness strategy for each GPU based on the performance of each GPU under different loads.

3. The method according to claim 2, characterized in that Determining the load state perception strategy of each GPU based on the performance of each GPU under different loads includes: Analyze the performance decay curves of each GPU executing multiple operation types at different utilization rates to obtain the load-performance correlation relationship; Determine the correlation between modeling memory usage and computing performance of each GPU; Determine a performance stability analysis result of each GPU under a target load within a target period; the cumulative duration of the target period is greater than or equal to a preset duration, and the target load is greater than or equal to a preset load threshold; A load state perception strategy for each GPU is determined based on the load-performance association relationship, the usage-performance association relationship, and the performance stability analysis result corresponding to each GPU.

4. The method according to claim 1, wherein The method of obtaining multiple operation types corresponding to the deep learning algorithm in the model to be trained and measuring the operation performance data of each GPU executing the operation corresponding to each operation type includes: Obtaining an attention mechanism operation type, a multi-layer perceptron operation type, and a preset operation type corresponding to the deep learning algorithm in the model to be trained; the preset operation type includes at least one of a normalization operation type, an embedding operation type, and an activation operation type; Measuring first operation performance data of each GPU executing an operation corresponding to the attention mechanism operation type; Measuring second operation performance data of each GPU executing an operation corresponding to the multilayer perceptron operation type; Measuring third operation performance data of each GPU executing an operation corresponding to the preset operation type; The operation performance data corresponding to each GPU is determined according to the first operation performance data, the second operation performance data, and the third operation performance data corresponding to each GPU.

5. The method according to claim 4, characterized in that The constructing of a parallel pipeline group according to multiple structural layers in the model to be trained includes: Sequentially number the multiple structural layers in the model to be trained to construct a model layer set; Constructing a plurality of pipeline stage sets in sequence according to the number of each structural layer in the model layer set; each pipeline stage set includes at least two consecutively numbered structural layers; The multiple pipeline stage sets are grouped to obtain a target number of the parallel pipeline groups.

6. The method according to claim 1, wherein The method for determining the target position of each GPU in the current parallel pipeline group includes: Get the batch size and sequence length of sample data; Constructing a batch configuration strategy based on the batch size and sequence length of the sample data; Constructing a training time prediction function according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer-stage mapping strategy, and the batch configuration strategy; Determine a target position of each GPU in the current parallel pipeline group according to the training time prediction function.

7. The method according to claim 6, characterized in that The constructing of a training time prediction function according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer-stage mapping strategy, and the batch configuration strategy includes: Determining a stage combination delay function for each pipeline stage in each parallel pipeline group according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer stage mapping strategy, and the batch configuration strategy; Determining a steady-state critical pipeline stage in each parallel pipeline group based on a stage combination delay function of each pipeline stage in each parallel pipeline group, and determining the stage combination delay of the steady-state critical pipeline stage as a critical path time function; Get the gradient synchronization time function corresponding to each parallel pipeline group; Determine a complete iteration time function of the model to be trained according to the critical path time function and the gradient synchronization time function corresponding to each parallel pipeline group; The training time prediction function is constructed according to the complete iteration time function and the scheduling management time function.

8. The method according to claim 7, characterized in that Determining the stage combination delay function of each pipeline stage in each parallel pipeline group according to the multi-dimensional operation performance matrix, the data parallel configuration strategy, the layer stage mapping strategy, and the batch configuration strategy includes: Determining the normalized performance of each GPU for each operation type based on the multidimensional operation performance matrix, the data parallel configuration strategy, the layer stage mapping strategy, and the batch configuration strategy; Determine, for each parallel pipeline group, a comprehensive load of the current pipeline stage; the comprehensive load includes a forward comprehensive load of each operation type during the forward propagation process and a reverse comprehensive load during the reverse propagation process; determining a forward delay function of the current pipeline stage according to a ratio between the forward integrated load of each operation type in the current pipeline stage and the normalized performance corresponding to each operation type; determining a reverse delay function for the current pipeline stage according to a ratio between the reverse integrated load of each operation type in the current pipeline stage and the normalized performance of each operation type; Determine a stage combination delay function of the current pipeline stage according to the forward delay function and the reverse delay function corresponding to the current pipeline stage.

9. The method according to claim 8, characterized in that Determining the comprehensive load of the current pipeline stage includes: Obtaining a first load corresponding to the attention mechanism operation type in the current pipeline stage; Obtaining a second load corresponding to the multilayer perceptron operation type in the current pipeline stage; Obtaining a third load corresponding to a preset operation type in the current pipeline stage; the preset operation type includes at least one of a normalization operation type, an embedding operation type, and an activation operation type; The comprehensive load of the current pipeline stage is determined according to the first load, the second load, and the third load corresponding to the current pipeline stage.

10. The method according to claim 8, characterized in that Determining the stage combined delay function of the current pipeline stage according to the forward delay function and the reverse delay function corresponding to the current pipeline stage includes: Determining a non-overlapping communication residual time function between the current pipeline stage and an adjacent pipeline stage based on the data volume of each GPU in the current pipeline stage and the ratio of the bandwidth between each GPU and the communication GPU; Determining a stage combined delay function of the current pipeline stage according to the forward delay function, the reverse delay function, and the communication residual time function corresponding to the current pipeline stage; Determining a target position of each GPU in the current parallel pipeline group according to the training time prediction function includes: Obtain GPU mapping uniqueness constraints, video memory capacity constraints, computing performance constraints, communication bandwidth constraints, and critical path constraints to obtain strategy constraints; Obtaining, based on the strategy constraints, the training time prediction function, and the hybrid parallel simulator, the training time corresponding to different positions of each GPU in the current parallel pipeline group; The position where the training time corresponding to each GPU meets a preset condition is determined as the target position of each GPU in the current parallel pipeline group.

11. The method according to claim 10, characterized in that The method for constructing the video memory capacity constraint condition includes: For the current pipeline stage in each parallel pipeline group, obtain the model parameter memory of the current structure layer and the activation data memory generated during the forward propagation process; Determining a target required parameter memory of the current pipeline stage according to an allocation probability of each current structural layer in the to-be-trained model being divided into the current pipeline stage and a model parameter memory of each current structural layer; Determining a target demand activation memory function of the current pipeline stage according to an allocation probability of each current structural layer in the to-be-trained model being divided into the current pipeline stage and an activation data memory corresponding to each current structural layer; The video memory capacity constraint condition is constructed according to the target requirement parameter memory and the target requirement activation memory function.

12. The method according to claim 11, characterized in that Determining a target position of each GPU in the current parallel pipeline group according to the training time prediction function includes: Obtaining, according to the stage combination delay function of the current pipeline stage, stage combination delay data corresponding to different positions of each GPU in the current parallel pipeline group; Calculating the pipeline parallel voiding rate and the data parallel voiding rate corresponding to different positions of each GPU in the current parallel pipeline group; Determine the training time of each GPU at different positions in the current parallel pipeline group based on the stage combination delay data corresponding to each GPU at different positions in the current parallel pipeline group, the pipeline parallel voiding rate, the data parallel voiding rate, and the training time prediction function; Determine a target position of each GPU in the current parallel pipeline group according to the training time of each GPU at different positions in the current parallel pipeline group.

13. A model training method based on a heterogeneous GPU cluster, characterized in that: include: Obtaining hardware indicator data of each GPU in a heterogeneous GPU cluster; the heterogeneous GPU cluster includes at least two heterogeneous GPUs; Obtain multiple operation types corresponding to the deep learning algorithm in the model to be trained, and measure the operation performance data of each GPU executing the operation corresponding to each operation type; Obtain multi-dimensional characteristics of communication bandwidth and load status perception strategy for each GPU; The load state perception strategy characterizes the correlation between the load and performance of each GPU; Constructing a multidimensional operation performance matrix for each GPU based on the hardware indicator data, the operation performance data, the multidimensional characteristics of the communication bandwidth, and the load state perception strategy corresponding to each GPU; Allocate a GPU to each structural layer in the to-be-trained model according to the multi-dimensional operation performance matrix corresponding to each GPU to obtain a GPU scheduling strategy; Determine the sample computation cost of each sample data according to the sequence length, floating-point operation amount, or historical execution time corresponding to similar sample data of each sample data in the sample data set; Determine the performance data of each GPU based on the current load status of each GPU; Construct sample allocation constraints based on the sample allocation probability of each sample data being assigned to each GPU; A load balancing constraint condition is constructed based on the sample calculation cost of each sample data, the sample allocation probability corresponding to each GPU, and the performance data; a comprehensive constraint condition is determined based on the sample allocation constraint condition and the load balancing constraint condition; a target time function is constructed based on the sample allocation probability corresponding to each GPU and the performance data; a GPU corresponding to each sample data is determined based on the comprehensive constraint condition and the target time function, and a sample allocation strategy is obtained; According to the GPU scheduling strategy and the sample allocation strategy, the model to be trained is trained to obtain a target model.

14. A model training device based on a heterogeneous GPU cluster, characterized in that: include: A hardware data acquisition module is configured to acquire hardware indicator data of each GPU in a heterogeneous GPU cluster; the heterogeneous GPU cluster includes at least two heterogeneous GPUs; An operation performance data acquisition module is configured to acquire multiple operation types corresponding to the deep learning algorithm in the model to be trained and measure the operation performance data of each GPU executing the operation corresponding to each operation type; A load strategy acquisition module is configured to execute acquisition of multi-dimensional characteristics of the communication bandwidth of each GPU and a load state perception strategy; the load state perception strategy characterizes the correlation between the load and performance of each GPU; a performance matrix construction module configured to construct a multidimensional operation performance matrix for each GPU based on the hardware indicator data, the operation performance data, the multidimensional characteristics of the communication bandwidth, and the load state perception strategy corresponding to each GPU; The model training module is configured to construct a parallel pipeline group according to multiple structural layers in the model to be trained, sequentially traverse a target number of the parallel pipeline groups, and determine an activation probability of the current parallel pipeline group for the traversed parallel pipeline group; Building a data parallel configuration strategy according to the target number of parallel pipeline groups and the activation probability of the current parallel pipeline group; Determining any structural layer in the to-be-trained model as the current structural layer, and determining any pipeline stage in the current parallel pipeline group as the current pipeline stage; Constructing a layer stage mapping strategy according to the pipeline stage constraints of the current structural layer, the allocation probability of the current structural layer being divided into the current pipeline stage, and the mapping probability of the current pipeline stage being mapped to each GPU in the heterogeneous GPU cluster; According to the multi-dimensional operation performance matrix, the data parallel configuration strategy and the layer stage mapping strategy, the target position of each GPU in the current parallel pipeline group is determined, and a GPU is allocated to each structural layer in the model to be trained for model training to obtain the target model.

15. A model training device based on a heterogeneous GPU cluster, characterized in that: include: A hardware data acquisition module is configured to acquire hardware indicator data of each GPU in a heterogeneous GPU cluster; the heterogeneous GPU cluster includes at least two heterogeneous GPUs; An operation performance data acquisition module is configured to acquire multiple operation types corresponding to the deep learning algorithm in the model to be trained and measure the operation performance data of each GPU executing the operation corresponding to each operation type; A load strategy acquisition module is configured to execute acquisition of multi-dimensional characteristics of the communication bandwidth of each GPU and a load state perception strategy; the load state perception strategy characterizes the correlation between the load and performance of each GPU; a performance matrix construction module configured to construct a multidimensional operation performance matrix for each GPU based on the hardware indicator data, the operation performance data, the multidimensional characteristics of the communication bandwidth, and the load state perception strategy corresponding to each GPU; The model training module is configured to allocate a GPU to each structural layer in the to-be-trained model according to a multi-dimensional operation performance matrix corresponding to each GPU, and obtain a GPU scheduling strategy; Determine the sample computation cost of each sample data based on the sequence length, floating-point operations, or historical execution time of similar sample data of each sample data in the sample data set; determine the performance data of each GPU based on the current load status of each GPU; and construct sample allocation constraints based on the sample allocation probability of each sample data to each GPU. A load balancing constraint is constructed based on the sample calculation cost of each sample data, the sample allocation probability corresponding to each GPU, and the performance data; a comprehensive constraint is determined based on the sample allocation constraint and the load balancing constraint; a target time function is constructed based on the sample allocation probability and performance data corresponding to each GPU; based on the comprehensive constraint and the target time function, the GPU corresponding to each sample data is determined to obtain a sample allocation strategy; based on the GPU scheduling strategy and the sample allocation strategy, the model to be trained is trained to obtain a target model.

16. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the model training method based on a heterogeneous GPU cluster as described in any one of claims 1-13.

17. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by an electronic device processor, the electronic device is enabled to execute the model training method based on a heterogeneous GPU cluster as described in any one of claims 1 to 13.

18. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by the processor, the model training method based on a heterogeneous GPU cluster described in any one of claims 1 to 13 is implemented.

Images