Adaptive data parallel optimization distributed training method and system for heterogeneous environment

Abstract

The application discloses a heterogeneous environment adaptive data parallel optimization distributed training method and system, which firstly builds a hybrid topology architecture composed of a parameter server and multiple computing devices, establishes a training time model, and comprehensively considers three types of time delays, namely data transmission, local calculation and parameter communication. Secondly, based on the training time model, a scheduling optimization function is defined to minimize the training bottleneck. Then, a relaxation variable and an approximation method are introduced, the original scheduling optimization function is equivalently converted into a convex optimization problem, and a corresponding relaxation function is constructed. Finally, based on the relaxation function, an enhanced Lagrange function is constructed, and the distributed solution under the constraint condition is realized through the Lagrange multiplier method, and finally the training scheduling strategy of asymptotic optimization is realized. The application ensures the load balancing between computing devices with different capabilities, significantly reduces the communication overhead, thereby accelerating the training process and improving the overall efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of machine learning, and particularly relates to a self-adaptive data parallel optimization distributed training method and system for a heterogeneous environment. BACKGROUND

[0002] The rapid development of Internet applications has led to an exponential increase in global data volume, with recent data generation exceeding the total of all previous data. The explosive growth of big data is driving rapid development in related fields. Deep learning models, from simple neural networks to large language models (LLM) containing hundreds of billions of parameters, therefore pose an increasingly urgent need for effective training of large-scale learning models.

[0003] To address this challenge, distributed training has emerged. Its core concept is to use distributed nodes to decompose and distribute computing tasks, achieving collaborative completion to effectively overcome the limitations of single-machine processing power. Data parallelism, as a key form of distributed computing, has been widely used in the training of large-scale learning models. By splitting and distributing training data to each computing node, data parallelism can efficiently train complex models. In the field of computer vision, data parallelism enables more efficient training of image classification and object detection tasks, and improves model accuracy by extracting rich visual features. In natural language processing (NLP), data parallelism ensures the processing of large-scale text data, helping models capture complex semantic and syntactic relationships, which drives continuous progress in NLP technology.

[0004] Although data parallelism brings significant benefits in improving computing efficiency, it faces many challenges in practical applications. First, distributed computing can result in significant communication overhead. In the data distribution phase, the transmission of large amounts of training data is limited by network bandwidth and latency, and during the training process, frequent gradient or parameter synchronization further increases communication costs. Second, the heterogeneity of computing devices can lead to uneven data distribution and load imbalance, as popular distributed training mechanisms such as Distributed Data Parallel (DDP) assume uniform data distribution. Although frameworks like DeepSpeed have improved memory management, there are still shortcomings in data balanced allocation. Finally, most distributed training frameworks, such as DDP and DeepSpeed, are primarily designed for homogeneous network conditions in co-located clusters, and therefore cannot handle geographically distributed clusters in heterogeneous network environments. SUMMARY

[0005] In view of the limitations of existing methods, the application develops a self-adaptive data parallel optimization distributed training method and system (ADP) for a heterogeneous environment, which realizes data parallel scheduling in a geographically distributed cluster that optimizes the consideration of computing and communication heterogeneity.

[0006] The proposed adaptive data-parallel optimization distributed training method for heterogeneous environments has the following specific steps:

[0007] S1. Build a hybrid topology architecture consisting of a parameter server and multiple computing devices, supporting heterogeneous computing resources such as GPUs, CPUs, and edge devices. The system adapts to wide area network and distributed deployment scenarios based on the computing power of each device and the network bandwidth and latency between the device and the server, providing system support for subsequent training scheduling.

[0008] S2. Based on the above system architecture, a unified training time model is established, comprehensively considering three types of latency: data distribution, local computation, and parameter communication. The model takes device computing power and network characteristics as input, dynamically evaluating the time required by each device at each stage of each training round, and quantifying the performance differences of the system in a heterogeneous environment.

[0009] S3. Based on the training time model, define a scheduling optimization function with the objective of minimizing the training bottleneck. The optimization function includes two constraints: the total amount of data allocated to all computing devices (which is an integer) is equal to the total amount of data required for training, and the maximum parallel processing capability of each computing device is not less than the number of training batches for that device.

[0010] S4. Since the original scheduling optimization function has non-convex characteristics and is not easy to solve directly, this invention introduces slack variables and approximation methods to transform it into an equivalent convex optimization problem and construct the corresponding slack function.

[0011] S5. Based on the relaxation function, an enhanced Lagrange function is constructed, and distributed solution under constraints is achieved through the Lagrange multiplier method. Combined with a data parallel scheduling algorithm, the data partitioning and synchronization frequency are dynamically updated according to the device status in each training round to ensure load balancing and optimal resource utilization among devices with different performance levels, ultimately achieving an asymptotically optimal training scheduling strategy.

[0012] The specific method for step S1 is as follows:

[0013] Construct a distributed hybrid topology architecture, including a parameter server (PS) for the data owner and... A computing device. PS will use a subset of data. Send to each participating computing device Each device Use stochastic gradient descent (SGD) on the received subset of data Perform local training, and then upload the locally updated model parameters based on the current situation. PS performs a weighted average based on the local model parameters of all devices to update the global model. and distribute it to all devices.

[0014] The step S2 is specifically as follows:

[0015] The parameter server is taken as device 0, and the product of the data subset and the size of the sample is divided by the transmission bandwidth to calculate the distribution time of each sample from the parameter server to the computing device . .

[0016] In distributed training, the computing time of each device is related to the amount of data allocated to it, and the maximum parallel capability of the device is represented as , that is, the maximum number of samples processed by the device at the same time, and the maximum parallel capability must not be less than the batch of the device , so that the computing time of the device in a cycle is .

[0017] Each computing device receives the current global model parameters from the PS, updates its local model using the allocated data subset, and sends the updated model parameters to the PS at the end of each epoch. Let denote the size of the model parameters. Therefore, the communication time of the device for data parallelism in an epoch is denoted as , and the training time model is constructed as follows:

[0018] (1)

[0019] The step S3 is specifically as follows:

[0020] Based on the distribution time , the computing time and the communication time , in order to minimize the total time of the data parallel training process, the present application aims to optimize the data allocation . Therefore, the data parallel scheduling optimization problem is expressed as:

[0021] (2)

[0022] The step S4 is specifically as follows:

[0023] In order to simplify the symbols, the above optimization model formula (2) is re-expressed as follows:

[0024] (3)

[0025] Note that it is independent of the number of variables , the communication bandwidth and the model size , and thus is a constant.

[0026] where is a diagonal matrix , is a diagonal matrix , is a diagonal matrix , is a diagonal matrix , is a diagonal matrix , and is a constant.

[0027] The function is approximated by the -norm, and thus the objective function can be rewritten as

[0028] (4)

[0029] whose feasible region is .

[0030] and the present invention defines the relaxed function as

[0031] (5)

[0032] whose feasible region is .

[0033] The step S5 is specifically as follows:

[0034] Since the relaxed function is convex in its feasible region, the global optimal solution of the relaxed function is first solved by using the augmented Lagrangian function , and then in the neighborhood of , i.e. , the can be found such that . It should be noted that is independent of the total number of data samples . Therefore, for the optimal solution of the original equation , there is , because is positively correlated with , so is obtained. That is, when the number of training data samples is large enough, the optimal solution can be found in an asymptotic sense. Therefore, the present invention constructs an augmented Lagrangian function as follows:

[0035] (6)

[0036] where is the Lagrange multiplier for the equality constraint, is the coefficient of the quadratic penalty term, is the coefficient of the logarithmic barrier to ensure The optimal solution , i.e., the optimal training scheduling strategy, is obtained by iteratively searching for the enhanced Lagrangian function using an optimization algorithm.

[0037] The present application also provides an adaptive data parallel optimization distributed training system for a heterogeneous environment, comprising the following modules:

[0038] A mixed topology architecture module is used to build a mixed topology architecture composed of a parameter server and multiple computing devices.

[0039] A training time module is used to establish a unified training time model based on the mixed topology architecture, comprehensively consider three types of time delay, i.e., data distribution, local computation and parameter communication, and define a scheduling optimization function based on the training time model, with the goal of minimizing the training bottleneck.

[0040] A relaxation function module is used to introduce relaxation variables and approximation methods, equivalently convert the original scheduling optimization function into a convex optimization problem, construct a corresponding relaxation function and enhanced Lagrangian function, realize distributed solving under the constraint condition through the Lagrange multiplier method, and finally realize the asymptotically optimal training scheduling strategy.

[0041] The present application has the following advantages:

[0042] The present application realizes more efficient and robust distributed training through the following design. (1) Delayed parameter synchronization. By adopting local training and performing delayed parameter synchronization at the end of each round instead of at the end of each batch, the present application greatly reduces the communication overhead. (2) Data parallel scheduling formalization. The present application constructs a data parallel scheduling optimization problem, and minimizes the training time by optimizing data distribution. (3) Efficient algorithm. The present application develops a very efficient algorithm to solve the optimization problem, and can prove that the obtained solution is optimal in the asymptotic sense. By strategically allocating data, the present application ensures load balancing between computing devices with different capabilities, and significantly reduces communication overhead, thereby accelerating the training process and improving overall efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The workflow of the optimal distributed training framework based on adaptive data parallelism;

[0044] Figure 2To test the accuracy and loss change of the application and single machine in specific cases;

[0045] Figure 3 The training loss curve of Wide-ResNet101 on heterogeneous devices;

[0046] Figure 4 The accuracy and loss curve of different training methods in the time dimension. DETAILED DESCRIPTION

[0047] The application will be described in detail below in conjunction with the accompanying Figure 1 drawings. The adaptive data parallel optimization distributed training system in a heterogeneous environment is as follows:

[0048] Step S1, the application constructs a distributed hybrid topology architecture, including a parameter server (PS) of a data owner and a plurality of computing devices, such as Figure 1 as shown. The PS sends a data subset to each participating computing device . Each device uses stochastic gradient descent (SGD) to perform local training on the received data subset , and then uploads the locally updated model parameters according to the current situation. The PS performs weighted averaging according to the local model parameters of all devices to update the global model and distribute it to all devices.

[0049] where is the total number of data samples. The total amount of data samples is constrained as:

[0050] (1)

[0051] where is the total amount of data allocated to the device , and is the total amount of data (the entire data set).

[0052] Step S2, for each sample, the distribution time from the parameter server (device 0) to the computing device can be expressed as:

[0053] (2)

[0054] where is the transmission bandwidth of the device , and is the size of each sample.

[0055] According to the computing power of the heterogeneous device, a training time model is constructed, and the specific method is as follows:

[0056] In distributed training, the computing time of each device is related to the amount of data allocated to it, that is, the larger the data set, the longer the computing time. The maximum parallel capability of the device is represented as , that is, the maximum number of samples that the device can process at the same time, then it can be obtained:

[0057] (3)

[0058] The maximum parallel capability must not be less than the batch of the device , that is:

[0059] (4)

[0060] Therefore, the computing time of the device in a cycle is represented as:

[0061] (5)

[0062] Where is the memory capacity of the device , is the amount of data processed by the device each iteration, represents the time required by the device to process a single data batch, including forward propagation and backward propagation calculation, is the sample size of the input model after preprocessing, is the memory consumption of the activation value increased by adding a sample.

[0063] Each computing device receives the current global model parameters from the PS, updates its local model using the allocated data subset, and sends the updated model parameters to the PS at the end of each epoch. Let represent the size of the model parameters. Therefore, the communication time of the device for data parallelism in an epoch is represented as:

[0064] (6)

[0065] Where is the communication delay.

[0066] The training time model is constructed as follows:

[0067] ​​ .

[0068] Step S3, in order to minimize the total time of data parallel training process, the present application aims to optimize data allocation Therefore, the data parallel scheduling optimization problem is formulated as:

[0069] (7)

[0070] Step S4, in order to simplify the symbols, the above optimization model formula (7) is re-expressed as follows:

[0071] (8)

[0072] Note that is irrelevant to variable , it depends on the communication bandwidth and the model size , so it is a constant.

[0073] where is a matrix with diagonal , is a matrix with diagonal , is a matrix with diagonal , is a constant.

[0074] By using order norm approximation function, therefore when is large, the objective function in equation (8) can be rewritten as

[0075] (9)

[0076] The feasible region of which is: .

[0077] And the present application defines the relaxation function as:

[0078] (10)

[0079] The feasible region of which is: .

[0080] Step S5, since the relaxation function (10) is a convex function in its feasible region, first use the augmented Lagrangian function to find the global optimal solution of the relaxation function (10) , then in the neighborhood of , that is You can find , making It is important to note that With the total number of data samples It is irrelevant. Therefore, for the optimal solution of the original equation (9) ,have ,because and They are positively correlated, therefore we conclude In other words, when the number of training data samples is large enough, the optimal solution can be found asymptotically. Therefore, this invention constructs an enhanced Lagrangian function as follows:

[0081] (11)

[0082] in These are Lagrange multipliers for equality constraints. It is the coefficient of the quadratic penalty term. It is used to ensure The coefficient of the logarithmic barrier. An iterative search for the enhanced Lagrangian function is performed using a data-parallel scheduling algorithm to obtain the optimal solution. That is, the optimal training scheduling strategy

[0083] The process steps of the data parallel scheduling algorithm are described as follows:

[0084] (1): Variable Initialize to Set the iteration counter and inner iteration counter All are 0. Initialize the Lagrange multipliers. Log barrier coefficient The learning rate .

[0085] (2): Enter the outer iteration, when the outer iteration number is... Less than or equal to the maximum number of outer loops At that time, perform the following steps:

[0086] (3): Start the inner iteration and initialize the inner iteration counter. When the inner iteration count Less than or equal to the maximum number of inner loops When this happens, perform the following operations:

[0087] (4): Calculate the gradient, calculate the objective function with respect to... gradient .

[0088] (5): According to the gradient descent method, use the learning rate renew : .

[0089] (6): Calculate the norm of the gradient, if the 2-norm of the gradient norm is less than the convergence threshold , then jump out of the inner loop, end the inner iteration.

[0090] (7): Inner loop counter update, if the convergence condition is not met, the inner iteration counter , continue the inner loop.

[0091] (8): Calculate the new Lagrange multiplier through the current inner loop updated , the specific update formula is: , wherein is the constraint term.

[0092] (9): According to the attenuation coefficient , update : .

[0093] (10): Outer gradient convergence judgment, calculate the norm of the gradient of the current Lagrange multiplier, if the 2-norm is less than the threshold , then jump out of the outer loop.

[0094] (11): Outer loop counter update, if the outer loop convergence condition is not met, the outer loop iteration counter is added by 1, and the outer loop is continued.

[0095] (12): Final result processing, assign the final iteration obtained to . Round to get the final result .

[0096] The application also provides an adaptive data parallel optimization distributed training system in a heterogeneous environment, comprising the following modules:

[0097] A mixed topology architecture module is used to build a mixed topology architecture composed of a parameter server and a plurality of computing devices, supporting GPU, CPU and edge devices and other heterogeneous computing resources. The system adapts to wide area network and distributed deployment scene according to the computing power of each device and the network bandwidth, delay and other states between the server.

[0098] Training time module: based on the above architecture, a unified training time model is established, and three types of time delay, data distribution, local calculation and parameter communication, are comprehensively considered. On the basis of the training time model, a scheduling optimization function is defined to minimize the training bottleneck. The optimization function includes two constraint conditions: the total amount of data allocated to all computing devices (belongs to an integer) is equal to the total amount of data required for training, and the maximum parallel processing capacity of each computing device is not less than the training batch number of its device.

[0099] Relaxation function module: since the original scheduling optimization function has non-convex characteristics and is not easy to solve directly, the present application introduces relaxation variables and approximation methods, converts it into a convex optimization problem, and constructs the corresponding relaxation function. On the basis of the relaxation function, an enhanced Lagrange function is constructed, and the distributed solution under the constraint condition is realized through the Lagrange multiplier method. Combined with the data parallel scheduling algorithm, the data partition and synchronization frequency are dynamically updated according to the device state in each round of training, ensuring the load balancing and resource utilization optimization among different performance devices, and finally realizing the asymptotically optimal training scheduling strategy.

[0100] The present application adopts the widely recognized Wide ResNet101 model and ImageNet100 data set for experiments. The data set covers 100 different categories, of which the training set contains 100,000 samples, and the test set consists of 10,000 samples. The present application architecture belongs to the PS architecture, in order to verify the performance of the application in a heterogeneous environment, the training configuration is designed as follows: rank0 is responsible for data sending and parameter synchronization; rank1 to rank4 are equipped with RTX3090 graphics cards (24GB video memory); rank5 and rank6 use 4-core pure CPUs with 8GB of memory; rank7 is equipped with a 16-core CPU with 32GB of memory; and rank8 is an 8-core CPU with 16GB of memory. All devices are distributed in different data centers, fully reflecting the training configuration characteristics in a heterogeneous environment. For WideResNet101, the batch size batchsize is 64. For CPU, the computing power is limited, and the 4-core is set to 8 and the 16-core is set to 16. The above settings ensure that the video memory or CPU can run and will not exceed the maximum computing power and video memory of the device. The maximum epoch is set to 50, and the initial learning rate of the device is adjusted based on batchsize=64 and learning rate 0.01 as the benchmark. The learning rate is adjusted according to the size of the batchsize, and the maximum setting is 0.3. In order to better converge, the present application sets the learning rate to decay to 0.5 of the previous one every 10 epochs. In order to compare fairly, the experiments all use the above initialization method. The present application and single machine test accuracy and loss change in specific cases as shown in Figure 2

[0101] ​Compared with mainstream training frameworks DDP and deep speed, the experimental results are shown in Tables 1 and 2. On the ImageNet100 dataset, the present application achieves fast convergence in a heterogeneous distributed training environment. Compared with distributed data parallel (DDP) and DeepSpeed, the present application achieves 21.6 times and 26.3 times improvement in the number of floating point operations, respectively, and the training time is shortened by about 72% and 47%, respectively. This distributed computing scheme can realize the computing power scheduling of idle GPU and CPU resources across regions, and fully release the potential of global distributed computing nodes. The training loss curve of Wide-ResNet101 on heterogeneous devices is shown in Figure 3 The accuracy and loss curves of different training methods in the time dimension are shown in Figure 4

[0102] Table 1 Accuracy and validation loss results of the present application (8 devices) and single-card training under interval period conditions

[0103]

[0104] Table 2 Comparison of training results of different training methods

[0105] ​

Claims

1. A method for adaptive data parallel optimization distributed training of heterogeneous environments, characterized in that, The method comprises the following steps: S1, a hybrid topology architecture composed of a parameter server and multiple computing devices is built; S2, based on the hybrid topology architecture, a unified training time model is established, and three types of time delays, including data distribution, local calculation and parameter communication, are comprehensively considered, and the specific implementation is as follows: The parameter server is taken as device 0, the data subset is divided by the product of the size of the sample and the transmission bandwidth, and the distribution time of each sample from the parameter server to the computing device is calculated represents ; In distributed training, the computation time of each device is related to the amount of data assigned to it, with devices being assigned more data than devices . The maximum parallel capability of a device is denoted as , i.e., the maximum number of samples that the device can process simultaneously, and the maximum parallel capability must not be smaller than the batch size of the device , resulting in a computation time of the device in one cycle. Each computing device receives current global model parameters from the parameter server PS, updates its local model using the assigned data subset, and sends the updated model parameters to the parameter server PS at the end of each epoch; with denoting the size of the model parameters; the device communication time for data parallelism within an epoch ; Constructing a training time model: ; S3, based on the training time model, a scheduling optimization function aiming at minimizing the training bottleneck is defined; The scheduling optimization function comprises two constraint conditions: the total amount of data allocated to all computing devices is equal to the total amount of data required for training, and the maximum parallel processing capacity of each computing device is not less than the training batch number of the device, and the specific implementation is as follows: Based on distribution time , computation time , communication time , minimizing the total time of the data parallel training process, optimizing data allocation ; data parallel scheduling optimization problem formulated as ; S4, a relaxation variable and an approximation method are introduced, the original scheduling optimization function is equivalently converted into a convex optimization problem, and a corresponding relaxation function is constructed; S5, based on the relaxation function, an enhanced Lagrange function is constructed, distributed solving under the constraint condition is realized through the Lagrange multiplier method, and finally an asymptotically optimal training scheduling strategy is realized.

2. The adaptive data parallel optimization distributed training method of heterogeneous environment according to claim 1, characterized in that, The step S1 is specifically: constructing a distributed hybrid topology architecture, including a parameter server PS of a data owner and a plurality of computing devices, the parameter server PS sends a data subset , to each participating computing device ; each device uses stochastic gradient descent to locally train the received data subset , and then uploads the locally updated model parameters according to the current situation ; the parameter server PS performs weighted averaging according to the local model parameters of all devices to update the global model and distribute it to all devices.

3. The method of claim 2, wherein, The step S4 is specifically implemented as follows: The data parallel scheduling optimization problem is restructured as: ， ； wherein is a diagonal matrix with is a diagonal matrix with is the size of each sample, is a diagonal matrix with is a diagonal matrix with is a diagonal matrix with is the amount of data processed by the device per iteration, is a constant;​​​​ Utilizing norm approximation function, function rewritten as: whose feasible region is: ; The relaxed function is defined as: with feasible region: .

4. The method of claim 3, wherein, The step S5 is specifically implemented as follows: The relaxation function is a convex function in its feasible region, the global optimal solution of the relaxation function is solved by using the enhanced Lagrange function , then in the neighborhood of , find , so that , , which is independent of the total number of data samples , for the function , the optimal solution of the rewritten equation , has , , and , is positively correlated, and , an enhanced Lagrange function is constructed as follows: ； in These are Lagrange multipliers for equality constraints. It is the coefficient of the quadratic penalty term. It is used to ensure The coefficient of the logarithmic barrier; an optimization algorithm is used to iteratively search for the enhanced Lagrangian function to obtain the optimal solution. That is, the optimal training scheduling strategy.

5. An adaptive data parallel optimization distributed training system of heterogeneous environments, for implementing the adaptive data parallel optimization distributed training method of any one of claims 1 to 4, characterized in that, The method comprises the following modules: A hybrid topology architecture module is used to build a hybrid topology architecture composed of a parameter server and multiple computing devices; A training time module is used to establish a unified training time model based on the hybrid topology architecture, to comprehensively consider three types of time delays, including data distribution, local calculation and parameter communication, and to define a scheduling optimization function aiming at minimizing the training bottleneck based on the training time model; A relaxation function module is used to introduce a relaxation variable and an approximation method, to equivalently convert an original scheduling optimization function into a convex optimization problem, to construct a corresponding relaxation function and an enhanced Lagrange function, to realize distributed solving under a constraint condition through a Lagrange multiplier method, and to finally realize an asymptotically optimal training scheduling strategy.

Images