Adaptive data parallel optimization distributed training method and system in heterogeneous environment

Abstract

The invention discloses a self-adaptive data parallel optimization distributed training method and system in a heterogeneous environment. The method comprises the following steps: firstly, building a mixed topology framework consisting of a parameter server and a plurality of computing devices, establishing a training time model, and comprehensively considering three types of time delays of data transmission, local computing and parameter communication; secondly, on the basis of the training time model, defining a scheduling optimization function aiming at minimizing a training bottleneck; secondly, a slack variable and an approximation method are introduced, an original scheduling optimization function is equivalently converted into a convex optimization problem, and a corresponding slack function is constructed; and finally, on the basis of the relaxation function, constructing an enhanced Lagrangian function, realizing distributed solution under constraint conditions through a Lagrangian multiplier method, and finally realizing an asymptotically optimal training scheduling strategy. According to the method, the load balance among the computing devices with different capabilities is ensured, and the communication overhead is remarkably reduced, so that the training process is accelerated, and the overall efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning, and in particular relates to an adaptive data parallel optimization distributed training method and system for heterogeneous environments. Background Art

[0002] The rapid development of internet applications has led to an exponential increase in global data volumes, with recent data generation exceeding all previous historical totals. This explosive growth of big data is driving rapid development in related fields. Deep learning models, ranging from simple neural networks to large language models (LLMs) containing tens of billions of parameters, are creating an increasingly urgent need for efficient training of these large-scale learning models.

[0003] To address this challenge, distributed training emerged. Its core concept is to leverage distributed nodes to decompose and distribute computing tasks, enabling collaborative completion and effectively overcoming the limitations of single-machine processing capabilities. Data parallelism, a key form of distributed computing, has been widely used in the training of large-scale learning models. By splitting and distributing training data across various computing nodes, data parallelism enables efficient training of complex models. In computer vision, data parallelism makes training for tasks such as image classification and object detection more efficient, and improves model accuracy by extracting rich visual features. In natural language processing (NLP), data parallelism ensures the processing of large amounts of text data and helps models capture complex semantic and syntactic relationships, which has driven the continued advancement of NLP technology.

[0004] Although data parallelism offers significant benefits in improving computational efficiency, it faces numerous challenges in practical applications. First, distributed computing can incur significant communication overhead. During the data distribution phase, the transmission of large amounts of training data is limited by network bandwidth and latency, while frequent gradient or parameter synchronization during training further increases communication costs. Second, the heterogeneity of computing devices can lead to uneven data distribution and unbalanced load, as popular distributed training mechanisms such as distributed data parallelism (DDP) assume a uniform data distribution. While frameworks like DeepSpeed ​​(a distributed training method) have improved memory management, they still struggle with balanced data allocation. Finally, most distributed training frameworks, such as DDP and DeepSpeed, are primarily designed for co-located clusters with homogeneous network conditions and are therefore unable to cope with geographically distributed clusters in heterogeneous network environments. Summary of the Invention

[0005] In response to the limitations of existing methods, the present invention develops an adaptive data parallel optimization distributed training method and system (ADP) for heterogeneous environments, which realizes data parallel scheduling by optimizing geographically distributed clusters that take into account computing and communication heterogeneity.

[0006] The proposed adaptive data parallel optimization distributed training method for heterogeneous environments has the following specific steps:

[0007] S1. Build a hybrid topology consisting of a parameter server and multiple computing devices, supporting heterogeneous computing resources such as GPUs, CPUs, and edge devices. Based on the computing power of each device and the network bandwidth and latency between the server and the server, the system adapts to wide-area networks and distributed deployment scenarios, providing system support for subsequent training scheduling.

[0008] S2. Based on the above system architecture, a unified training time model is established, comprehensively considering three types of latency: data distribution, local computation, and parameter communication. This model uses device computing power and network characteristics as inputs to dynamically evaluate the time required for each device at each stage of each training round, quantifying the performance differences of the system in heterogeneous environments.

[0009] S3. Based on the training time model, define a scheduling optimization function that aims to minimize training bottlenecks. This optimization function includes two constraints: the total amount of data (an integer) allocated to all computing devices is equal to the total amount of data required for training, and the maximum parallel processing capability of each computing device is no less than the number of training batches for that device.

[0010] S4. Since the original scheduling optimization function has non-convex characteristics and is not easy to solve directly, the present invention introduces relaxation variables and approximation methods to convert it into a convex optimization problem and construct the corresponding relaxation function.

[0011] S5. Based on the relaxation function, an enhanced Lagrangian function is constructed, and the Lagrangian multiplier method is used to achieve distributed solution under constraints. Combined with the data parallel scheduling algorithm, the data partitioning and synchronization frequency are dynamically updated according to the device status in each training round, ensuring load balancing and optimal resource utilization among devices with different performance, ultimately achieving an asymptotically optimal training scheduling strategy.

[0012] The specific method of step S1 is as follows:

[0013] Build a distributed hybrid topology architecture, including the parameter server (PS) of the data owner and computing devices. PS subsets the data Sent to each participating computing device Each device Use stochastic gradient descent (SGD) on the received data subset Perform local training and then upload the locally updated model parameters based on the current situation PS performs weighted averaging based on the local model parameters of all devices to update the global model and distribute it to all devices.

[0014] The specific method of step S2 is as follows:

[0015] Set the parameter server as device 0, divide the product of the data subset and the sample size by the transmission bandwidth, and calculate the distribution of each sample from the parameter server to the computing device The distribution time of .

[0016] In distributed training, the computation time of each device is related to the amount of data assigned to it. The maximum parallel capability is expressed as , that is, the maximum number of samples that the device can process simultaneously, and the maximum parallel capability Not less than the equipment Batch, get equipment Computation time in one cycle .

[0017] Each computing device Receives the current global model parameters from the PS, updates its local model using the assigned data subset, and sends the updated model parameters to the PS at the end of each epoch. represents the size of the model parameters. Therefore, the device The communication time used for data parallelism within an epoch is denoted as , construct the training time model as follows:

[0018] (1)

[0019] The specific method of step S3 is as follows:

[0020] Based on distribution time , calculation time and communication time In order to minimize the total time of the data parallel training process, the present invention aims to optimize the data distribution Therefore, the data parallel scheduling optimization problem is expressed as:

[0021] (2)

[0022] The specific method of step S4 is as follows:

[0023] In order to simplify the notation, the above optimization model formula (2) is reformulated as follows:

[0024] (3)

[0025] Notice With variables It doesn't matter, it depends on the communication bandwidth and model size , and is therefore a constant.

[0026] in The diagonal is of matrix, The diagonal is of matrix, The diagonal is of matrix, is a constant.

[0027] use Level norm approximation function, so in When it is large, the objective function can be rewritten as:

[0028] (4)

[0029] Its feasible domain is .

[0030] And the present invention defines the relaxation function as:

[0031] (5)

[0032] Its feasible domain is: .

[0033] The specific method of step S5 is as follows:

[0034] Since the relaxation function is a convex function in its feasible domain, the global optimal solution of the relaxation function is first obtained using the enhanced Lagrangian function. , then In the neighborhood of , you can find , making It should be noted that The total number of data samples Therefore, for the original equation The optimal solution ,have ,because and There is a positive correlation, so we can conclude That is to say, when the number of training data samples is large enough, the optimal solution can be found in an asymptotic sense. Therefore, the present invention constructs an enhanced Lagrangian function as follows:

[0035] (6)

[0036] in is the Lagrange multiplier for the equality constraint, is the coefficient of the quadratic penalty term, Is used to ensure The coefficient of the logarithmic barrier. Use the optimization algorithm to iteratively search for the enhanced Lagrangian function to obtain the optimal solution , which is the optimal training scheduling strategy.

[0037] The present invention also provides an adaptive data parallel optimization distributed training system for heterogeneous environments, comprising the following modules:

[0038] Hybrid topology architecture module: used to build a hybrid topology architecture consisting of a parameter server and multiple computing devices;

[0039] Training Time Module: Based on a hybrid topology architecture, a unified training time model is established, comprehensively considering three types of delays: data distribution, local computing, and parameter communication. Based on the training time model, a scheduling optimization function is defined to minimize training bottlenecks.

[0040] Relaxation function module: used to introduce relaxation variables and approximation methods, transform the original scheduling optimization function into a convex optimization problem, construct the corresponding relaxation function and enhanced Lagrangian function, and realize distributed solution under constraints through the Lagrangian multiplier method, ultimately achieving an asymptotically optimal training scheduling strategy.

[0041] Beneficial effects of the present invention:

[0042] The present invention achieves more efficient and robust distributed training through the following designs. (1) Delayed parameter synchronization. By adopting local training and performing delayed parameter synchronization at the end of each round rather than the end of each batch, the present invention significantly reduces communication overhead. (2) Data parallel scheduling formulation. The present invention constructs a data parallel scheduling optimization problem to minimize training time by optimizing data distribution. (3) Efficient algorithm. The present invention develops a very efficient algorithm to solve the optimization problem and can prove that the obtained solution is optimal in asymptotic terms. By strategically distributing data, the present invention ensures load balancing between computing devices with different capabilities and significantly reduces communication overhead, thereby accelerating the training process and improving overall efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The workflow of the optimal distributed training framework based on adaptive data parallelism;

[0044] Figure 2To test the accuracy and loss changes of the present invention and a single machine under specific circumstances;

[0045] Figure 3 This is the training loss curve of Wide-ResNet101 on heterogeneous devices;

[0046] Figure 4 Accuracy and loss curves of different training methods in the time dimension. DETAILED DESCRIPTION

[0047] Combined with attachment Figure 1 As shown, the present invention will be described in detail below. The specific implementation scheme of the adaptive data parallel optimization distributed training system in heterogeneous environments is as follows:

[0048] Step S1, the present invention builds a distributed hybrid topology architecture, including the parameter server (PS) of the data owner and computing devices, such as Figure 1 As shown. PS subsets the data Sent to each participating computing device Each device Use stochastic gradient descent (SGD) on the received data subset Perform local training and then upload the locally updated model parameters based on the current situation PS performs weighted averaging based on the local model parameters of all devices to update the global model and distribute it to all devices.

[0049] in, is the total number of data samples. The total number of data samples is constrained as:

[0050] (1)

[0051] in It is a device The total amount of data allocated, is the total amount of data (the entire dataset).

[0052] Step S2, for each sample, it is distributed from the parameter server (device 0) to the computing device The distribution time can be expressed as:

[0053] (2)

[0054] in For devices 、 The transmission bandwidth, The size of each sample.

[0055] Based on the computing power of heterogeneous devices, a training time model is constructed. The specific method is as follows:

[0056] In distributed training, the computation time of each device is related to the amount of data assigned to it, that is, the larger the dataset, the longer the computation time. The maximum parallel capability is expressed as , that is, the maximum number of samples that the device can process simultaneously, we can get:

[0057] (3)

[0058] Maximum parallel capability Not less than the equipment batches, namely:

[0059] (4)

[0060] So get the equipment Computation time in one cycle Expressed as:

[0061] (5)

[0062] in For devices Memory capacity, For devices The amount of data processed in each iteration, Representation device The time required to process a single batch of data, including forward and backward propagation calculations, is the sample size of the input model after preprocessing, The memory consumption of the activation value increased by one additional sample.

[0063] Each computing device Receives the current global model parameters from the PS, updates its local model using the assigned data subset, and sends the updated model parameters to the PS at the end of each epoch. represents the size of the model parameters. Therefore, the device Communication time used for data parallelism within an epoch Expressed as:

[0064] (6)

[0065] in is the communication delay.

[0066] Constructing the training time model is as follows:

[0067] .

[0068] Step S3: In order to minimize the total time of the data parallel training process, the present invention aims to optimize the data distribution Therefore, the data parallel scheduling optimization problem is expressed as:

[0069] (7)

[0070] Step S4: To simplify the notation, the above optimization model formula (7) is restated as follows:

[0071] (8)

[0072] Notice With variables It doesn't matter, it depends on the communication bandwidth and model size , and is therefore a constant.

[0073] in The diagonal is of matrix, The diagonal is of matrix, The diagonal is of matrix, is a constant.

[0074] use Level norm approximation function, so in When is large, the objective function in Equation (8) can be rewritten as

[0075] (9)

[0076] Its feasible domain is: .

[0077] And the present invention defines the relaxation function as:

[0078] (10)

[0079] Its feasible domain is: .

[0080] Step S5: Since the relaxation function (10) is a convex function in its feasible domain, the global optimal solution of the relaxation function (10) is first obtained using the enhanced Lagrangian function. , then In the neighborhood of , you can find , making It should be noted that The total number of data samples Therefore, the optimal solution to the original equation (9) is ,have ,because and There is a positive correlation, so we can conclude That is to say, when the number of training data samples is large enough, the optimal solution can be found in an asymptotic sense. Therefore, the present invention constructs an enhanced Lagrangian function as follows:

[0081] (11)

[0082] in is the Lagrange multiplier for the equality constraint, is the coefficient of the quadratic penalty term, Is used to ensure The coefficient of the logarithmic barrier. Use the data parallel scheduling algorithm to iteratively search for the enhanced Lagrangian function to obtain the optimal solution , that is, the optimal training scheduling strategy

[0083] The process steps of the data parallel scheduling algorithm are described as follows:

[0084] (1): Set the variable Initialized to , set the iteration counter and the inner iteration counter All are 0. Initialize the Lagrange multiplier , logarithmic barrier coefficient and the learning rate is .

[0085] (2): Enter the outer iteration, when the outer iteration number Less than or equal to the maximum number of outer loops , perform the following steps:

[0086] (3): Start inner iteration and initialize inner iteration counter When the number of inner iterations Less than or equal to the maximum number of inner loops , do the following:

[0087] (4): Calculate the gradient and calculate the objective function with respect to Gradient .

[0088] (5): According to the gradient descent method, use the learning rate renew : .

[0089] (6): Calculate the norm of the gradient. If the 2-norm of the gradient norm is less than the convergence threshold , then jump out of the inner loop and end the inner iteration.

[0090] (7): Inner counter update, if the convergence condition is not met, the inner iteration counter , continue the inner loop.

[0091] (8): After updating the current inner layer Calculate the new Lagrange multiplier. The specific update formula is: ,in, is a constraint.

[0092] (9): According to the attenuation coefficient ,right To update: .

[0093] (10): Outer gradient convergence judgment, calculate the current Lagrange multiplier gradient norm, if its 2-norm is less than the threshold , then jump out of the outer loop.

[0094] (11): The outer counter is updated. If the outer convergence condition is not met, the outer iteration counter Add 1 and continue the outer loop.

[0095] (12): Final result processing, the final iteration obtained Assigned to .right Round off to get the final result .

[0096] The present invention also provides an adaptive data parallel optimization distributed training system for heterogeneous environments, comprising the following modules:

[0097] The hybrid topology architecture module is used to build a hybrid topology consisting of a parameter server and multiple computing devices, supporting heterogeneous computing resources such as GPUs, CPUs, and edge devices. The system adapts to wide-area networks and distributed deployment scenarios based on the computing power of each device and the network bandwidth and latency between the server and the server.

[0098] Training Time Module: Based on the aforementioned architecture, a unified training time model is established, comprehensively accounting for three types of latency: data distribution, local computation, and parameter communication. Based on this training time model, a scheduling optimization function is defined to minimize training bottlenecks. This optimization function includes two constraints: the total amount of data allocated to all computing devices (an integer) equals the total amount of data required for training, and the maximum parallel processing capability of each computing device is no less than the number of training batches for that device.

[0099] Relaxation function module: Because the original scheduling optimization function is non-convex and difficult to solve directly, this paper introduces relaxation variables and approximation methods to convert it into a convex optimization problem and construct a corresponding relaxation function. Based on the relaxation function, an enhanced Lagrangian function is constructed, and the Lagrangian multiplier method is used to achieve distributed solution under constraints. Combined with the data parallel scheduling algorithm, the data partitioning and synchronization frequency are dynamically updated according to the device status in each training round to ensure load balancing and optimal resource utilization among devices with different performance, ultimately achieving an asymptotically optimal training scheduling strategy.

[0100] This paper uses the widely recognized Wide ResNet101 model and the ImageNet100 dataset for experiments. This dataset covers 100 different categories, with the training set consisting of 100,000 samples and the test set consisting of 10,000 samples. The present invention adopts a PS architecture. To verify the performance of the invention in a heterogeneous environment, the training configuration is designed as follows: Rank 0 is responsible for data transmission and parameter synchronization; Ranks 1 to 4 are equipped with RTX3090 graphics cards (24GB of video memory); Ranks 5 and 6 use 4-core pure CPUs with 8GB of memory; Rank 7 uses a 16-core CPU with 32GB of memory; and Rank 8 uses an 8-core CPU with 16GB of memory. All devices are distributed in different data centers, fully demonstrating the characteristics of training configuration in a heterogeneous environment. For Wide ResNet101, the batch size is 64. For CPUs with limited computing power, the batch size is set to 8 for 4 cores and 16 for 16 cores. These settings ensure that the video memory or CPU can operate normally and do not exceed the maximum computing power and video memory of the device. The maximum epoch is set to 50, where the initial learning rate of the device is based on batchsize=64, and the learning rate is adjusted based on 0.01. The learning rate is adjusted according to the ratio of the batchsize, and the maximum setting is 0.3. In order to better converge, the invention sets the learning rate to decay to 0.5 of the previous one every 10 epochs. For fair comparison, the experiments all use the above initialization method. The accuracy and loss changes of the present invention and the single machine under specific conditions are as follows Figure 2 shown.

[0101] The method of the present invention is compared with the mainstream training frameworks DDP and deep speed. The experimental results are shown in Tables 1 and 2. On the ImageNet100 dataset, the present invention achieves rapid convergence in a heterogeneous distributed training environment. Compared with distributed data parallelism (DDP) and DeepSpeed, the present invention achieves an improvement of 21.6 times and 26.3 times in the number of floating-point operations, respectively, and shortens the training time by approximately 72% and 47%, respectively. This distributed computing solution can realize the computing power scheduling of idle GPU and CPU resources across regions, fully releasing the potential of globally distributed computing nodes. The training loss curve of Wide-ResNet101 on heterogeneous devices is shown in Figure 2. Figure 3 As shown; the accuracy and loss curves of different training methods in the time dimension are as follows Figure 4 shown.

[0102] Table 1 Accuracy index and verification loss results of the present invention (8 devices) and single card training under interval period conditions

[0103]

[0104] Table 2 Comparison of training results of different training methods

[0105]

Claims

1. Adaptive data parallel optimization distributed training method for heterogeneous environments, characterized by: The following steps are involved: S1. Build a hybrid topology consisting of a parameter server and multiple computing devices; S2. Based on the hybrid topology architecture, a unified training time model is established, which comprehensively considers the three types of delays: data distribution, local computing, and parameter communication. S3. Based on the training time model, define a scheduling optimization function with the goal of minimizing the training bottleneck; S4. Introduce slack variables and approximation methods to convert the original scheduling optimization function into a convex optimization problem and construct the corresponding relaxation function; S5. Based on the relaxation function, an enhanced Lagrangian function is constructed, and the distributed solution under the constraints is realized through the Lagrangian multiplier method, and finally an asymptotically optimal training scheduling strategy is achieved.

2. The adaptive data parallel optimization distributed training method for heterogeneous environments according to claim 1 is characterized in that: The step S1 is specifically as follows: constructing a distributed hybrid topology architecture, including the parameter server PS of the data owner and computing devices, PS subsets the data , , sent to each participating computing device ; Each device Use stochastic gradient descent on the received data subset Perform local training and then upload the locally updated model parameters based on the current situation ; PS performs weighted averaging based on the local model parameters of all devices to update the global model and distribute it to all devices.

3. The adaptive data parallel optimization distributed training method for heterogeneous environments according to claim 2, characterized in that: The step S2 is specifically implemented as follows: Set the parameter server as device 0, divide the product of the data subset and the sample size by the transmission bandwidth, and calculate the distribution of each sample from the parameter server to the computing device The distribution time of ; In distributed training, the computation time of each device is related to the amount of data assigned to it. The maximum parallel capability is expressed as , that is, the maximum number of samples that the device can process simultaneously, and the maximum parallel capability Not less than the equipment Batch, get equipment Computation time in one cycle ; Each computing device Receive the current global model parameters from PS, update its local model using the assigned data subset, and send the updated model parameters to PS at the end of each epoch; Indicates the size of the model parameters; device Communication time used for data parallelism within an epoch ; Build a training time model: 。 4. The adaptive data parallel optimization distributed training method for heterogeneous environments according to claim 3, characterized in that: The scheduling optimization function includes two constraints: the total amount of data allocated to all computing devices is equal to the total amount of data required for training, and the maximum parallel processing capability of each computing device is not less than the number of training batches for the device, as follows: Based on distribution time , calculation time , communication time , minimize the total time of the data parallel training process and optimize data distribution ; The data parallel scheduling optimization problem is expressed as .

5. The adaptive data parallel optimization distributed training method for heterogeneous environments according to claim 4, characterized in that: The step S4 is specifically implemented as follows: The data parallel scheduling optimization problem is reconstructed as: ， ； in The diagonal is of matrix, The diagonal is of matrix, The diagonal is of matrix, is a constant; use Level norm approximation Function, Function Rewritten as: , its feasible domain is: ; The relaxation function is defined as: , its feasible domain is: .

6. The adaptive data parallel optimization distributed training method for heterogeneous environments according to claim 5, characterized in that: The step S5 is specifically implemented as follows: The relaxation function is a convex function in its feasible domain. First, the enhanced Lagrangian function is used to find the global optimal solution of the relaxation function. , then In the neighborhood of , making , The total number of data samples It doesn't matter, for the function The optimal solution of the rewritten equation ,have , and Positively correlated, , construct an enhanced Lagrangian function as follows: ； in is the Lagrange multiplier for the equality constraint, is the coefficient of the quadratic penalty term, Is used to ensure The coefficient of the logarithmic barrier; use the optimization algorithm to iteratively search for the enhanced Lagrangian function to obtain the optimal solution , which is the optimal training scheduling strategy.

7. An adaptive data parallel optimization distributed training system for heterogeneous environments, used to implement the adaptive data parallel optimization distributed training method according to any one of claims 1 to 6, characterized in that: Includes the following modules: Hybrid topology architecture module: used to build a hybrid topology architecture consisting of a parameter server and multiple computing devices; Training Time Module: Based on a hybrid topology architecture, a unified training time model is established, comprehensively considering three types of delays: data distribution, local computing, and parameter communication. Based on the training time model, a scheduling optimization function is defined to minimize training bottlenecks. Relaxation function module: used to introduce relaxation variables and approximation methods, transform the original scheduling optimization function into a convex optimization problem, construct the corresponding relaxation function and enhanced Lagrangian function, and realize distributed solution under constraints through the Lagrangian multiplier method, ultimately achieving an asymptotically optimal training scheduling strategy.

Images