A distributed parallel training method of a deep neural network in a dynamic computing environment

Abstract

The application belongs to the technical field of distributed system optimization and scheduling, and particularly relates to a distributed parallel training method of a deep neural network in a dynamic computing environment, comprising: determining a model splitting scheme by minimizing an actual convergence time of an asynchronous pipeline, and distributing each model slice corresponding to the splitting scheme to a corresponding device node, wherein the actual convergence time is a product of an upper limit of an iteration number corresponding to a preset convergence error and an optimal running time of the pipeline required for a single iteration; each device node is configured with a trained lightweight random forest model for evaluating the interference influence of an additional load located at the device node on the training efficiency of a model slice located at the device node. After starting training, interference coefficients corresponding to the interference influence obtained by the random forest models on each device node are obtained in real time, a new splitting scheme is obtained through model search space pruning and traversal algorithms, and efficient training is realized through layer migration.

Description

Technical Field

[0001] This invention belongs to the technical field of distributed system optimization and scheduling, and more specifically, relates to a distributed parallel training method for deep neural networks in a dynamic computing environment. Background Technology

[0002] With the widespread application of deep neural networks in fields such as artificial intelligence, natural language processing, and computer vision, the size and complexity of models are constantly increasing. Training these large-scale neural networks typically requires a distributed environment. However, the transmission of large amounts of data during distributed training can lead to severe communication bottlenecks, thus affecting training efficiency. Especially in dynamic computing environments, interference from external tasks can further degrade system performance. Therefore, how to optimize the utilization of communication and computing resources while ensuring model accuracy and improving training efficiency is a crucial problem that urgently needs to be solved. Summary of the Invention

[0003] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a distributed parallel training method for deep neural networks in a dynamic computing environment. Its purpose is to optimize the utilization of communication and computing resources and improve training efficiency while ensuring model accuracy.

[0004] To achieve the above objectives, according to one aspect of the present invention, a distributed parallel training method for deep neural networks in a dynamic computing environment is provided, comprising:

[0005] S1. By minimizing the actual convergence time of the asynchronous pipeline, a deep neural network model segmentation scheme is determined, and the deep neural network model slices corresponding to the segmentation scheme are distributed to the corresponding device nodes. The actual convergence time of the asynchronous pipeline is the product of the upper limit of the number of iterations corresponding to the preset convergence error and the optimal pipeline running time required for a single iteration. Each device node is configured with a trained lightweight random forest model to evaluate the interference effect of the additional load on the device node on the training efficiency of the model slices on the device node.

[0006] S2. After starting the training of the deep neural network model, the interference coefficients corresponding to the interference effects predicted by the lightweight random forest model on each device node are obtained in real time. Based on the communication and computing time of each layer in the deep neural network model in its current computing environment, the top k layers that are most important to the training efficiency of the deep neural network model are identified. A preset number of adjacent layers of each of the top k layers are merged with this layer as a logical unit layer in the model slicing plan to obtain a simplified model structure. Based on the simplified model structure, the input parameters required for the traversal algorithm are determined, and the interference coefficients are used as an input parameter of the traversal algorithm. A new deep neural network model slicing scheme is obtained through the traversal algorithm. If the currently configured deep neural network model slicing scheme is different from the new deep neural network model slicing scheme, the load distribution between device nodes is dynamically adjusted by gradually migrating the computing tasks of some layers during the idle time period of training to realize the configuration of the new model slicing scheme. Step S2 is repeated until the distributed parallel training of the deep neural network in the dynamic computing environment is completed.

[0007] Furthermore, the way to minimize the actual convergence time of the asynchronous pipeline is as follows:

[0008]

[0009] In the formula, S is the number of model slices; W(S) is the optimal pipeline runtime required for a single iteration; N is the total number of GPUs in all device nodes; ∈ is the preset convergence error; η is the learning rate required for training the neural network model; L is the smoothing constant; ξ is the quadratic moment; f(w (0) ) represents the gradient variance.

[0010] Furthermore, the lightweight random forest model on each device node predicts the interference coefficient based on the model slice features and device statistics located on that device node.

[0011] Furthermore, the model slice features include flop computation count, layer features, and network demand; the device statistics include GPU utilization, data bus throughput, GPU memory usage, and PCIe data transfer rate.

[0012] Furthermore, the interference coefficient is the ratio of the running time of a single model slice under the same additional load interference condition on the same device node to the baseline running time of the same model slice under no interference condition.

[0013] Furthermore, the method for dynamically adjusting the load distribution among device nodes by gradually migrating the computational tasks of partial layers is as follows:

[0014] The state information of each layer to be migrated is serialized into a data stream, and the data stream is stored in the local CPU memory of the layer to be migrated.

[0015] The migration task is divided into multiple small independent tasks, each responsible for migrating a portion of the layer data. These independent tasks are executed in parallel. In subsequent training iterations, during the idle time periods of each training iteration, each independent task is controlled to gradually distribute the serialized data stream to the target device node.

[0016] Furthermore, the state information for each layer to be migrated includes layer parameters, random seed, and optimizer state.

[0017] Furthermore, the migration of layer data streams is achieved in the following way: by introducing a priority queue in the network stack of the operating system through a priority flow control protocol, the training data stream is given a higher priority, and the migration data stream is marked as a low priority, so as to avoid the conflict between the layer migration data stream and the normal training data stream during network transmission.

[0018] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0019] According to another aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is run by a processor, it controls the device where the storage medium is located to perform the steps of the method described above.

[0020] In summary, compared with the prior art, the technical solutions conceived by this invention have the following main advantages:

[0021] 1. This invention proposes a distributed deep neural network training method with end-to-end pipeline parallelism in a dynamic environment based on asynchronous convergence analysis and task interference awareness mechanism. It parallelizes the traditional deep neural network computation-communication process through a multi-stage asynchronous strategy, improving the utilization of GPU and network bandwidth. Simultaneously, this invention introduces a model partitioning optimization algorithm based on asynchronous convergence analysis, reducing communication overhead while ensuring model training accuracy. Furthermore, this invention employs a lightweight random forest model to evaluate the impact of task interference on training efficiency. When task interference occurs, it imperceptibly completes the model repartitioning plan during idle training periods, thus theoretically achieving the optimal end-to-end pipeline partitioning strategy and further improving resource utilization efficiency during training.

[0022] 2. This invention also proposes a transparent layer migration mechanism (i.e., a seamless model layer migration mechanism), further optimizing the model redeployment process in dynamic environments. When task interference causes uneven GPU load, the workload is redistributed by migrating the computational tasks of some layers. Specifically, the state information of the layers to be migrated is first serialized, and this state data is packaged into checkpoint files and distributed to designated GPUs. The migration process adopts an asynchronous approach, effectively reducing the impact of migration on the training process by parallelizing the layer migration operation with idle time periods during training, thereby achieving seamless layer migration. Attached Figure Description

[0023] Figure 1 This is a flowchart of a distributed parallel training method for deep neural networks in a dynamic computing environment provided by an embodiment of the present invention;

[0024] Figure 2 This is a diagram of the distributed parallel training system architecture for deep neural networks in a dynamic computing environment provided by an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of the seamless migration process of the model layer provided in an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of model search space pruning provided in an embodiment of the present invention;

[0027] Figure 5 These are comparison charts of system training under steady-state conditions provided in embodiments of the present invention, wherein (a) is a comparison chart of average single-iteration time, (b) is a comparison chart of final training accuracy, and (c) is a comparison chart of throughput.

[0028] Figure 6 This is a schematic diagram of system sensitivity under dynamic conditions provided in an embodiment of the present invention;

[0029] Figure 7 This is a comparison chart of end-to-end training in a dynamic environment provided by an embodiment of the present invention;

[0030] Figure 8 This is a comparison chart of average iteration time under dynamic conditions provided by an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0032] Example 1

[0033] A distributed parallel training method for deep neural networks in a dynamic computing environment, such as Figure 1 As shown, it includes:

[0034] S1. By minimizing the actual convergence time of the asynchronous pipeline, a deep neural network model segmentation scheme is determined, and the deep neural network model slices corresponding to the segmentation scheme are distributed to the corresponding device nodes. The actual convergence time of the asynchronous pipeline is the product of the upper limit of the number of iterations corresponding to the preset convergence error and the optimal pipeline running time required for a single iteration. Each device node is configured with a trained lightweight random forest model to evaluate the interference effect of the additional load on the device node on the training efficiency of the model slices on the device node.

[0035] S2. After starting the training of the deep neural network model, the interference coefficients corresponding to the interference effects predicted by the lightweight random forest model on each device node are obtained in real time. Based on the communication and computing time of each layer in the deep neural network model in its current computing environment, the top k layers that are most important to the training efficiency of the deep neural network model are identified. A preset number of adjacent layers of each of the top k layers are merged with the layer as a logical unit layer in the model slicing plan to obtain a simplified model structure. Based on the simplified model structure, the input parameters required for the traversal algorithm are determined, and the interference coefficients are used as an input parameter of the traversal algorithm. A new deep neural network model slicing scheme is obtained through the traversal algorithm. If the currently configured deep neural network model slicing scheme is different from the new deep neural network model slicing scheme, the load distribution between device nodes is dynamically adjusted by gradually migrating the computing tasks of some layers during the idle time period of training to realize the configuration of the new model slicing scheme.

[0036] S3. Repeat step S2 until the distributed parallel training of the deep neural network in the dynamic computing environment is completed.

[0037] This embodiment proposes a distributed deep neural network training method with end-to-end pipeline parallelism in a dynamic environment based on asynchronous convergence analysis and task interference awareness mechanism. It parallelizes the traditional deep neural network computation-communication process through a multi-stage asynchronous strategy, improving the utilization of GPU and network bandwidth. Simultaneously, this embodiment introduces a model partitioning optimization algorithm based on asynchronous convergence analysis, reducing communication overhead while ensuring model training accuracy. Furthermore, this embodiment also employs a lightweight random forest model to evaluate the impact of task interference on training efficiency. When task interference occurs, it imperceptibly completes the model repartitioning plan during idle periods in training, thus theoretically achieving the optimal end-to-end pipeline partitioning strategy and further improving resource utilization efficiency during training.

[0038] Regarding the model partitioning algorithm based on asynchronous convergence analysis, during distributed training, by analyzing the model lag at each stage and adjusting the model partitioning strategy, the computational load balance at each stage is ensured, communication latency is reduced, and training time is minimized while maintaining convergence efficiency. As a preferred implementation, the quantified convergence degree of the partitioned model and the actual training time are as follows:

[0039]

[0040] In the formula, S is the number of model slices; W(S) is the optimal pipeline runtime required for a single iteration; N is the total number of GPUs in all device nodes; ∈ is the preset convergence error; η is the learning rate required for training the neural network model; L is the smoothing constant; ξ is the quadratic moment; r(w (0) ) represents the gradient variance.

[0041] In practice, the optimal segmentation scheme can be obtained by traversing all the stages (i.e., the value of S).

[0042] This paper proposes an end-to-end pipelined parallel training model partitioning scheme based on convergence analysis. Specifically, it aims to address the problems of suboptimal partitioning schemes and training efficiency caused by the mismatch between computation and communication models in traditional asynchronous pipelined models during deep neural network training. In asynchronous pipelined parallel training models, how to partition the model is one of the key factors determining training performance. The goal of model partitioning is to divide a complete deep neural network into several parts (i.e., stages) and allocate these parts to different computing nodes (such as GPUs) for parallel computation. However, simply partitioning according to computational load often fails to achieve optimal results, especially in asynchronous training, where communication overhead and convergence speed are also important factors affecting training efficiency. Therefore, this embodiment proposes an optimization strategy based on detailed convergence analysis and accurate pipeline feature characterization to achieve the optimal model partitioning scheme. First, by analyzing the three main stages of pipelined training—the warm-up stage, the iterative update stage, and the termination stage—the key factors affecting training efficiency are identified. The warm-up stage refers to the process in which each stage of the model sequentially enters the computational state. This stage is usually short and has a relatively small impact on the overall training time. The iterative update phase is the core of the training process, occupying most of the time. In this phase, forward and backward propagation calculations are repeatedly performed at each stage. The termination phase is the final stage of training, mainly used to synchronize the calculation results of all stages and complete the final model update. During the iterative update phase, the decisive factors for training time include the computation time and communication time of each iteration. In the design process, particular attention was paid to accurately characterizing the communication overhead. Traditional pipelined parallel training models typically assume that the communication time of the computation stages can be completely masked within the computation time; however, in practical applications, this assumption often does not hold. Therefore, the communication time between each computation stage was measured and analyzed in detail, and the segmentation scheme of the entire pipeline was optimized based on this. Through a reasonable model segmentation strategy, this invention effectively reduces the non-overlapping parts of communication between stages, further reducing the total training time.

[0043] To facilitate convergence analysis, several widely adopted assumptions are introduced: (1) the gradient of the loss function is L-smooth; (2) the stochastic gradient is bounded, i.e. (3) The variance of the stochastic gradient on the training sample x is bounded, i.e. Assume ηL≤1, it satisfies:

[0044]

[0045] The above formula shows that, despite the existence The convergence error is still affected by the number of update iterations T and the number of stages S. Therefore, although model partitioning can reduce the time per iteration, it often leads to a decrease in statistical performance (i.e., convergence performance, convergence error). Referring to the convergence error, assume that pipeline training has converged when the error decreases to a specified minimum threshold ∈. Therefore, the upper limit of the number of update iterations T required for pipeline training to reach the specified convergence threshold ∈ is:

[0046]

[0047] Finally, the actual convergence time of the asynchronous pipeline can be expressed as:

[0048]

[0049] The convergence quantification (convergence error) of each stage is incorporated into the model splitting scheme (the optimal running scheme under S), comprehensively considering the trade-off between running time and convergence (W(S) is the pipeline's optimal running time (time per iteration) when the model is split into S model slices, and N is the number of GPUs). This scheme not only maximizes the utilization of hardware resources but also effectively reduces convergence problems caused by stage delays in an asynchronous environment.

[0050] The end-to-end pipelined parallel training model partitioning scheme based on convergence analysis proposed in this embodiment significantly improves the training speed of deep neural networks by employing asynchronous computation and communication processing, detailed convergence analysis, optimized model partitioning strategies, and precise characterization of communication features. It also maximizes hardware resource utilization efficiency while ensuring model convergence. In a steady-state computing environment, this algorithm achieves an efficient and stable training process. The combination of these innovative technologies provides strong technical support for handling large-scale deep learning tasks, greatly promoting the efficiency and performance improvement of deep neural network training.

[0051] Regarding the lightweight random forest model, this model non-intrusively predicts the impact of external task interference on training efficiency by evaluating runtime statistics such as GPU status, network bandwidth, and data bus status at each stage. This dynamically adjusts the model partitioning plan, further reducing the training efficiency decline caused by task interference and ensuring efficient training in dynamic computing environments. As a preferred implementation, the lightweight random forest model on each device node predicts the interference coefficient based on the model slice features and device statistics located on that device node.

[0052] As a preferred implementation, model slicing features include flop computation count, layer features, and network demand; device statistics include GPU utilization, data bus throughput, GPU memory usage, and PCIe (i.e., system bus) data transfer rate.

[0053] As a preferred embodiment, the interference coefficient is the ratio of the running time of a single model slice under the same additional load interference condition on the same device node to the baseline running time of the model slice under no interference condition.

[0054] S2 in the above implementation method can be viewed as a lightweight task interference perception and adjustment mechanism based on random forest, used to evaluate the interference impact of external tasks on the deep neural network training process in real time and dynamically optimize the partitioning strategy of the training model. In the current complex distributed computing environment, deep neural network training tasks are often interfered with by various external factors, such as resource competition between other parallel tasks and the training task, fluctuations in network bandwidth, dynamic allocation of GPU resources, and the insertion of sudden tasks. These interference factors may cause the load of some computing nodes to suddenly increase or decrease, thereby disrupting the balance of the entire training process and affecting the training efficiency and model convergence speed. To address these challenges, a random forest-based interference perception model is designed, which can monitor and evaluate the degree of impact of these external interferences in real time during runtime, and intelligently adjust the model partitioning strategy based on the evaluation results to minimize the negative impact of interference on the training process.

[0055] The Random Forest model is designed as a lightweight, non-intrusive monitoring tool capable of accurately sensing and predicting the impact of task disturbances without significantly increasing system load. First, the system continuously collects various runtime statistics, including but not limited to: GPU utilization, network bandwidth, data bus throughput, forward and backward propagation times for each computation stage, GPU memory usage, PCIe data transfer rates, and the execution time of each task. This data provides the model with comprehensive runtime environment information, enabling the Random Forest model to accurately capture changes in the current computing environment. During the system design and implementation phases, the Random Forest model is built through offline training on a self-generated dataset. This dataset covers real-world runtime data for typical deep learning training tasks under different disturbance conditions. Specifically, the training process includes constructing a training sample set and training and optimizing model parameters. The training sample set is constructed by first capturing the standard runtime of a single GPU training task under disturbance-free conditions. Then, these training tasks are exposed to various external disturbance conditions of varying intensities. By comparing the runtime of each training task under these perturbation conditions with the baseline runtime of the same training task under no-perturbation conditions, the actual impact of the perturbation on the training task can be obtained. This actual impact becomes the "true value" of the training model, used to guide the learning of the random forest model. The model uses this data, taking GPU state (such as SM activity, DRAM throughput, etc.) and task characteristics as input features, and through repeated training and validation, optimizes the predictive ability of the random forest model, enabling it to efficiently and accurately assess the impact of perturbation in real-time applications. In practical applications, when the system detects a new task or environmental change, the random forest model is activated to evaluate the adaptability of the current model's partitioning strategy. By analyzing the working state of the current computing nodes in real time, the model can quickly identify nodes that may become bottlenecks due to external perturbation. These bottleneck nodes typically exhibit a sudden increase in GPU utilization, increased data transmission latency, or a significant extension of forward / backward propagation time. Based on this real-time runtime data, the model can predict whether the current partitioning strategy is still optimal and suggest adjustments to the strategy if necessary. Adjustments may include reallocating computational tasks, such as moving computationally intensive tasks from overloaded nodes to less loaded ones, or optimizing data transmission paths to reduce communication latency caused by network bandwidth bottlenecks. Additionally, the system may adjust the model's parallelism or change partition boundaries to ensure a more balanced workload across computing nodes, thereby improving overall training efficiency.

[0056] A significant advantage of the Random Forest model lies in its non-intrusive design. It does not significantly interfere with the ongoing training process, nor does it require stopping the training task or interrupting the training flow. Instead, it can run seamlessly in the background, continuously monitoring the system status and taking immediate action when problems are detected. This non-intrusiveness allows the system to dynamically adjust without affecting training stability, thus ensuring the efficiency and continuity of the training process. Furthermore, the Random Forest model can continuously optimize its predictive capabilities based on feedback from actual operation. Over time, the system can accumulate more operational data, which can be used to further train and optimize the Random Forest model, enabling it to exhibit higher adaptability and accuracy when facing different types of disturbances. This adaptive learning mechanism ensures that the system can maintain high efficiency in complex and ever-changing computational environments.

[0057] Through this random forest-based task interference perception and adjustment mechanism, this invention achieves accurate perception and intelligent response to external task interference, significantly improving the efficiency of deep neural network training and effectively reducing the negative impact of external interference on system performance. The flexibility and adaptability of this mechanism enable continuous optimization in complex computing environments, thus ensuring the efficiency, stability, and robustness of the training task.

[0058] This embodiment employs a 1F1B scheduling strategy, dynamically adjusting the pipeline partitioning strategy to fully utilize the computing power of the GPU and network bandwidth, thereby improving the training efficiency of distributed deep neural networks. The core idea is to utilize a lightweight random forest model during training to evaluate and minimize the impact of task interference, adjust the model partitioning plan in real time to adapt to the dynamically changing computing environment, and prune the model partitioning scheme space through model representation compression to ensure the stability and efficiency of the training process.

[0059] As a preferred implementation, a transparent layer migration mechanism (i.e., a seamless model layer migration mechanism) is introduced to further optimize the model redeployment process in dynamic environments. When task interference causes uneven GPU load, the workload is redistributed by migrating the computational tasks of some layers. Specifically, the layer states to be migrated (such as layer parameters, random seeds, and optimizer states) are first serialized, and these state data are packaged into checkpoint files and distributed to designated GPUs. The migration process is asynchronous, effectively reducing the impact of migration on the training process by processing layer migration operations in parallel with idle time periods during training, thus achieving seamless layer migration. This transparent layer migration mechanism ensures that the system maintains optimal training efficiency and model convergence speed even under high-load interference environments.

[0060] The proposed transparent layer migration mechanism aims to address the problem of unbalanced computing node load caused by external task interference or uneven resource utilization during distributed deep neural network training. This mechanism allows the system to reallocate computing tasks without affecting the training process, thereby rebalancing system resource utilization and ensuring the efficiency and stability of the training process. In complex distributed computing environments, deep neural network training tasks are often affected by various external disturbances. For example, a sudden increase in other parallel tasks, fluctuations in network bandwidth, or dynamic scheduling of GPU resources can all lead to a sharp increase in the load of certain computing nodes, becoming a bottleneck in the entire training process. Traditional solutions typically require pausing the training task and then manually reconfiguring computing resources, which is not only time-consuming but also interrupts the training process, slowing down model convergence. To solve these problems, this embodiment proposes a preferred method of a transparent layer migration mechanism that can dynamically adjust the load distribution among device nodes without interrupting the training task. This mechanism embeds layer migration operations into idle periods during the training process (e.g., time gaps between computation stages) through asynchronous processing, thereby minimizing interference with the training process and achieving seamless migration and repartitioning.

[0061] Specifically, the layer migration mechanism described above includes the following key steps:

[0062] (1) Model search space pruning: First, the system will divide the deep neural network model into blocks, merging multiple computational layers into a single logical unit to form a block, thereby reducing the amount of data to be processed during the migration process. To determine which layers should be migrated, the system will identify the key layers that have the greatest impact on overall performance based on the computation time, communication overhead, and other key characteristics of each layer. These key layers are often the most computationally intensive parts, and migrating these layers can significantly balance the computational load and improve the overall performance of the system. (2) Traversal algorithm to determine a new partitioning scheme, i.e., repartitioning. (3) Serialization and distribution of layer states: After determining the layers that need to be migrated, the system will serialize the states of these layers (including model parameters, random seeds, optimizer states, etc.) into data streams and store these data streams in local CPU memory. To ensure the efficiency of the migration process, the system divides the migration task into multiple small independent tasks, each task responsible for migrating a portion of the data. These tasks can be executed in parallel, thereby accelerating the migration process. Asynchronous snapshot and transmission: During the migration process, the system adopts an asynchronous snapshot mechanism to separate the snapshot of layer states from the transmission operations (serialization and transmission). Specifically, the system will gradually distribute the serialized data stream to the migration target node in the subsequent training iterations. Since these operations are performed asynchronously, they will not block the normal training process, and the migration operation can be performed without affecting the overall system performance. Through this pipelined processing method, the system can efficiently complete the layer migration task. As a further preferred option, (4) migration flow control: In order to avoid conflicts between the layer migration data stream and the normal training data stream during network transmission, the system designs a priority flow control protocol. This protocol introduces a priority queue in the network stack of the operating system. The training data stream is given a higher priority, while the migration data stream is marked as a low priority. In this way, while prioritizing the processing of training data, the network resources can still ensure the smooth transmission of migration data. The system will dynamically adjust the size of the migration data queue to adapt to the current network bandwidth conditions, thereby completing the migration task while minimizing interference with the training data stream. Through the above steps, the method of this embodiment can dynamically adjust the load distribution between computing nodes without affecting the training task, and repartition the model when necessary. The repartitioning process is also transparent and does not require pausing the training task or restarting the training process. In this way, the system can react quickly when it encounters external interference, and ensure the continuous and efficient operation of the training task by adjusting the allocation of computing resources.

[0063] The aforementioned transparent layer transfer mechanism not only enhances the robustness of deep neural network training but also significantly improves the system's adaptability to complex computing environments. By reducing training interruption time and optimizing resource utilization, the method in this embodiment can maintain an efficient and stable training process in the face of dynamically changing computing environments, greatly improving the model's training speed and convergence efficiency.

[0064] In practical implementation, the key steps can be summarized as follows:

[0065] 1. First, the system starts a Pipeline Planner ( Figure 2 It performs a detailed analysis of each layer of the deep neural network model, recording the forward and backward propagation times, as well as the magnitudes of gradients and activations. Based on this data, the Pipeline Planner runs Algorithm 1 to develop an optimized model partitioning scheme that strikes a balance between hardware speed and statistical convergence. Once determined, each stage of the model is assigned to a designated computation node and pipeline training begins under a 1F1B strategy.

[0066] 2. Runtime Scheduler Figure 2 A RuntimeScheduler is launched on each GPU worker node to manage communication and computation tasks. This scheduler is responsible for the transmission of intermediate activations: during forward propagation, it receives input from the previous stage and sends the generated activations; during backpropagation, it processes incoming gradients and ensures they are accurately distributed to the designated worker nodes. Furthermore, the scheduler logs runtime statistics for each stage and worker node, such as forward and backpropagation times per layer, GPU SM activity, and DRAM receive throughput. This information forms the basis for dynamic model repartitioning and is periodically pushed to the Global Manager with minimal disruption.

[0067] 3. Global Manager Figure 2 The Global Manager's role is to integrate information from all Runtime Schedulers and formulate a new partitioning plan using a lightweight repartitioning algorithm (Algorithm 2). This recalibrated plan is then communicated back to each scheduler to rebalance communication and computational load among GPU worker nodes to accommodate interference from external tasks. The Global Manager avoids the need for additional communication processes by co-locating with the final stage, as runtime statistics collection can be achieved through forward-activated transmissions.

[0068] 4. Migration Module Figure 2This module handles layer migration across stages and reconfiguration of batch sizes within stages. To achieve seamless model repartitioning in the event of unexpected interruptions, it maintains a comprehensive deep neural network model graph, covering network architecture declarations and tensor transfer processes. During repartitioning, the module rewrites the graph structure description and encapsulates layer states (e.g., parameters, random seeds, optimizer states) into checkpoints (asynchronous snapshot mechanism), then distributes them to the affected stages based on the differences between the old and new partitioning plans. The system divides layer migration into serialization and distribution stages. Figure 3 The system minimizes migration time through pipelined parallel serialization and transmission. Furthermore, it implements a priority-based flow control protocol to manage the output of training and checkpoint traffic, thereby reducing network contention.

[0069] The above modules can significantly improve the efficiency of deep neural network training in dynamic computing environments, ensuring that the system can respond quickly and continue to operate in a highly efficient and stable manner when faced with external interference.

[0070] The end-to-end pipelined parallel training model splitting scheme based on convergence analysis is shown in Algorithm 1:

[0071]

[0072] The specific steps and corresponding operations of Algorithm 1 are as follows:

[0073] 1. Initialization parameters: Set the number of GPUs (N), the number of model layers (M), the model configuration file (including the forward and backward propagation times), and the network bandwidth. This corresponds to line 1 of the algorithm.

[0074] 2. Outer loop iterates through the number of partitions. For different numbers of partitions (S) (from 1 to N), the algorithm begins to iterate through the possible number of partitions. This corresponds to line 2 of the algorithm.

[0075] 3. Calculate the number of iterations: Calculate the number of iterations required for the current number of partitions (S), and initialize a temporary record set (Rec') to store the current partitioning scheme. This corresponds to lines 3-4 of the algorithm.

[0076] 4. Inner loop iterates through partition points: For each possible partition point (r) (from 1 to N), the algorithm calls the dynamic programming algorithm to calculate the corresponding partition scheme (Partition') and its corresponding iteration time (W(M,S,r,N)), and adds the calculation result to the temporary record set (Rec'). This corresponds to lines 5-7 of the algorithm.

[0077] 5. Select the optimal partitioning time: For the current number of partitions (S), select the minimum iteration time from the temporary record set as the optimal solution W(S) for the current number of partitions, and calculate the total training time WallClockTime(S) for the current number of partitions. Then, save it along with the corresponding partitioning solution to the record set (Rec). This corresponds to lines 8-11 of the algorithm.

[0078] 6. Final selection of the best solution: After traversing all partition numbers, select the minimum total training time (WallClockTime) as the final best partitioning solution and return it. This corresponds to lines 12-14 of the algorithm.

[0079] Through the above steps, the algorithm gradually explores different partitioning schemes and iteratively adjusts the partitioning boundaries to find the optimal partitioning scheme that can achieve the highest training efficiency in a specific computing environment.

[0080] The re-segmentation algorithm is shown in Algorithm 2:

[0081]

[0082] Algorithm 2 details the model's repartitioning process, which mainly includes the following key steps:

[0083] 1. Model search space pruning (Lines 3-6) Figure 4 The algorithm first identifies the k most important layers in the deep neural network (DNN), determined based on their communication and computation time in the current computing environment. These key layers play a crucial role in the entire model, directly impacting its computational efficiency and communication overhead. After identifying the key layers, the algorithm progressively merges consecutive layers in the model into larger blocks by combining them with their adjacent layers. This process, known as model search space pruning, aims to simplify the model structure, allowing subsequent partitioning operations to be performed more efficiently. By combining layers into blocks, the algorithm reduces the complexity of partitioning and ensures that the layers within each block have similar computational and communication characteristics, thus enabling better partitioning optimization.

[0084] 2. Pipeline Partition Adjustment (Lines 7-15): After pruning the model search space, the algorithm enters the partition adjustment phase. At this point, the simplified model structure is formed, with each block representing a group of layers with similar computational characteristics. The algorithm comprehensively explores different partitioning schemes by traversing all possible partition boundaries. Each partitioning scheme needs to determine the most effective partition boundary and select a load-balanced batch size suitable for parallelization within the pipeline. To evaluate each potential repartitioning scheme, the algorithm introduces a custom random forest model. This model is used to predict the performance of these partitioning schemes when run on a specified GPU alongside interfering tasks. The random forest model can accurately predict the computational performance under different partitioning schemes based on the input model structure and the characteristics of the current computing environment. This prediction, based on historical data and the specific configuration of the model, helps the algorithm better select the optimal partitioning scheme. Finally, the algorithm uses an evaluation simulator to comprehensively evaluate all potential partitioning schemes based on various performance metrics, including computational efficiency, communication overhead, and training speed. Through comprehensive analysis, the algorithm can identify the partitioning scheme that best improves overall training efficiency and ultimately select the optimal model repartitioning scheme.

[0085] 3. Final Output: After the above steps, the algorithm finally outputs an optimal partitioning scheme that achieves the highest training efficiency under the current computing environment. This scheme rationally distributes the various parts of the model across different computing nodes to ensure load balance and efficient utilization of computing resources.

[0086] In summary, Algorithm 2 provides an effective method for model repartitioning through model search space pruning, partition boundary exploration, and performance evaluation. This method can dynamically adjust the model's partitioning strategy based on the current computing environment and the interference tasks, thereby optimizing the training efficiency and stability of deep neural networks.

[0087] The end-to-end pipelined parallel training model partitioning scheme based on convergence analysis, the lightweight random forest model, and the layer transfer mechanism can independently or jointly accelerate distributed deep neural network training in both stable and dynamic environments. Experiments evaluated the training of three typical deep neural network models: BERT-48 (680M parameters), VGG-16 (110M parameters), and GPT-2 1.7B (1.7B parameters). BERT-48 and GPT-2 1.7B were trained on the BookCorpus text dataset, while VGG-16 was trained on the ImageNet-1K and CIFAR-10 image datasets. The effectiveness of this invention for distributed deep neural network training will be illustrated below by examining performance in both stable and dynamic environments.

[0088] First, the comparison results are presented under stable computing conditions where external disturbances can be ignored.

[0089] Reduced Convergence Time (CT): Our system (DynPipe) achieves convergence in a shorter time, 1.07 to 2.78 times faster than PipeDream and 1.496 to 3.375 times faster than Pipeline-1F1B, while maintaining comparable test accuracy or global loss. Furthermore, when applied to the pipelined architecture DAPPLE, end-to-end optimizations improve training speed by 1.11 to 2.55 times across various models, such as... Figure 5 As shown in (a) above, this demonstrates the system's versatility in synchronous training scenarios. Combined with... Figure 5 In (b), the system consistently outperforms the benchmark, with CT reductions of 7% to 76% without impacting model performance.

[0090] Training throughput: Figure 5 (c) in the middle shows the relationship with Figure 5 (b) Average training throughput under the same settings. On image datasets, compared to PipeDream and Pipeline-1F1B, this system improves throughput by 60.6% to 83.7% and 280.1% to 423.5%, respectively. For BERT-48 and GPT-2 1.7B on text datasets, throughput improvements are 6.8% to 15.4% and 55.2% to 297.6%, respectively.

[0091] This system can adapt to dynamic task interference. To verify this, an external DNN training task is introduced to simulate real-world interference in a production cluster.

[0092] Sensitivity to interference: VGG-16 was trained on the CIFAR-10 dataset to obtain the iteration time in the presence of task interference. Figure 6 As can be seen, when external task interference is low and GPU is available, this system (DynPipe) reduces time by approximately 10% compared to PipeDream. However, under heavy workloads (case 3), the time reduction increases to 40%. Due to training restarts, PipeDream-Re always requires a one-epoch delay before repartitioning, making its response to external interference slower than DynPipe.

[0093] Dynamic repartitioning: Figure 7 and Figure 8The time-to-accuracy and average iteration time are presented under different levels of task interference. Performance degrades significantly with task interference; in PipeDream, the iteration time per update increases by 1.2 to 3 times. In contrast, our system can quickly re-establish load balancing, reducing the time by 39% to 67%, which equates to an overall reduction in convergence time (CT) of 25% to 64% under heavy task interference. For systems without random forests, the overall CT is reduced by 10% to 41%. For PipeDream-Re, we observe a CT reduction of 10% to 33%, demonstrating our invention's ability to seamlessly redeploy pipeline training in dynamic environments.

[0094] Ablation Study: Pipeline Checkpointing Mechanism: This system minimizes checkpoint time by pipelined the serialization and distribution stages. Without the pipeline mechanism, the number of iterations required for repartitioning across different models would increase by 8 to 15 times, highlighting the crucial role of the pipeline mechanism in maintaining the efficiency of this system.

[0095] Flow control: The distribution of checkpoints competes with training traffic for limited network bandwidth. Without priority-based flow control, the number of iterations required for each repartition can increase by 61% to 165%.

[0096] This invention considers the design and application of three aspects: a model partitioning scheme for asynchronous training convergence, a lightweight random forest model evaluation, and transparent layer transfer. The system's underlying layer employs a model partitioning method based on dynamic programming, enabling optimization of training performance in dynamic computing environments. The system also includes a non-intrusive interference evaluation model based on a lightweight random forest, which effectively predicts the impact of external task interference on training efficiency and adaptively adjusts the model partitioning plan when interference occurs. Furthermore, the system introduces a transparent layer transfer mechanism, achieving seamless repartitioning of pipeline training without interrupting the training workflow. Through these innovations, this invention can significantly improve the training efficiency of deep neural networks and shorten model convergence time in complex distributed environments.

[0097] Example 2

[0098] This application also relates to an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0099] The electronic device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor performs various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory.

[0100] The relevant technical solutions are the same as above, and will not be repeated here.

[0101] Example 3

[0102] This application also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0103] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0104] The relevant technical solutions are the same as above, and will not be repeated here.

[0105] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for distributed parallel training of a deep neural network in a dynamic computing environment, the method comprising: The method comprises the following steps: S1, determining a deep neural network model segmentation scheme by minimizing the actual convergence time of an asynchronous pipeline, and distributing each model slice of the deep neural network model corresponding to the segmentation scheme to a corresponding device node, wherein the actual convergence time of the asynchronous pipeline is the product of the upper limit of the iteration number corresponding to a preset convergence error and the optimal running time of the pipeline required for a single iteration; each device node is configured with a trained lightweight random forest model for evaluating the interference effect of additional load on the model slice of the device node; S2, after starting the training of the deep neural network model, real-time acquisition of the interference coefficient corresponding to the interference effect predicted by the lightweight random forest model on each device node, identification of the most important first k layers of the deep neural network model affecting the training efficiency based on the communication and calculation time of each layer of the deep neural network model in its current computing environment, merging of a preset number of adjacent layers of each layer in the first k layers as a logical unit layer in the model slice planning, obtaining a model simplified structure, determining the input parameters required by the traversal algorithm based on the model simplified structure, taking the interference coefficient as an input parameter of the traversal algorithm, and obtaining a new deep neural network model segmentation scheme through the traversal algorithm; if the current configured deep neural network model segmentation scheme is different from the new deep neural network model segmentation scheme, dynamically adjusting the load distribution between device nodes by gradually migrating the calculation tasks of part of the layers in the idle time period of the training to realize the configuration of the new model segmentation scheme, repeating the step S2 until the distributed parallel training of the deep neural network under the dynamic computing environment is completed; the way to minimize the actual convergence time of the asynchronous pipeline is: In the formula, S is the number of model slices; the optimal running time of the pipeline required for a single iteration, the total number of GPUs in all device nodes; a preset convergence error; a learning rate required when training the neural network model; a smoothing constant; a second moment; a gradient variance.

2. The distributed parallel training method of claim 1, wherein, The lightweight random forest model on each device node is used to predict the interference coefficient based on the model slice features and device statistical data on the device node.

3. The distributed parallel training method of claim 2, wherein, The model slice features include flop calculation number, layer features and network demand degree; the device statistical data include GPU utilization, data bus throughput, GPU memory usage and PCIe data transmission rate.

4. The distributed parallel training method of claim 2, wherein, The interference coefficient is the ratio of the running time of a single model slice under the same additional load interference condition to the baseline running time of the model slice under the non-interference condition.

5. The distributed parallel training method of claim 1, wherein, The way to dynamically adjust the load distribution between device nodes by gradually migrating the calculation tasks of part of the layers is: Serializing the state information of each layer to be migrated into a data stream, and storing the data stream in the CPU memory of the local device node where the layer to be migrated is located; Dividing the migration task into multiple small independent tasks, each independent task being responsible for the migration of a part of layer data, and the independent tasks being executed in parallel, and in the idle time period of each iteration training in the subsequent training iteration, the independent tasks are controlled to gradually distribute the serialized data stream to the target device node for migration.

6. The distributed parallel training method of claim 5, wherein, The state information of each layer to be migrated includes layer parameters, random seeds and optimizer states.

7. The distributed parallel training method of claim 5, wherein, The migration of layer data flow is realized in the following way: a priority queue is introduced in the network stack of the operating system through the priority flow control protocol, the training data flow is given a higher priority, and the migration data flow is marked as a low priority to avoid the conflict between the migration data flow and the normal training data flow in the network transmission process. 8.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored computer program, wherein the computer program, when executed by a processor, controls the device in which the storage medium is located to perform the steps of the method of any one of claims 1 to 7.

Images