Distributed parallel training method of deep neural network in dynamic computing environment

Abstract

The invention belongs to the related technical field of distributed system optimization and scheduling, and particularly relates to a distributed parallel training method of a deep neural network in a dynamic computing environment, which comprises the following steps of: determining a model segmentation scheme by minimizing the actual convergence time of an asynchronous pipeline, the model slices corresponding to the segmentation scheme are distributed to corresponding equipment nodes, and the actual convergence time is the product of the upper limit of the number of iterations corresponding to a preset convergence error and the optimal running time of the assembly line required by single iteration; each device node is configured with a trained lightweight random forest model which is used for evaluating the interference influence of the additional load located at the device node on the training efficiency of the model slice located at the device node. After training is started, interference coefficients corresponding to interference influences obtained by the random forest models on all equipment nodes are obtained in real time, a new segmentation scheme is obtained through model search space pruning and a traversal algorithm, and efficient training is achieved through layer migration.

Description

Technical Field

[0001] The present invention belongs to the technical field related to distributed system optimization and scheduling, and more specifically, to a distributed parallel training method for deep neural networks in a dynamic computing environment. Background Art

[0002] With the widespread application of deep neural networks in fields such as artificial intelligence, natural language processing, and computer vision, the scale and complexity of models are increasing. Training these large-scale neural networks usually needs to be carried out in a distributed environment. However, the transmission of large amounts of data during distributed training will lead to serious communication bottlenecks, which in turn affects the training efficiency. Especially in a dynamic computing environment, the interference of external tasks will further reduce the performance of the system. Therefore, how to optimize the use of communication and computing resources and improve training efficiency while ensuring model accuracy is an important issue that needs to be solved urgently. Summary of the invention

[0003] In response to the above defects or improvement needs of the prior art, the present invention provides a distributed parallel training method for deep neural networks in a dynamic computing environment, which aims to optimize the utilization of communication and computing resources and improve training efficiency while ensuring model accuracy.

[0004] To achieve the above object, according to one aspect of the present invention, a distributed parallel training method for a deep neural network in a dynamic computing environment is provided, comprising:

[0005] S1. Determine a deep neural network model segmentation scheme by minimizing the actual convergence time of the asynchronous pipeline, and distribute each model slice of the deep neural network model corresponding to the segmentation scheme to the corresponding device nodes, wherein the actual convergence time of the asynchronous pipeline is the product of the upper limit of the number of iterations corresponding to the preset convergence error and the optimal running time of the pipeline required for a single iteration; each device node is configured with a trained lightweight random forest model to evaluate the interference effect of the additional load located at the device node on the training efficiency of the model slice located at the device node;

[0006] S2. After starting the training of the deep neural network model, the interference coefficient corresponding to the interference impact predicted by the lightweight random forest model on each device node is obtained in real time, and based on the communication and computing time of each layer in the deep neural network model in its current computing environment, the most important first k layers that affect the training efficiency in the deep neural network model are identified, and a preset number of adjacent layers of each layer in the first k layers are merged with the layer as a logical unit layer in the model slicing planning to obtain a simplified model structure, and the input parameters required for the traversal algorithm are determined based on the simplified model structure, and the interference coefficient is used as an input parameter of the traversal algorithm to obtain a new deep neural network model slicing scheme through the traversal algorithm; if the currently configured deep neural network model slicing scheme is different from the new deep neural network model slicing scheme, the load distribution between the device nodes is dynamically adjusted by gradually migrating the computing tasks of some layers during the idle time period in the training to realize the configuration of the new model slicing scheme, and the S2 step is repeated until the distributed parallel training of the deep neural network in the dynamic computing environment is completed.

[0007] Furthermore, the way to minimize the actual convergence time of the asynchronous pipeline is:

[0008]

[0009] Where S is the number of model slices; W(S) is the optimal pipeline running time required for a single iteration; N is the total number of GPUs in all device nodes; ∈ is the preset convergence error; η is the learning rate required for neural network model training; L is the smoothing constant; ξ is the quadratic moment; f(w (0) ) is the gradient variance.

[0010] Furthermore, the lightweight random forest model on each device node predicts the interference coefficient based on the model slice features and device statistical data located on the device node.

[0011] Furthermore, the model slice characteristics include the number of flop calculations, layer characteristics and network requirements; the device statistics include GPU utilization, data bus throughput, GPU memory usage and PCIe data transfer rate.

[0012] Furthermore, the interference coefficient is the ratio of the running time of a single model slice under the same additional load interference condition on the same device node to the benchmark running time of the model slice under the interference-free condition.

[0013] Furthermore, the implementation method of gradually migrating computing tasks of some layers to dynamically adjust the load distribution between device nodes is as follows:

[0014] Serialize the state information of each layer to be migrated into a data stream, and store the data stream in the local CPU memory where the layer to be migrated is located;

[0015] The migration task is divided into multiple small independent tasks. Each independent task is responsible for migrating a part of the layer data, and each independent task is executed in parallel. In subsequent training iterations, during the idle time period of each iterative training, each independent task is controlled to gradually distribute the serialized data stream to the migration target device node.

[0016] Furthermore, the state information of each layer to be migrated includes layer parameters, random seeds, and optimizer status.

[0017] Furthermore, the migration of layer data streams is achieved in the following way: through the priority flow control protocol, a priority queue is introduced in the network stack of the operating system to give the training data stream a higher priority and mark the migration data stream as a low priority to avoid conflicts between the layer migration data stream and the normal training data stream during network transmission.

[0018] According to another aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the steps of the method described above.

[0020] In general, compared with the prior art, the technical solution conceived by the present invention has the following beneficial effects:

[0021] 1. The present invention proposes an end-to-end pipeline parallel distributed deep neural network training method in a dynamic environment based on asynchronous convergence analysis and task interference perception mechanism. The traditional deep neural network calculation-communication process is parallelized through a multi-stage asynchronous strategy to improve the utilization rate of GPU and network bandwidth; at the same time, the present invention introduces a model segmentation optimization algorithm based on asynchronous convergence analysis, which can reduce communication overhead while ensuring the training accuracy of the model; in addition, the method of the present invention also refers to a lightweight random forest model to evaluate the impact of task interference on training efficiency, and when task interference occurs, the model repartitioning plan is completed imperceptibly during the idle time period in training, thereby theoretically realizing the optimal end-to-end pipeline partitioning strategy, further improving the resource utilization efficiency in the training process.

[0022] 2. The present invention also proposes a transparent layer migration mechanism (i.e., a seamless model layer migration mechanism) to further optimize the model redeployment process in a dynamic environment. When task interference causes unbalanced load on certain GPUs, the workload is redistributed by migrating the computing tasks of some layers. Specifically, the state information of the layer to be migrated is first serialized, and these state data are packaged into checkpoint files and distributed to the specified GPU. The migration process adopts an asynchronous method, and by processing the layer migration operation in parallel with the idle time period in training, the impact of migration on the training process is effectively reduced, thereby achieving seamless layer migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flowchart of a distributed parallel training method for a deep neural network in a dynamic computing environment provided by an embodiment of the present invention;

[0024] Figure 2 This is a diagram of the architecture of a distributed parallel training system for a deep neural network in a dynamic computing environment provided by an embodiment of the present invention;

[0025] Figure 3 It is a schematic diagram of the model layer non-sensing migration process provided by an embodiment of the present invention;

[0026] Figure 4 is a schematic diagram of model search space pruning provided by an embodiment of the present invention;

[0027] Figure 5 1 is a comparison diagram of system training conditions in a steady-state environment provided by an embodiment of the present invention, wherein (a) is a comparison diagram of average single-iteration time, (b) is a comparison diagram of final training accuracy, and (c) is a comparison diagram of throughput;

[0028] Figure 6 is a schematic diagram of system sensitivity under a dynamic environment provided by an embodiment of the present invention;

[0029] Figure 7 This is a comparison diagram of end-to-end training in a dynamic environment provided by an embodiment of the present invention;

[0030] Figure 8 This is a comparison chart of average iteration time in a dynamic environment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0032] Embodiment 1

[0033] A distributed parallel training method for deep neural networks in a dynamic computing environment, such as Figure 1 As shown, including:

[0034] S1. Determine a deep neural network model segmentation scheme by minimizing the actual convergence time of the asynchronous pipeline, and distribute each model slice of the deep neural network model corresponding to the segmentation scheme to the corresponding device nodes, wherein the actual convergence time of the asynchronous pipeline is the product of the upper limit of the number of iterations corresponding to the preset convergence error and the optimal running time of the pipeline required for a single iteration; each device node is configured with a trained lightweight random forest model to evaluate the interference effect of the additional load located at the device node on the training efficiency of the model slice located at the device node;

[0035] S2. After starting the training of the deep neural network model, the interference coefficient corresponding to the interference impact predicted by the lightweight random forest model on each device node is obtained in real time, and based on the communication and computing time of each layer in the deep neural network model in its current computing environment, the most important first k layers that affect the training efficiency in the deep neural network model are identified, and a preset number of adjacent layers of each layer in the first k layers are merged with the layer as a logical unit layer in the model slicing planning to obtain a simplified model structure, and the input parameters required for the traversal algorithm are determined based on the simplified model structure, and the interference coefficient is used as an input parameter of the traversal algorithm to obtain a new deep neural network model slicing scheme through the traversal algorithm; if the currently configured deep neural network model slicing scheme is different from the new deep neural network model slicing scheme, the load distribution between the device nodes is dynamically adjusted by gradually migrating the computing tasks of some layers during the idle time period in the training to realize the configuration of the new model slicing scheme;

[0036] S3. Repeat step S2 until the distributed parallel training of the deep neural network in the dynamic computing environment is completed.

[0037] This embodiment proposes an end-to-end pipeline parallel distributed deep neural network training method in a dynamic environment based on asynchronous convergence analysis and task interference perception mechanism. The traditional deep neural network calculation-communication process is parallelized through a multi-stage asynchronous strategy to improve the utilization rate of GPU and network bandwidth; at the same time, this embodiment introduces a model segmentation optimization algorithm based on asynchronous convergence analysis to reduce communication overhead while ensuring the training accuracy of the model; in addition, the method of this embodiment also refers to a lightweight random forest model to evaluate the impact of task interference on training efficiency, and when task interference occurs, the model repartitioning plan is completed imperceptibly during the idle time period in training, thereby theoretically realizing the optimal end-to-end pipeline partitioning strategy and further improving the resource utilization efficiency during the training process.

[0038] Regarding the model partitioning algorithm based on asynchronous convergence analysis, during the distributed training process, by analyzing the model lag at each stage and adjusting the model partitioning strategy, the computational load balance at each stage is ensured, the communication delay is reduced, and the training time is minimized while maintaining the convergence efficiency. As a preferred implementation, the degree of model convergence after quantification and the time spent on actual model training are:

[0039]

[0040] Where S is the number of model slices; W(S) is the optimal pipeline running time required for a single iteration; N is the total number of GPUs in all device nodes; ∈ is the preset convergence error; η is the learning rate required for neural network model training; L is the smoothing constant; ξ is the quadratic moment; r(w (0) ) is the gradient variance.

[0041] In actual operation, the optimal segmentation solution can be obtained by traversing all the stage numbers (that is, the value of S).

[0042] About an end-to-end pipeline parallel training model segmentation scheme based on convergence analysis. Specifically, it aims to solve the problem of non-optimal segmentation scheme and training efficiency caused by the mismatch of computing communication models faced by traditional asynchronous pipeline models in the process of deep neural network training. In the asynchronous pipeline parallel training model, how to segment the model is one of the key factors that determine the training performance. The goal of model segmentation is to divide a complete deep neural network into several parts (i.e., stages) and assign these parts to different computing nodes (such as GPUs) for parallel computing. However, simply segmenting according to the computing load often cannot achieve the optimal effect, especially in asynchronous training, where communication overhead and convergence speed are also important factors affecting training efficiency. To this end, this embodiment proposes an optimization strategy based on detailed convergence analysis and precise pipeline feature characterization to achieve the optimal model segmentation scheme. First, by analyzing the three main stages of pipeline training-the warm-up stage, the iterative update stage, and the termination stage, the key factors affecting the training efficiency are clarified. The warm-up stage refers to the process in which each stage of the model enters the computing state in turn. This stage is usually short and has little impact on the overall training time. The iterative update phase is the core part of the training, which occupies most of the time of the entire training process. In this phase, each stage repeatedly performs forward propagation and back propagation calculations. The termination phase is the final stage of the training, which is mainly used to synchronize the calculation results of all stages and complete the final model update. In the iterative update phase, the determining factors of the training time include the calculation time and communication time of each iteration. During the design process, special attention was paid to the precise characterization of the communication overhead. Traditional pipeline parallel training models usually assume that the communication time of the calculation stage can be completely covered by the calculation time, but in practical applications, this assumption is often not true. To this end, the communication time between each calculation stage was measured and analyzed in detail, and the segmentation scheme of the entire pipeline was optimized based on this. Through a reasonable model segmentation strategy, the present invention effectively reduces the non-overlapping part of the communication between the stages, further reducing the total training time.

[0043] To facilitate convergence analysis, several widely used assumptions are introduced: (1) the gradient of the loss function is L-smooth; (2) the stochastic gradient is bounded, that is, (3) The variance of the stochastic gradient on the training sample x is bounded, that is, Assuming ηL≤1, it satisfies:

[0044]

[0045] The above formula shows that, despite the existence of Term, the convergence error is still affected by the number of update iterations T and the number of stages S. Therefore, although model partitioning can reduce the time of each iteration, it often leads to a decrease in statistical performance (i.e., convergence performance, convergence error). Referring to the convergence error, it is assumed that when the error drops to a specified minimum threshold ∈, the pipeline training reaches convergence. Therefore, the upper limit of the number of update iterations T required for pipeline training to reach the specified convergence threshold ∈ is:

[0046]

[0047] Finally, the actual convergence time of the asynchronous pipeline can be expressed as:

[0048]

[0049] The convergence quantification (convergence error) of each stage is integrated into the model partitioning scheme (the optimal running scheme under S), and the trade-off between running time and convergence is comprehensively considered (W(S) is the optimal running time of the pipeline (single iteration time) when the model is divided into S model slices, and N is the number of GPUs). This scheme not only maximizes the utilization of hardware resources, but also effectively reduces the convergence problem caused by stage delay in an asynchronous environment.

[0050] The end-to-end pipeline parallel training model segmentation scheme based on convergence analysis proposed in this embodiment not only significantly improves the training speed of deep neural networks through asynchronous computing and communication processing, detailed convergence analysis, optimized model segmentation strategy and accurate communication feature characterization, but also maximizes the utilization efficiency of hardware resources while ensuring model convergence. In a steady-state computing environment, the algorithm realizes an efficient and stable training process. The combination of these innovative technologies enables the present invention to provide strong technical support when processing large-scale deep learning tasks, greatly promoting the efficiency and performance improvement of deep neural network training.

[0051] Regarding the lightweight random forest model, the model non-invasively predicts the impact of external task interference on training efficiency by evaluating runtime statistics such as GPU status, network bandwidth, and data bus status at each stage, thereby dynamically adjusting the model partitioning plan, further reducing the decline in training efficiency caused by task interference, and ensuring efficient training in a dynamic computing environment. As a preferred implementation, the lightweight random forest model on each device node predicts the interference coefficient based on the model slice characteristics and device statistical data located on the device node.

[0052] As a preferred implementation, model slicing characteristics include flop calculation number, layer characteristics and network demand; device statistics include GPU utilization, data bus throughput, GPU memory usage and PCIe (ie, system bus) data transmission rate.

[0053] As a preferred implementation, the interference coefficient is the ratio of the running time of a single model slice under the same additional load interference condition on the same device node to the benchmark running time of the model slice under the interference-free condition.

[0054] S2 in the above implementation method can be regarded as a lightweight task interference perception and adjustment mechanism based on random forest, which is used to evaluate the interference effect of external tasks on the deep neural network training process in real time and dynamically optimize the partitioning strategy of the training model. In the current complex distributed computing environment, deep neural network training tasks are often interfered by various external factors, such as resource competition between other tasks executed in parallel and training tasks, fluctuations in network bandwidth, dynamic allocation of GPU resources, and insertion of burst tasks. These interference factors may cause the load of some computing nodes to suddenly increase or decrease, thereby destroying the balance of the entire training process, affecting the efficiency of training and the convergence speed of the model. In order to meet these challenges, a random forest-based interference perception model is designed, which can monitor and evaluate the impact of these external interferences in real time during runtime, and intelligently adjust the model partitioning strategy based on the evaluation results to minimize the negative impact of interference on the training process.

[0055] The random forest model is designed as a lightweight, non-intrusive monitoring tool that can accurately sense and predict the impact of task interference without significantly increasing the system burden. First, the system continuously collects various runtime statistics, including but not limited to the following aspects: GPU utilization, network bandwidth, data bus throughput, forward propagation and backpropagation time of each computing stage, GPU memory usage, PCIe data transfer rate, and execution time of each task. These data provide the model with comprehensive operating environment information, allowing the random forest model to accurately capture changes in the current computing environment. During the design and implementation phase of the system, the random forest model was built by offline training on a self-generated dataset. The dataset covers the actual running data of typical deep learning training tasks under different interference conditions. Specifically, the training process includes constructing a training sample set and training optimization model parameters. The training sample set is constructed by first capturing the standard running time of a single GPU training task under interference-free conditions. Then, these training tasks are exposed to these interferences by simulating external interference conditions of various intensities. By comparing the running time of each training task under these interference conditions with the baseline running time of the training task under the interference-free condition, the actual impact of the interference on the training task can be obtained. This actual impact becomes the "true value" of the training model and is used to guide the learning of the random forest model. Using this data, the model takes the GPU state (such as SM activity, DRAM throughput, etc.) and task characteristics as input features, and optimizes the prediction ability of the random forest model through repeated training and verification, so that it can efficiently and accurately evaluate the impact of interference in real-time applications. In actual applications, when the system detects new tasks or changes in the environment, the random forest model will be activated to evaluate the adaptability of the current model partitioning strategy. By analyzing the working status of the current computing nodes in real time, the model can quickly identify nodes that may become bottlenecks due to external interference. These bottleneck nodes are usually manifested as a sudden increase in GPU utilization, increased data transmission latency, or significantly extended forward / backward propagation time. Based on these real-time runtime data, the model can predict whether the current partitioning strategy is still optimal and recommend adjustments to the strategy when necessary. Adjustments may include reallocating computing tasks, such as transferring certain computing-intensive tasks from overloaded nodes to less loaded nodes, or optimizing data transmission paths to reduce communication delays caused by network bandwidth bottlenecks. In addition, the system may also adjust the model's parallelism or change partition boundaries to ensure a more balanced workload on each computing node, thereby improving overall training efficiency.

[0056] A significant advantage of the random forest model is its non-intrusive design. It does not significantly interfere with the ongoing training process, nor does it require stopping training tasks or interrupting the training process. Instead, it runs seamlessly in the background, continuously monitoring the system status and taking immediate action when problems are detected. This non-intrusiveness enables the system to make dynamic adjustments without affecting the stability of training, thereby ensuring the efficiency and continuity of the training process. In addition, the random forest model can continuously optimize its prediction capabilities based on feedback from actual operations. Over time, the system can accumulate more operating data, and use this data to further train and optimize the random forest model, so that it can show higher adaptability and accuracy when facing different types of interference. This adaptive learning mechanism ensures that the system can always operate efficiently in a complex and changing computing environment.

[0057] Through this random forest-based task interference perception and adjustment mechanism, the present invention achieves accurate perception and intelligent response to external task interference, which not only greatly improves the efficiency of deep neural network training, but also effectively reduces the negative impact of external interference on system performance. The flexibility and adaptability of this mechanism enable it to achieve continuous optimization in a complex computing environment, thereby ensuring the efficiency, stability and robustness of training tasks.

[0058] This embodiment adopts the 1F1B scheduling strategy, and fully utilizes the computing power and network bandwidth of the GPU by dynamically adjusting the pipeline partitioning strategy, thereby improving the training efficiency of the distributed deep neural network. The core idea of ​​the technology is to use a lightweight random forest model to evaluate and minimize the impact of task interference during the training process, adjust the model partitioning plan in real time to adapt to the dynamically changing computing environment, and prune the model segmentation plan space through model representation compression to ensure the stability and efficiency of the training process.

[0059] As a preferred implementation, a transparent layer migration mechanism (i.e., a seamless model layer migration mechanism) is introduced to further optimize the process of model redeployment in a dynamic environment. When task interference causes unbalanced load on certain GPUs, the workload is redistributed by migrating the computing tasks of some layers. Specifically, the layer states that need to be migrated (such as layer parameters, random seeds, and optimizer states) are first serialized, and these state data are packaged into checkpoint files and distributed to the specified GPU. The migration process adopts an asynchronous approach, and by processing the layer migration operation in parallel with the idle time period in training, the impact of migration on the training process is effectively reduced, thereby achieving seamless connection of layer migration. This transparent layer migration mechanism ensures that the system can maintain the optimization of training efficiency and model convergence speed even in a high-load interference environment.

[0060] The transparent layer migration mechanism is proposed to deal with the problem of unbalanced load of computing nodes caused by external task interference or uneven resource utilization during the training of distributed deep neural networks. Through this mechanism, the system can redistribute computing tasks without affecting the training process, thereby rebalancing the resource utilization of the system and ensuring the efficiency and stability of the training process. In a complex distributed computing environment, the training tasks of deep neural networks are often affected by various external interferences. For example, factors such as the sudden increase of other parallel tasks, fluctuations in network bandwidth, or dynamic scheduling of GPU resources may cause the load of certain computing nodes to increase sharply, thus becoming a bottleneck for the entire training process. Traditional solutions usually require pausing the training task and then manually reconfiguring the computing resources, which is not only time-consuming, but also interrupts the training process, resulting in a slowdown in model convergence. In order to solve these problems, the preferred method of this embodiment proposes a transparent layer migration mechanism that can dynamically adjust the load distribution between device nodes without interrupting the training task. The mechanism embeds the layer migration operation into the idle cycle (such as the time gap between the calculation stages) during the training process through asynchronous processing, thereby minimizing interference with the training process and achieving seamless migration and repartitioning.

[0061] Specifically, the above layer migration mechanism includes the following key steps:

[0062] (1) Model search space pruning: First, the system will process the deep neural network model in blocks, merge multiple computing layers into a logical unit to form a block, so as to reduce the amount of data that needs to be processed during the migration process. To determine which layers should be migrated, the system will identify the key layers that have the greatest impact on the overall performance based on the computation time, communication overhead and other key features of each layer. These key layers are often the most computationally intensive parts. Migrating these layers can significantly balance the computational load and improve the overall performance of the system. (2) The traversal algorithm determines the new partitioning scheme, that is, repartitioning. (3) Serialization and distribution of layer states: After determining the layers that need to be migrated, the system will serialize the states of these layers (including model parameters, random seeds, optimizer states, etc.) into data streams and store these data streams in the local CPU memory. To ensure the efficiency of the migration process, the system divides the migration task into multiple small independent tasks, each of which is responsible for migrating a part of the data. These tasks can be executed in parallel, thereby accelerating the migration process. Asynchronous snapshot and transmission: During the migration process, the system adopts an asynchronous snapshot mechanism to process the snapshot of the layer state separately from the transmission operation (serialization and transmission). Specifically, the system will gradually distribute the serialized data stream to the migration target node in the subsequent training iterations. Since these operations are performed asynchronously, the normal training process will not be blocked, and the migration operation can be performed without affecting the overall performance of the system. Through this pipeline processing method, the system can efficiently complete the layer migration task. As a further preferred embodiment, (4) migration flow control: During the network transmission process, in order to avoid the conflict between the layer migration data flow and the normal training data flow, the system designs a priority flow control protocol. The protocol introduces a priority queue in the network stack of the operating system, and the training data flow is given a higher priority, while the migration data flow is marked as a low priority. In this way, while the network resources give priority to the training data, they can still ensure the smooth transmission of the migration data. The system will dynamically adjust the queue size of the migration data to adapt to the current network bandwidth conditions, so as to complete the migration task while minimizing the interference to the training data flow. Through the above steps, the method of this embodiment can dynamically adjust the load distribution between the computing nodes without affecting the training task, and repartition the model when necessary. The repartitioning process is also transparent, and there is no need to pause the training task or restart the training process. In this way, the system can respond quickly when encountering external interference and ensure the continuous and efficient operation of training tasks by adjusting the allocation of computing resources.

[0063] The above transparent layer migration mechanism not only improves the robustness of deep neural network training, but also significantly improves the system's adaptability in complex computing environments. By reducing training interruption time and optimizing resource utilization, the method of this embodiment can always maintain an efficient and stable training process in the face of a dynamically changing computing environment, greatly improving the training speed and convergence efficiency of the model.

[0064] In actual implementation, it can be summarized into the following key steps:

[0065] 1. First, the system starts a Pipeline Planner ( Figure 2 ), which performs a detailed analysis of each layer of the deep neural network model, recording the time of forward propagation and backpropagation as well as the size of gradients and activations. Based on this data, Pipeline Planner runs Algorithm 1 to develop an optimized model partitioning scheme that strikes a balance between hardware speed and statistical convergence. After being determined, the various stages of the model are assigned to the designated computing nodes and begin pipeline training under the 1F1B strategy.

[0066] 2. Runtime Scheduler Figure 2 )A RuntimeScheduler is launched on each GPU worker node to manage communication and computation tasks. The scheduler is responsible for the transmission of intermediate activations: during forward propagation, it receives input from the previous stage and sends the resulting activations; during backpropagation, it processes incoming gradients and ensures that they are accurately distributed to the designated worker nodes. In addition, the scheduler also records runtime statistics for each stage and worker node, such as forward and backward propagation time for each layer, SM activity of the GPU, DRAM reception throughput, etc. This information is the basis for dynamic model repartitioning and is pushed to the Global Manager regularly with minimal interference.

[0067] 3.Global Manager( Figure 2 )’s role is to consolidate information from all Runtime Schedulers and formulate a new partitioning plan through a lightweight repartitioning algorithm (Algorithm 2). This recalibrated plan is then communicated back to each scheduler with the goal of rebalancing the communication and computational load between GPU worker nodes to adapt to the interference of external tasks. The Global Manager, by co-locating with the last stage, avoids the need to create an additional communication process, since the collection of runtime statistics can be achieved through the transmission of forward activations.

[0068] 4.Migration Module( Figure 2) handles layer migration across stages and reconfiguration of batch sizes within stages. To enable seamless model repartitioning in the event of unexpected outages, it maintains a comprehensive deep neural network model graph, covering network structure declarations and tensor transfer processes. During the repartitioning process, the module rewrites the graph structure description and encapsulates layer states (such as parameters, random seeds, optimizer states) into checkpoints (asynchronous snapshot mechanism), which are then distributed to the affected stages based on the differences between the new and old partitioning plans. The system divides layer migration into serialization and distribution stages ( Figure 3 ), minimizing migration time by pipeline parallel serialization and transmission. In addition, the system implements a priority-based flow control protocol to manage the output of training and checkpoint traffic to reduce network contention.

[0069] Through the above modules, the efficiency of deep neural network training can be significantly improved in a dynamic computing environment, ensuring that the system can respond quickly when faced with external interference and continue to operate in an efficient and stable manner.

[0070] The generation of the end-to-end pipeline parallel training model segmentation scheme based on convergence analysis is shown in Algorithm 1:

[0071]

[0072] The specific steps and corresponding operations of Algorithm 1 are as follows:

[0073] 1. Initialize parameters: Set the number of GPUs (N), the number of model layers (M), the model configuration file (including the time for forward propagation and backpropagation), and the network bandwidth. This corresponds to line 1 of the algorithm.

[0074] 2. The outer loop iterates over the number of partitions For different numbers of partitions (S) (from 1 to N), the algorithm starts iterating over the possible numbers of partitions. This corresponds to line 2 of the algorithm.

[0075] 3. Calculate the number of iterations: Calculate the number of iterations required for the current number of partitions (S) and initialize a temporary record set (Rec') for saving the current partition scheme. This corresponds to lines 3-4 of the algorithm.

[0076] 4. Inner loop traverses partition points: For each possible partition point (r) (from 1 to N), the algorithm calls the dynamic programming algorithm to calculate the corresponding partition scheme (Partition') and its corresponding iteration time (W(M,S,r,N)), and adds the calculation results to the temporary record set (Rec'). This corresponds to lines 5-7 of the algorithm.

[0077] 5. Select the best partition time: For the current number of partitions (S), select the minimum iteration time from the temporary record set as the best solution W(S) for the current number of partitions, and calculate the total training time WallClockTime(S) under the current number of partitions, and then save it together with the corresponding partition solution to the record set (Rec). This corresponds to lines 8-11 of the algorithm.

[0078] 6. Finally select the best solution: After traversing all the partition numbers, select the minimum total training time (WallClockTime) as the final best partitioning solution and return it. This corresponds to lines 12-14 of the algorithm.

[0079] Through the above steps, the algorithm gradually explores different partitioning schemes and iteratively adjusts the partition boundaries to find the optimal partitioning scheme that can achieve the highest training efficiency in a specific computing environment.

[0080] The re-segmentation algorithm is shown in Algorithm 2:

[0081]

[0082] Algorithm 2 describes the model repartitioning process in detail, which mainly includes the following key steps:

[0083] 1. Model search space pruning (Lines 3-6, Figure 4 ): The algorithm first identifies the top k most important layers in the deep neural network (DNN), which are determined based on their communication and computation time under the current computing environment. These key layers play an important role in the entire model and directly affect the computational efficiency and communication overhead of the model. After identifying the key layers, the algorithm gradually merges consecutive layers in the model into larger blocks by combining these key layers with their adjacent layers. This process is called model search space pruning, and its purpose is to simplify the model structure so that subsequent partitioning operations can be performed more efficiently. By combining layers into blocks, the algorithm reduces the complexity of partitioning and ensures that the layers within each block have similar computational and communication characteristics, which can better perform partitioning optimization.

[0084] 2. Adjusting Pipeline Partitions (Lines 7-15): After the model search space is pruned, the algorithm enters the partition adjustment phase. At this point, the simplified model structure has been formed, and each block represents a group of layers with similar computational characteristics. The algorithm fully explores different partitioning schemes by traversing all possible partition boundaries. Each partitioning scheme needs to determine the most effective partition boundary and select a load-balancing batch size suitable for parallelization within the pipeline. To evaluate each potential repartitioning scheme, the algorithm introduces a customized random forest model. The model is used to predict the performance of these partitioning schemes when running with interfering tasks on a specified GPU. The random forest model can accurately predict the computational performance under different partitioning schemes based on the input model structure and the characteristics of the current computing environment. This prediction is based on historical data and the specific configuration of the model, which can help the algorithm better select the optimal partitioning scheme. Finally, the algorithm uses an evaluation simulator to comprehensively evaluate all potential partitioning schemes by combining various performance indicators. These performance indicators include computational efficiency, communication overhead, training speed, etc. Through comprehensive analysis, the algorithm can identify the partitioning scheme that can best improve the overall training efficiency and ultimately select the optimal model repartitioning scheme.

[0085] 3. Final output: After the above steps, the algorithm finally outputs an optimal partitioning scheme that can achieve the highest training efficiency in the current computing environment. This scheme rationally distributes the various parts of the model to different computing nodes to ensure load balance and efficient use of computing resources.

[0086] In summary, Algorithm 2 provides an effective model repartitioning method through model search space pruning, partition boundary exploration and performance evaluation. This method can dynamically adjust the model partitioning strategy according to the current computing environment and interference tasks, thereby optimizing the training efficiency and stability of deep neural networks.

[0087] The end-to-end pipeline parallel training model segmentation scheme based on convergence analysis, the lightweight random forest model, and the layer migration mechanism can act independently or together on distributed deep neural network training, accelerating the distributed deep neural network training in stable / dynamic environments. The experiment evaluated the training of three typical deep neural network models, including BERT-48 (680M parameters), VGG-16 (110M parameters), and GPT-2 1.7B (1.7B parameters). Among them, BERT-48 and GPT-21.7B are trained on the BookCorpus text dataset, while VGG-16 is trained on the ImageNet-1K and CIFAR-10 image datasets. The effectiveness of the present invention for distributed deep neural network training will be explained below from the performance in both stable and dynamic environments.

[0088] First, the comparison results are shown in a stable computing environment where external interference can be ignored.

[0089] Reduction in Convergence Time (CT): The system (DynPipe) achieves convergence in a shorter time, 1.07 to 2.78 times faster than PipeDream and 1.496 to 3.375 times faster than Pipeline-1F1B, while maintaining comparable test accuracy or global loss. In addition, when applied to the pipeline architecture DAPPLE, the system's end-to-end optimizations speed up training by 1.11 to 2.55 times on various models, such as Figure 5 This shows the versatility of the system in synchronous training scenarios. Figure 5 In (b), our system consistently outperforms the baseline, with CT reductions ranging from 7% to 76% without compromising model performance.

[0090] Training Throughput: Figure 5 (c) in Figure 2 shows the Figure 5 (b) Average training throughput under the same settings. On the image dataset, compared with PipeDream and Pipeline-1F1B, our system improves the throughput by 60.6% to 83.7% and 280.1% to 423.5%, respectively. For BERT-48 and GPT-2 1.7B on the text dataset, the throughput is improved by 6.8% to 15.4% and 55.2% to 297.6%.

[0091] Our system is able to adapt to dynamic task interference. To verify this, we introduce external DNN training tasks to simulate real-world interference in production clusters.

[0092] Sensitivity to distraction: Train VGG-16 on the CIFAR-10 dataset to obtain the per-iteration time in the presence of task distractors. Figure 6 It can be seen that when the interference of external tasks is small and the GPU is used, this system (DynPipe) reduces the time by about 10% compared with PipeDream. However, when the workload is large (Case 3), the time reduction rises to 40%. Due to the restart of training, PipeDream-Re always needs to postpone one epoch before repartitioning, making it respond more slowly to external interference than DynPipe.

[0093] Dynamic repartitioning: Figure 7 and Figure 8The time (time-to-accuracy) and average iteration time required for the model to reach the target accuracy under different levels of task interference are shown respectively. When encountering task interference, the performance drops significantly, and under PipeDream, the iteration time for each update is extended by 1.2 to 3 times. In contrast, the system is able to quickly re-establish load balance, reducing the time by 39% to 67%, which is equivalent to an overall reduction of the convergence time (CT) by 25% to 64% under heavy task interference. For system solutions that do not use random forests, the overall CT is reduced by 10% to 41%. For PipeDream-Re, we observed a 10% to 33% reduction in CT, which demonstrates the ability of the present invention to seamlessly redeploy pipeline training in a dynamic environment.

[0094] Ablation Study: Pipeline Checkpointing Mechanism: Our system minimizes the time to checkpoint by pipelineizing the serialization and distribution stages. Without the pipelining mechanism, the number of iterations required to repartition across different models increases by 8 to 15 times, highlighting the important role of the pipelining mechanism in maintaining the efficiency of our system.

[0095] Traffic control: Checkpoint distribution competes with training traffic for limited network bandwidth. Without priority-based traffic control, the number of iterations required for each repartitioning can increase by 61% to 165%.

[0096] The present invention considers the design and application of three aspects: model segmentation scheme for asynchronous training convergence, lightweight random forest model evaluation, and transparent layer migration. The underlying system adopts a model partitioning method based on dynamic programming, which can optimize training performance in a dynamic computing environment; the system also includes a non-intrusive interference assessment model based on lightweight random forest, which can effectively predict the impact of external task interference on training efficiency and adaptively adjust the model partitioning plan when task interference occurs; in addition, the system introduces a transparent layer migration mechanism, which realizes seamless repartitioning of pipeline training without interrupting the training workflow. Through these innovations, the present invention can significantly improve the training efficiency of deep neural networks in complex distributed environments and shorten the convergence time of the model.

[0097] Embodiment 2

[0098] The present application also relates to an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0099] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The memory may be used to store computer programs and / or modules, and the processor may perform various functions of the electronic device by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory.

[0100] The relevant technical solutions are the same as above and will not be described again here.

[0101] Embodiment 3

[0102] The present application also relates to a computer-readable storage medium having a computer program stored thereon, and the steps of the above method are implemented when the computer program is executed by a processor.

[0103] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0104] The relevant technical solutions are the same as above and will not be described again here.

[0105] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed parallel training method for deep neural networks in a dynamic computing environment, characterized in that: include: S1. Determine a deep neural network model segmentation scheme by minimizing the actual convergence time of the asynchronous pipeline, and distribute each model slice of the deep neural network model corresponding to the segmentation scheme to the corresponding device nodes, wherein the actual convergence time of the asynchronous pipeline is the product of the upper limit of the number of iterations corresponding to the preset convergence error and the optimal running time of the pipeline required for a single iteration; each device node is configured with a trained lightweight random forest model to evaluate the interference effect of the additional load located at the device node on the training efficiency of the model slice located at the device node; S2. After starting the training of the deep neural network model, the interference coefficient corresponding to the interference impact predicted by the lightweight random forest model on each device node is obtained in real time, and based on the communication and computing time of each layer in the deep neural network model in its current computing environment, the most important first k layers that affect the training efficiency in the deep neural network model are identified, and a preset number of adjacent layers of each layer in the first k layers are merged with the layer as a logical unit layer in the model slicing planning to obtain a simplified model structure, and the input parameters required for the traversal algorithm are determined based on the simplified model structure, and the interference coefficient is used as an input parameter of the traversal algorithm to obtain a new deep neural network model slicing scheme through the traversal algorithm; if the currently configured deep neural network model slicing scheme is different from the new deep neural network model slicing scheme, the load distribution between the device nodes is dynamically adjusted by gradually migrating the computing tasks of some layers during the idle time period in the training to realize the configuration of the new model slicing scheme, and the S2 step is repeated until the distributed parallel training of the deep neural network in the dynamic computing environment is completed.

2. The distributed parallel training method according to claim 1, characterized in that: The way to minimize the actual convergence time of the asynchronous pipeline is: Where S is the number of model slices; W(S) is the optimal pipeline running time required for a single iteration; N is the total number of GPUs in all device nodes; ∈ is the preset convergence error; η is the learning rate required for neural network model training; L is the smoothing constant; ξ is the quadratic moment; f(w (0) ) is the gradient variance.

3. The distributed parallel training method according to claim 1, characterized in that: The lightweight random forest model on each device node predicts the interference coefficient based on the model slice characteristics and device statistical data located on the device node.

4. The distributed parallel training method according to claim 3, characterized in that: The model slice characteristics include the number of flop calculations, layer characteristics and network requirements; the device statistics include GPU utilization, data bus throughput, GPU memory usage and PCIe data transfer rate.

5. The distributed parallel training method according to claim 3, characterized in that: The interference coefficient is the ratio of the running time of a single model slice under the same additional load interference condition on the same device node to the benchmark running time of the model slice under the interference-free condition.

6. The distributed parallel training method according to claim 1, characterized in that: The implementation method of gradually migrating computing tasks of some layers to dynamically adjust the load distribution between device nodes is as follows: Serialize the state information of each layer to be migrated into a data stream, and store the data stream in the local CPU memory where the layer to be migrated is located; The migration task is divided into multiple small independent tasks. Each independent task is responsible for migrating a part of the layer data, and each independent task is executed in parallel. In subsequent training iterations, during the idle time period of each iterative training, each independent task is controlled to gradually distribute the serialized data stream to the migration target device node.

7. The distributed parallel training method according to claim 6, characterized in that: The status information of each layer to be migrated includes layer parameters, random seeds, and optimizer status.

8. The distributed parallel training method according to claim 6, characterized in that: The migration of layer data streams is achieved in the following way: a priority queue is introduced into the network stack of the operating system through a priority flow control protocol, the training data stream is given a higher priority, and the migration data stream is marked as a low priority to avoid conflicts between the layer migration data stream and the normal training data stream during network transmission.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the steps of the method according to any one of claims 1 to 8.

Images