Deep learning model training method suitable for network and equipment heterogeneous environment

Abstract

The invention relates to the technical field of model training, and particularly provides a deep learning model training method suitable for a network and equipment heterogeneous environment. The method comprises the following steps: designing an improved hybrid parallel strategy; according to the improved hybrid parallel strategy, constructing a model division and task placement scheme based on dynamic programming; according to an improved hybrid parallel strategy and a model division and task placement scheme based on dynamic planning, a self-adaptive weighted semi-synchronous hybrid training mechanism is designed, the method realizes reasonable distribution of communication loads, and the hardware utilization rate and the overall training efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of model training, and in particular, to a deep learning model training method applicable to a heterogeneous environment of networks and devices. Background Art

[0002] With the development of deep learning, especially the rise of large-scale pre-trained models, the number of parameters in the models has increased exponentially. Thanks to the stronger generalization ability brought by the increase in parameter scale, significant progress has been made in many fields such as question answering systems and image generation. Training such a large model places higher requirements on computing resources. Single-machine training faces double limitations of computing power and video memory capacity and can no longer meet the needs of modern large-scale model training. Therefore, distributed training has become an inevitable choice for large-scale model training. Existing distributed training strategies include hybrid parallel strategies such as data parallelism, pipeline model parallelism, and tensor parallelism. These methods have been integrated into frameworks such as Megatron, DeepSpeed, and Alpa and are widely used in model training practices. These frameworks mainly target high-performance homogeneous computing clusters, and due to cost and resource limitations, such high-performance clusters are often difficult to obtain for most enterprises and researchers.

[0003] Currently, when directly migrating the frameworks in the homogeneous environment to a complex heterogeneous environment, many problems still exist: First of all, existing frameworks usually assume that all devices have the same computing power, memory capacity, and network bandwidth, ignoring the heterogeneity between devices. As a result, high-performance devices are often dragged down by low-performance devices, unable to fully exert their performance, and the resource utilization efficiency is low. This not only wastes the computing power of high-performance devices but also significantly increases the training time. Most existing methods only focus on single-dimensional heterogeneity. For example, some methods focus on optimizing the communication efficiency in a heterogeneous network environment but ignore the differences in computing power of the devices themselves; some other methods focus on the computing optimization of heterogeneous devices but do not fully consider the heterogeneity of network bandwidth between devices. Such single-dimensional optimization strategies are difficult to adapt to the dual heterogeneous environment of network communication and computing devices at the same time, resulting in insufficient hardware utilization or low training efficiency; Secondly, the network bandwidths between heterogeneous devices are different, and simple model partitioning cannot adapt to the network topology characteristics, making it difficult to achieve reasonable distribution of communication loads; Finally, in the dual heterogeneous environment of networks and devices, network latency and bandwidth fluctuations are large. In hybrid parallel training, the speeds of the nodes that perform gradient synchronization in data parallelism may not be consistent. In frequent cross-node communications, for the nodes that are at a disadvantage in computing and transmission speeds, the faster nodes in data parallelism need to wait for the slower nodes to complete the calculation, significantly reducing the overall training efficiency.

[0004] In traditional gradient synchronization methods, both synchronous parallel methods and asynchronous parallel methods have certain limitations. Synchronous parallelism causes fast nodes to wait for slow nodes, wasting computing power; while asynchronous parallelism may affect the convergence and accuracy of the model due to outdated parameters, making it even less applicable to heterogeneous environments. Summary of the Invention

[0005] In view of this, the present invention provides a deep learning model training method applicable to network and device heterogeneous environments to achieve reasonable allocation of communication load, improve hardware utilization, and overall training efficiency.

[0006] In a first aspect, the present invention provides a deep learning model training method applicable to network and device heterogeneous environments, the method comprising: Step 1, design an improved hybrid parallel strategy; Step 2, construct a model partitioning and task placement scheme based on dynamic programming according to the improved hybrid parallel strategy; Step 3, design an adaptive weighted semi-synchronous hybrid training mechanism according to the improved hybrid parallel strategy and the model partitioning and task placement scheme based on dynamic programming.

[0007] Optionally, the Step 1 includes: Design an improved hybrid parallel strategy PatchPipe. Within the same stage, allocate corresponding batches according to the memory capacity and computing power of heterogeneous devices; between different stages, combine multiple low-end heterogeneous devices into a group, use data parallelism within the group, and use pipeline parallelism between groups to jointly process a complete training batch, achieving a balance in memory capacity with high-end devices; PatchPipe releases the memory of the activations generated by the current micro-batch forward propagation in advance by scheduling the micro-batch backpropagation in advance, so as to free up memory for the next injected micro-batch.

[0008] Optionally, the Step 2 includes: For the synchronous training strategy, use the iteration time of a single batch of data as the performance metric; minimize the single iteration time under the condition of considering all solution spaces of data parallelism and pipeline parallelism; In pipeline parallel training, divide the model into multiple stages and allocate them to different devices for processing in a pipeline manner; during training, record the total iteration time of a stage s in the pipeline as T , which includes 3 parts, namely the time at the start stage, the time during the execution stage, and the time of the final gradient synchronization stage; the expression of the total iteration time of stage s is: (1) Minimize the total iteration time of stage s, and the expression of its optimization objective is: (2) Among them, the time at the start stage of stage s , estimate it, and the expression is: (3) Among them, and are respectively the calculation time in the forward propagation process and the communication time sent to the next stage of the i th stage; Use All-Reduce for gradient synchronization. For the gradient synchronization time of stage s, its expression is: (4) Among them, is the device group for data parallelism, and the communication volume of the All-Reduce stage is , and the communication bandwidth depends on the minimum connection bandwidth among all devices; Next, determine the execution time of each stage stage, and define the stage with the fewest bubbles in the pipeline as the pivot stage; based on the execution time of the pivot stage, further derive the execution times of other stages; for the execution time of the pivot stage, its expression is: (5) Among them, M is the number of micro-batches processed in one iteration; and are respectively the calculation times of the forward and backward propagations of stage s; calculate the execution times of other stages through the execution time of the pivot stage, and its expression is: (6) Based on the above formula, obtain , , and , that is, obtain the iteration time of each stage. When i takes the value of s, their expressions are respectively:

[0009] Among them, is the batch size allocated to device d; and On the l th layer, when the batch size processed by device d is , the execution times of forward and backward propagation; and represent, respectively, the magnitudes of the activation values transmitted during the forward propagation of stage s and the gradients transmitted during the backward propagation when the batch size processed by device d is ; Denote as the latency of the parallel plan that divides the continuous neural network of L layers into S stages and maps them to N heterogeneous devices, with the goal of minimizing , and the expression is: (8) where has the property of optimal substructure. Decompose it into sub-problems, which consist of two parts: (1) An independent stage that contains l layers of neural network, placed on n devices, where data parallelism is used, and its time is represented by ; (2) An optimal sub-configuration which consists of the last layers of neural network, divided into stages and mapped to devices. The optimal time of this sub-problem is denoted as ; During the execution of the algorithm, record the division points of the model, the grouping strategy of devices, and the number of training samples assigned to devices, and finally backtrack the results to identify the optimal model division and task placement scheme.

[0010] Optionally, determine the pivot stage through iterative optimization: First, evaluate each stage as a candidate pivot stage in turn. For each candidate pivot stage, calculate the total iteration time of the entire training process; select the candidate pivot stage with the shortest iteration time as the pivot stage for this training; through the above iterative process, dynamically evaluate the impact of each stage as the pivot stage on the overall training time, and select the stage that minimizes the training iteration time.

[0011] Optionally, step 3 includes: Design a selective synchronization strategy in Adaptive Weighted Semi-Synchronous SSAW. For each iteration, the synchronization of gradients introduces a two-stage mechanism of local synchronization and global synchronization; local synchronization is a lightweight gradient synchronization operation for some nodes; in each iteration, the running states of each node are dynamically evaluated, and nodes that complete the current iteration quickly are preferentially selected to perform an All-Reduce operation once; after several rounds of local synchronization, SSAW will periodically perform a global synchronization operation, and the global synchronization forces all nodes to uniformly perform a complete All-Reduce, so that all nodes are realigned on the model parameters; at the same time, for nodes that do not participate in local synchronization, gradient accumulation is introduced therein. When synchronizing gradients, assign a weight to each node , and this weight is dynamically adjusted according to its historical synchronization times, and is used to measure its contribution degree in the current synchronization; for the device i The j gradient information at the th iteration is denoted as (9) where represents the set of devices participating in gradient synchronization in the current data parallelism; The weight is expressed as: (10) where is the number of times the current node has participated in historical local synchronization.

[0012] In a second aspect, an embodiment of the present invention provides a computer-readable storage medium, and the computer-readable storage medium includes a stored program. When the program runs, it controls the device where the computer-readable storage medium is located to execute the deep learning model training method applicable to network and device heterogeneous environments in the first aspect or any possible implementation manner of the first aspect.

[0013] In a third aspect, an embodiment of the present invention provides an electronic device, including: one or more processors; a memory; and one or more computer programs, where the one or more computer programs are stored in the memory, and the one or more computer programs include instructions. When the instructions are executed by the device, the device is caused to execute the deep learning model training method applicable to network and device heterogeneous environments in the first aspect or any possible implementation manner of the first aspect.

[0014] In the technical solution provided by the present invention, the method includes designing an improved hybrid parallel strategy; constructing a model partitioning and task placement scheme based on dynamic programming according to the improved hybrid parallel strategy; and designing an adaptive weighted semi-synchronous hybrid training mechanism according to the improved hybrid parallel strategy and the model partitioning and task placement scheme based on dynamic programming. This method realizes the reasonable allocation of communication load and improves the hardware utilization rate and the overall training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0016] Figure 1 It is a flowchart of a deep learning model training method applicable to a network and device heterogeneous environment provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of deep learning model training applicable to a network and device heterogeneous environment provided by an embodiment of the present invention; Figure 3 It is a schematic diagram of the improved hybrid parallel strategy provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of pipeline parallelism provided by an embodiment of the present invention; Figure 5 It is a schematic diagram of the comparison of the synchronization time between All-Reduce and SSAW provided by an embodiment of the present invention; Figure 6 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0018] It should be clear that the described embodiments are only some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0019] The terms used in the embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The singular forms "a", "said", and "the" used in the embodiments of the present invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0020] It should be understood that the term "and / or" used herein is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, a and / or b may represent: a exists alone, a and b exist simultaneously, and b exists alone. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after.

[0021] Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detected (stated condition or event)" may be interpreted as "when determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)".

[0022] Figure 1 The flowchart of the deep learning model training method applicable to the network and device heterogeneous environment provided by the embodiments of the present invention is as Figure 1 shown, and the method includes: In the embodiments of the present invention, as Figure 2 shown, the method is applicable to distributed training in a network and device dual heterogeneous environment, which consists of three core modules: an analyzer, a planner, and a runtime system; Specifically, a hybrid parallel strategy, PatchPipe, was designed for the runtime system. By assigning different batch sizes within stages based on the device's compute power and memory capacity, and distributing the number of heterogeneous devices between stages, the original model graph is converted into a pipelined parallel graph, enabling collaborative training of heterogeneous devices. To apply this hybrid parallel strategy to actual training, an analyzer and planner were designed. First, at the input level, the analyzer analyzes the number of parameters, activation values, gradients, forward propagation time, and backward propagation time of each layer of the model, taking the analyzer's output as input. The planner then models the communication and computation of the training process using a dynamic programming algorithm to generate a parallel plan for the given global batch size, including model partitioning, device grouping, and the batch size processed by each device. In addition, the runtime system also combines the Semi-Sync with Adaptive Weighting (SSAW) optimization method to effectively alleviate the waiting problem caused by laggards in gradient synchronization.

[0023] First, we designed an efficient hybrid parallel strategy for training tasks on heterogeneous devices. In real-world scenarios, heterogeneous devices have varying memory capacities. High-end devices typically have larger memory capacities and can handle larger batches of training samples, while low-end devices, due to memory limitations, cannot independently complete computational tasks of the same scale.

[0024] Step 1: Design an improved hybrid parallel strategy.

[0025] In the embodiment of the present invention, step 1 includes: To address the differences in memory capacity across heterogeneous devices, we designed an improved hybrid parallel strategy, PatchPipe. Within the same stage, batches are allocated based on the memory capacity and computing power of heterogeneous devices, thereby achieving resource load balancing. Between different stages, multiple low-end heterogeneous devices are combined into a group, using data parallelism within the group and pipeline parallelism between groups to jointly process a complete training batch, achieving a balance with high-end devices in terms of memory capacity. Figure 3 As shown in Figure 2, in stage 2 and stage 3, two devices (such as CPU2 and CPU3 in stage 2; CPU4 and CPU5 in stage 3) are used to achieve memory balance with the devices in stage 1. PatchPipe improves the memory efficiency of heterogeneous device collaborative training by optimizing the scheduling strategy of micro-batches on the pipeline, further reducing the memory pressure of low-end devices. Figure 3As shown in the figure, PatchPipe schedules micro-batch backpropagation in advance, timely releases the memory of the activations generated by the current micro-batch forward propagation, thus making room for the next injected micro-batch, realizing the reuse of memory and expanding the supportable training scale. In the next iteration of the training process, the above process is repeated, thereby achieving the improvement of training efficiency.

[0026] Step 2: According to the improved hybrid parallel strategy, construct a model partitioning and task placement scheme based on dynamic programming.

[0027] To support the actual deployment and execution of the above hybrid parallel strategy, the problem will be formally modeled, and a model partitioning and task placement scheme will be designed to generate a parallel training plan that conforms to the above hybrid parallel strategy.

[0028] In the embodiment of the present invention, as Figure 4 shown, taking stage3 as an example, the total execution time of a stage can be divided into three parts, including the time of the start stage, the time of the execution stage, and the time of the gradient synchronization stage. First, stage3 needs to wait for the data of stage2 to complete transmission before it can execute the calculation, and this time-consuming is the start time. When the devices in stage3 receive the data sent by stage2, they can calculate until the calculation of a single batch of data is completed, and this time is the execution time. When the calculation is completed, the devices in stage3 perform gradient synchronization, and this time is the synchronization time. When the gradient synchronization of all stages is completed, the next iteration starts, and the above process is repeated until all data calculations are completed. We select the stage with the shortest idle time (the fewest bubbles) of the device during execution as the pivot stage (pivot stage), as Figure 4 shown, and stage3 is defined as the pivot stage.

[0029] In the embodiment of the present invention, step 2 includes: For the synchronous training strategy, use the iteration time of a single batch of data as the performance metric; minimize the single iteration time under the condition of considering all solution spaces of data parallelism and pipeline parallelism; In pipeline parallel training, the model is divided into multiple stages stage and assigned to different devices for processing in a pipeline manner; during the training process, the total iteration time of a stage s in the pipeline is denoted as T , which includes 3 parts, namely the time of the start stage , the time of the execution stage and the time of the final gradient synchronization stage ; the expression of the total iteration time of stage s is: (1) Minimize the total iteration time of stage s, and the expression of its optimization objective is: (2) Among them, the time at the start stage of stage s , estimate it, and the expression is: (3) Among them, and are respectively the calculation time in the forward propagation process of the i th stage and the communication time sent to the next stage; Adopt All-Reduce for gradient synchronization. For the gradient synchronization time of stage s, its expression is: (4) Among them, is the device group for data parallelism, and the communication volume of the All-Reduce stage is , and the communication bandwidth depends on the minimum connection bandwidth among all devices; Next, determine the execution time of each stage stage, and define the stage with the fewest bubbles in the pipeline as the pivot stage; according to the execution time of the pivot stage, further derive the execution time of other stages; For the execution time of the pivot stage, its expression is: (5) Among them, M is the number of micro-batches processed in one iteration; and are respectively the calculation times of the forward and backward propagations of stage s; calculate the execution times of other stages through the execution time of the pivot stage, and its expression is: (6) Based on the above formulas, obtain , , , and , that is, obtain the iteration time of each stage. When [[ID=id=60]] i takes the value of s, their expressions are respectively:

[0030] Among them, is the batch size allocated to device d; and are respectively at the lOn the layer, the batch size processed by device d is The execution time of forward and backward propagation; and are respectively the magnitudes of the activation values transmitted during the forward propagation and the gradients transmitted during the backward propagation when the batch size processed by device d is ; For the assigned to each device, design the algorithm as follows: Input: (Set of devices in phase s); B (total batch size); Output: (Batch size assigned to each device); ; ; ; ; end for; Return β ; ; ; (Execution time of each device measured under the current batch); end for; do; Select the slowest and fastest devices in terms of execution time; Transfer the number of samples from the slowest device to the fastest device and recalculate the execution time; Re-select the new fastest and slowest devices; While the computing times of all devices reach relative balance; Return β ; InitialAcollate(); DynamicBalance(); The main goal of the algorithm is to optimally allocate the number of batches among heterogeneous devices, minimize the execution time of data parallelism, and not exceed the memory of each device. The algorithm is divided into two stages. The initial allocation is completed according to the memory capacity of the device group, and then dynamic adjustment is performed according to the execution time of the devices. By migrating a part of the samples from the slowest device to the fastest device, the time difference between devices is gradually reduced until a relatively balanced state is reached.

[0031] To find the optimal model partitioning and task placement scheme between heterogeneous devices and network models, a dynamic programming method is proposed: is expressed as the latency of parallel planning that divides L consecutive neural network layers into S stages and maps them to N heterogeneous devices. The goal is to minimize , and the expression is: (8) Among them, has the property of optimal substructure and is decomposed into sub-problems, which consists of two parts: (1) An independent stage that contains l layers of neural network, placed on n devices, where data parallelism is used, and its time is represented by ; (2) An optimal sub-configuration , which is composed of the last layers of neural network, divided into stages, and mapped to devices. The optimal time of this sub-problem is denoted as ; Based on the recurrence relation, the algorithm process is as follows: (Traverse the possible stages, from 1 to the minimum of the number of layers and the number of devices); (Traverse the number of devices to consider device partitioning); (Traverse the number of model layers to try different layer partitioning methods); (For the current device partitioning, traverse the sub-partitioning schemes); (For the current layer partitioning, traverse the sub-partitioning schemes); Get and 、 and (Calculate the forward / backward calculation and communication time of each stage under the current partitioning); (Update the key position of the current stage); Get , , and (Calculate the time of the start stage, the time of the execution stage, and the time of the gradient synchronization stage under this partitioning); (Get the total iteration time under the current partitioning); Update (Update the final model partitioning and task placement scheme and solve the sub-problem); During the execution of the algorithm, record the partitioning points of the model, the grouping strategy of the devices, and the number of training samples assigned to the devices. Finally, backtrack the results to identify the optimal model partitioning and task placement scheme.

[0032] In the embodiments of the present invention, the pivot stage is determined through iterative optimization: First, each stage is sequentially evaluated as a candidate pivot stage. For each candidate pivot stage, calculate the total iteration time of the entire training process; take the candidate pivot stage with the shortest iteration time as the pivot stage for this training; through the above iterative process, dynamically evaluate the impact of each stage as the pivot stage on the overall training time, and select the stage that minimizes the training iteration time. Once the pivot stage is determined, other relevant variables can be determined accordingly.

[0033] Step 3: Design an adaptive weighted semi-synchronous hybrid training mechanism according to the improved hybrid parallel strategy and the model partitioning and task placement scheme based on dynamic programming.

[0034] In an actual heterogeneous network training environment, the network condition is often not ideal, and the connection state may fluctuate, resulting in the straggler problem during the training process. As Figure 5 shown, this problem will significantly extend the execution time of each iteration due to waiting for the straggler nodes in the gradient synchronization group, reducing the overall training efficiency. To this end, an adaptive weighted semi-synchronous (SSAW) hybrid training mechanism is proposed. This method combines the stability of the synchronous mechanism and the flexibility of the asynchronous mechanism, thus maintaining the effective convergence of the model while significantly improving the training efficiency.

[0035] In the embodiments of the present invention, Step 3 includes: To improve the overall training efficiency, a selective synchronization strategy is designed in the Adaptive Weighted Semi-Synchronous (SSAW). For each iteration, the synchronization of gradients does not require all nodes to participate. Instead, a two-stage mechanism of local synchronization and global synchronization is introduced. Local synchronization is a lightweight gradient synchronization operation for some nodes. In each iteration, the running states of each node are dynamically evaluated, and nodes that have completed the current iteration quickly are preferentially selected to perform an All-Reduce operation once. To ensure the convergence of the training process and the consistency of model parameters, after several rounds of local synchronization, SSAW will periodically perform a global synchronization operation. Global synchronization forces all nodes to uniformly execute a complete All-Reduce, aligning all nodes again on the model parameters and preventing the model from deviating too much due to long-term asynchronous updates. At the same time, for nodes that do not participate in local synchronization, gradient accumulation is introduced. In the embodiment of the present invention, as Figure 5 shown, taking device GPU2 as an example, during the gradient synchronization in the second iteration, the All-Reduce synchronization method will wait for the lagging GPU2, thus slowing down the training time. In SSAW, the gradient synchronization of the lagging node GPU2 in the second iteration is selected to be skipped to improve the training efficiency. To ensure the stable convergence of the training process, when performing gradient synchronization, each node is given a weight , and this weight is dynamically adjusted according to its historical synchronization times to measure its contribution degree in the current synchronization. For device i the j gradient information at the th iteration is denoted as (9) where represents the set of devices participating in gradient synchronization in the current data parallelism; the weight is expressed as: (10) where is the number of times the current node has participated in historical local synchronization.

[0036] Through the SSAW method, while ensuring the effective convergence of the model, the training efficiency is improved.

[0037] In the technical solution provided by the present invention, the method includes designing an improved hybrid parallel strategy; constructing a model partitioning and task placement scheme based on dynamic programming according to the improved hybrid parallel strategy; and designing an adaptive weighted semi-synchronous hybrid training mechanism according to the improved hybrid parallel strategy and the model partitioning and task placement scheme based on dynamic programming. This method realizes the reasonable allocation of communication load and improves the hardware utilization rate and overall training efficiency.

[0038] Each step of the embodiment of the present invention can be executed by an electronic device. Among them, the electronic device includes, but is not limited to, a tablet computer, a portable PC, a desktop computer, etc.

[0039] The embodiment of the present invention provides a computer-readable storage medium. The computer-readable storage medium includes a stored program. When the program runs, it controls the electronic device where the computer-readable storage medium is located to execute the embodiment of the above-mentioned deep learning model training method applicable to network and device heterogeneous environments.

[0040] Figure 6 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. As Figure 6 shown, the electronic device 21 includes: a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, it implements the deep learning model training method applicable to network and device heterogeneous environments in the embodiment. To avoid repetition, details are not described here one by one.

[0041] The electronic device 21 includes, but is not limited to, a processor 211 and a memory 212. Those skilled in the art can understand that Figure 6 this is only an example of the electronic device 21 and does not constitute a limitation on the electronic device 21. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0042] The so-called processor 211 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0043] The memory 212 may be an internal storage unit of the electronic device 21, such as the hard disk or memory of the electronic device 21. The memory 212 may also be an external storage device of the electronic device 21, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a FlashCard, etc. equipped on the electronic device 21. Further, the memory 212 may also include both the internal storage unit of the electronic device 21 and the external storage device. The memory 212 is used to store computer programs and other programs and data required by the network device. The memory 212 may also be used to temporarily store the data that has been output or will be output.

[0044] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be described herein again.

[0045] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A deep learning model training method applicable to network and device heterogeneous environments, characterized in that, The method includes: Step 1, design an improved hybrid parallel strategy; Step 2, construct a model partitioning and task placement scheme based on dynamic programming according to the improved hybrid parallel strategy; Step 3, design an adaptive weighted semi-synchronous hybrid training mechanism according to the improved hybrid parallel strategy and the model partitioning and task placement scheme based on dynamic programming.

2. The method according to claim 1, wherein The said Step 1 includes: Design an improved hybrid parallel strategy PatchPipe. Inside the same stage, allocate corresponding batches according to the memory capacity and computing power of heterogeneous devices. Between different stages, combine multiple low-end heterogeneous devices into a group. Data parallelism is adopted within the group, and pipeline parallelism is adopted between groups to jointly process a complete training batch, achieving a balance with high-end devices in terms of memory capacity. PatchPipe schedules the micro-batch backpropagation in advance, and timely releases the memory of the activations generated by the current micro-batch forward propagation, so as to free up memory for the next injected micro-batch.

3. The method according to claim 1, wherein The said Step 2 includes: For the synchronous training strategy, use the iteration time of a single batch of data as the performance metric; minimize the single iteration time under the condition of considering all solution spaces of data parallelism and pipeline parallelism; In pipeline parallel training, the model is divided into multiple stages and assigned to different devices for processing in a pipeline manner; during training, the total iteration time of a stage s in the pipeline is denoted as T , which consists of three parts, namely the time at the start stage , the time at the execution stage and the time at the final gradient synchronization stage ; the expression for the total iteration time of stage s is: （1） Minimize the total iteration time of stage s, and the expression of its optimization objective is: （2） Among them, the time at the start stage of stage s , is estimated, and the expression is: （3） Among them, and are the calculation time in the forward propagation process and the communication time sent to the next stage in the i th stage, respectively; Gradient synchronization is performed using All-Reduce. For the gradient synchronization time of stage s , its expression is as follows: （4） Among them, is a device group for data parallelism, and the communication volume in the All-Reduce stage is , and the communication bandwidth depends on the minimum connection bandwidth among all devices; Next, determine the execution time of each stage. Define the stage with the fewest bubbles in the pipeline as the pivot stage; based on the execution time of the pivot stage, further deduce the execution times of other stages. For the execution time of the pivot stage, its expression is: （5） Among them, M is the number of mini - batches processed in one iteration; and are the computation times of the forward and backward propagations in stage s respectively; the execution times of other stages are calculated through the execution time of the pivot stage, and its expression is: （6） Based on the above formula, the following are obtained , , , and , that is, the iterative time of each stage is obtained. When i takes the value of s, their expressions are respectively as follows: Among them, is the batch size allocated to device d; and are respectively, at the l layer, the execution times of forward and backward propagation when the batch size processed by device d is ; and are respectively the magnitudes of the activation values transmitted during the forward propagation and the gradients transmitted during the backward propagation in stage s when the batch size processed by device d is . Denote as the latency of the parallel planning that divides the L-layer continuous neural network into S stages and maps them to N heterogeneous devices, and its goal is to minimize , and the expression is: （8） Among them, has the property of optimal substructure. It is decomposed into sub-problems and consists of two parts: (1) An independent stage that contains l layers of neural networks placed on n devices, where data parallelism is used and its time is represented by ; (2) An optimal sub-configuration , which consists of the last layers of neural networks, divided into stages, and mapped to devices. The optimal time of this sub-problem is denoted as . During the execution of the algorithm, record the partitioning points of the model, the grouping strategy of the devices, and the number of training samples allocated to the devices. Finally, backtrack the results to identify the optimal model partitioning and task placement scheme.

4. The method according to claim 3, wherein Determine the pivot stage through iterative optimization: First, evaluate each stage as a candidate pivot stage in turn. For each candidate pivot stage, calculate the total iteration time of the entire training process; Take the candidate pivot stage with the shortest iteration time as the pivot stage for this training; through the above iterative process, dynamically evaluate the impact of each stage as the pivot stage on the overall training time, and select the stage that minimizes the training iteration time.

5. The method according to claim 1, wherein The said Step 3 includes: Design a selective synchronization strategy in Adaptive Weighted Semi-Synchronous SSAW. For each iteration, the synchronization of gradients introduces a two-stage mechanism of local synchronization and global synchronization; local synchronization is a lightweight gradient synchronization operation for some nodes; in each iteration, dynamically evaluate the running status of each node, and preferentially select the nodes that complete the current iteration quickly to perform an All-Reduce operation once; after several rounds of local synchronization, SSAW will periodically perform a global synchronization operation, and the global synchronization forces all nodes to uniformly perform a complete All-Reduce, so that all nodes are realigned on the model parameters; at the same time, for the nodes that do not participate in local synchronization, gradient accumulation is introduced therein. When performing gradient synchronization, assign a weight to each node , and this weight is dynamically adjusted according to its historical synchronization times, and is used to measure its contribution degree in the current synchronization; for the device i The j gradient information at the th iteration is denoted as, and when it participates in gradient synchronization, the expression of the weighted gradient is: （9） Among them, represents the set of devices participating in gradient synchronization in the current data parallelism; Weight value The expression is: （10） Among them, is the number of times the current node has been locally synchronized historically.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program runs, it controls the device where the computer-readable storage medium is located to execute the deep learning model training method applicable to network and device heterogeneous environments described in any one of claims 1 to 5.

7. An electronic device, characterized in that, Comprising: One or more processors; A memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to execute the deep learning model training method applicable to network and device heterogeneous environments described in any one of claims 1 to 5.

Images