A deep learning model training method suitable for heterogeneous network and device environments

Abstract

The present invention relates to the field of model training technology, and in particular, provides a deep learning model training method suitable for heterogeneous network and device environments. The method includes designing an improved hybrid parallel strategy; constructing a model partitioning and task placement scheme based on the improved hybrid parallel strategy; and designing an adaptive weighted semi-synchronous hybrid training mechanism based on the improved hybrid parallel strategy and the dynamic programming-based model partitioning and task placement scheme. This method achieves reasonable distribution of communication load, improves hardware utilization, and improves overall training efficiency.

Description

Technical Field

[0001] The present invention relates to the field of model training technology, and in particular to a deep learning model training method suitable for heterogeneous network and device environments. Background Art

[0002] With the advancement of deep learning, especially the rise of large-scale pre-trained models, the number of model parameters has grown exponentially. This increased parameter size has enabled significant progress in fields such as question-answering systems and image generation, thanks to the enhanced generalization capabilities afforded by this large parameter size. Training such large models places increasing demands on computing resources. Single-machine training faces dual limitations in computing power and graphics memory capacity, making it inadequate for modern large-scale model training. Therefore, distributed training has become an inevitable choice for large-scale model training. Existing distributed training strategies include hybrid parallelism strategies such as data parallelism, pipelined model parallelism, and tensor parallelism. These methods have been integrated into frameworks such as Megatron, DeepSpeed, and Alpa, and are widely used in model training practices. These frameworks primarily target high-performance, homogeneous computing clusters, which are often difficult for most companies and researchers to access due to cost and resource constraints.

[0003] Currently, there are still many problems when migrating a framework from a homogeneous environment directly to a complex heterogeneous environment:

[0004] First, existing frameworks usually assume that all devices have the same computing power, memory capacity, and network bandwidth, ignoring the heterogeneity between devices. As a result, high-performance devices are often dragged down by low-performance devices, unable to fully exert their performance, and resource utilization efficiency is low. This not only wastes the computing power of high-performance devices, but also significantly increases training time. Most existing methods only focus on heterogeneity in a single dimension. For example, some methods focus on optimizing communication efficiency in heterogeneous network environments, but ignore the differences in computing power of the devices themselves; other methods focus on computing optimization of heterogeneous devices, but do not fully consider the heterogeneity of network bandwidth between devices. This single-dimensional optimization strategy is difficult to adapt to the dual heterogeneous environment of network communication and computing devices at the same time, resulting in insufficient hardware utilization or low training efficiency.

[0005] Secondly, the network bandwidths between heterogeneous devices are different. Simple model division cannot adapt to the network topology characteristics and it is difficult to achieve reasonable distribution of communication load.

[0006] Finally, in a dual heterogeneous environment of networks and devices, network latency and bandwidth fluctuate greatly. In hybrid parallel training, the speeds of nodes performing gradient synchronization in data parallelism may be inconsistent. In frequent cross-node communication, nodes with inferior computing and transmission speeds must wait for slower nodes in data parallelism to complete calculations, significantly reducing overall training efficiency.

[0007] Traditional gradient synchronization methods, both synchronous and asynchronous, have limitations. Synchronous parallelism causes fast nodes to wait for slow nodes, wasting computing power. Asynchronous parallelism can affect model convergence and accuracy due to outdated parameters, making it even less suitable for heterogeneous environments. Summary of the Invention

[0008] In view of this, the present invention provides a deep learning model training method suitable for heterogeneous network and device environments, so as to achieve reasonable distribution of communication load, improve hardware utilization and overall training efficiency.

[0009] In a first aspect, the present invention provides a deep learning model training method applicable to a heterogeneous network and device environment, the method comprising:

[0010] Step 1: Design an improved hybrid parallel strategy;

[0011] Step 2: Based on the improved hybrid parallel strategy, a model partitioning and task placement scheme based on dynamic programming is constructed;

[0012] Step 3: Based on the improved hybrid parallel strategy and the model partitioning and task placement scheme based on dynamic programming, an adaptive weighted semi-synchronous hybrid training mechanism is designed.

[0013] Optionally, step 1 includes:

[0014] The improved hybrid parallel strategy PatchPipe is designed. Within the same stage, corresponding batches are allocated according to the memory capacity and computing power of heterogeneous devices. Between different stages, multiple low-end heterogeneous devices are combined into a group, using data parallelism within the group and pipeline parallelism between groups to jointly process a complete training batch, achieving a balance in memory capacity with high-end devices. PatchPipe also schedules micro-batch backpropagation in advance to promptly release the activated memory generated by the current micro-batch forward propagation, thereby freeing up memory for the next injected micro-batch.

[0015] Optionally, step 2 includes:

[0016] For synchronous training strategies, the iteration time of a single batch of data is used as the performance indicator; the single iteration time is minimized while considering all solution spaces of data parallelism and pipeline parallelism;

[0017] In pipeline parallel training, the model is divided into multiple stages and distributed to different devices for processing in a pipeline manner. During the training process, the total iteration time of a stage s in the pipeline is recorded as T , which contains 3 parts, namely the time of the start phase , execution phase time and the time of the final gradient synchronization phase The total iteration time of stage s is expressed as:

[0018] (1)

[0019] Minimize the total iteration time of stage s. The expression of the optimization objective is:

[0020] (2)

[0021] Among them, the time of the start of stage s , and estimate it, the expression is:

[0022] (3)

[0023] in, and Respectively i The computation time of each stage during the forward propagation process and the communication time to the next stage;

[0024] All-Reduce is used for gradient synchronization. For stage s, the gradient synchronization time is , whose expression is:

[0025] (4)

[0026] in, For a data parallel device group, the communication volume in the All-Reduce phase is ,The communication bandwidth depends on the minimum connection bandwidth between all ,devices;

[0027] Next, determine the execution time of each stage. Define the stage with the fewest bubbles in the pipeline as the pivot stage. Based on the execution time of the pivot stage, deduce the execution time of other stages. The execution time of the pivot stage is expressed as:

[0028] (5)

[0029] in, M is the number of micro-batches processed in one iteration; and The forward and backward propagation calculation times of stage s are respectively; the execution time of other stages is calculated by the execution time of the pivot stage, and its expression is:

[0030] (6)

[0031] Based on the above formula, we can get 、 、 ,and , that is, to obtain the iteration time of each stage, when i When the value is s, the expressions are:

[0032]

[0033] in, The batch size allocated to device d; and In the l At the layer, the batch size processed by device d is , the execution time of forward and backward propagation; and The batch size processed by device d is When , the magnitude of the activation value transmitted by forward propagation and the gradient transmitted by backward propagation in stage s;

[0034] Will It is represented as dividing the L-layer continuous neural network into S stages and mapping them to the parallel planning delay on N heterogeneous devices, with the goal of minimizing , the expression is:

[0035] (8)

[0036] in, It has the optimal substructure property and is decomposed into two subproblems: (1) an independent stage, which contains l Layer neural network, placed on n devices, using data parallelism, its time is given by Represents; (2) an optimal sub-configuration , which consists of the last Layer neural network, divided into stage, and mapped to On each device, the optimal time for this subproblem is recorded as ;

[0037] During the execution of the algorithm, the model partitioning points, the device grouping strategy, and the number of training samples allocated to the devices are recorded. Finally, the results are backtracked to identify the optimal model partitioning and task placement scheme.

[0038] Optionally, the pivot stage is determined through iterative optimization: First, each stage is evaluated as a candidate pivot stage. For each candidate pivot stage, the total iteration time of the entire training process is calculated. The candidate pivot stage with the shortest iteration time is used as the pivot stage for this training. Through this iterative process, the impact of each stage as a pivot stage on the overall training time is dynamically evaluated, and the stage that minimizes the training iteration time is selected.

[0039] Optionally, step 3 includes:

[0040] A selective synchronization strategy is designed in the adaptive weighted semi-synchronous SSAW. For each iteration, gradient synchronization introduces a two-stage mechanism of local synchronization and global synchronization. Local synchronization is a lightweight gradient synchronization operation for some nodes. In each iteration, the operating status of each node is dynamically evaluated, and the nodes that complete the current iteration quickly are preferentially selected to perform an All-Reduce operation. After several rounds of local synchronization, SSAW periodically performs a global synchronization operation. Global synchronization forces all nodes to uniformly execute a complete All-Reduce operation, realigning all nodes on model parameters. At the same time, gradient accumulation is introduced for nodes that do not participate in local synchronization.

[0041] When synchronizing gradients, each node is given a weight This weight is dynamically adjusted according to its historical synchronization times and is used to measure its contribution to the current synchronization. i No. j The gradient information at the iteration is recorded as , when it participates in gradient synchronization, the expression of weighted gradient is:

[0042] (9)

[0043] in, Indicates the set of devices participating in gradient synchronization in the current data parallel process;

[0044] Weight The expression is:

[0045] (10)

[0046] in, The number of times the current node has been synchronized locally in history.

[0047] In a second aspect, an embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the deep learning model training method applicable to heterogeneous network and device environments in the first aspect or any possible implementation of the first aspect.

[0048] In a third aspect, an embodiment of the present invention provides an electronic device comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, enable the device to execute the deep learning model training method applicable to a heterogeneous network and device environment in the first aspect or any possible implementation of the first aspect.

[0049] In the technical solution provided by the present invention, the method includes designing an improved hybrid parallel strategy; constructing a model partitioning and task placement scheme based on dynamic programming based on the improved hybrid parallel strategy; designing an adaptive weighted semi-synchronous hybrid training mechanism based on the improved hybrid parallel strategy and the model partitioning and task placement scheme based on dynamic programming. This method realizes the reasonable distribution of communication load, improves hardware utilization and overall training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A flowchart of a deep learning model training method applicable to heterogeneous network and device environments provided by an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of a deep learning model training method suitable for heterogeneous network and device environments provided by an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of an improved hybrid parallel strategy provided by an embodiment of the present invention;

[0054] Figure 4A schematic diagram of pipeline parallelism provided by an embodiment of the present invention;

[0055] Figure 5 A schematic diagram comparing the synchronization time of All-Reduce and SSAW provided in an embodiment of the present invention;

[0056] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0058] It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0059] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0060] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the associated objects.

[0061] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0062] Figure 1 The flowchart of the deep learning model training method applicable to the network and device heterogeneous environment provided by the embodiment of the present invention is as follows: Figure 1As shown, the method includes:

[0063] In the embodiment of the present invention, Figure 2 As shown in Figure 2, this method is suitable for distributed training in a dual heterogeneous environment of network and device. It consists of three core modules: analyzer, planner, and runtime system.

[0064] Specifically, a hybrid parallel strategy, PatchPipe, was designed for the runtime system. By assigning different batch sizes within stages based on the device's compute power and memory capacity, and distributing the number of heterogeneous devices between stages, the original model graph is converted into a pipelined parallel graph, enabling collaborative training of heterogeneous devices. To apply this hybrid parallel strategy to actual training, an analyzer and planner were designed. First, at the input level, the analyzer analyzes the number of parameters, activation values, gradients, forward propagation time, and backward propagation time of each layer of the model, taking the analyzer's output as input. The planner then models the communication and computation of the training process using a dynamic programming algorithm to generate a parallel plan for the given global batch size, including model partitioning, device grouping, and the batch size processed by each device. In addition, the runtime system also combines the Semi-Sync with Adaptive Weighting (SSAW) optimization method to effectively alleviate the waiting problem caused by laggards in gradient synchronization.

[0065] First, we designed an efficient hybrid parallel strategy for training tasks on heterogeneous devices. In real-world scenarios, heterogeneous devices have varying memory capacities. High-end devices typically have larger memory capacities and can handle larger batches of training samples, while low-end devices, due to memory limitations, cannot independently complete computational tasks of the same scale.

[0066] Step 1: Design an improved hybrid parallel strategy.

[0067] In the embodiment of the present invention, step 1 includes:

[0068] To address the differences in memory capacity across heterogeneous devices, we designed an improved hybrid parallel strategy, PatchPipe. Within the same stage, batches are allocated based on the memory capacity and computing power of heterogeneous devices, thereby achieving resource load balancing. Between different stages, multiple low-end heterogeneous devices are combined into a group, using data parallelism within the group and pipeline parallelism between groups to jointly process a complete training batch, achieving a balance with high-end devices in terms of memory capacity. Figure 3As shown in Figure 2, in stage 2 and stage 3, two devices (such as CPU2 and CPU3 in stage 2; CPU4 and CPU5 in stage 3) are used to achieve memory balance with the devices in stage 1. PatchPipe improves the memory efficiency of heterogeneous device collaborative training by optimizing the scheduling strategy of micro-batches on the pipeline, further reducing the memory pressure of low-end devices. Figure 3 As shown in Figure 1, PatchPipe schedules micro-batch backpropagation in advance, promptly releasing the memory activated by the current micro-batch forward propagation, freeing up memory for the next injected micro-batch. This enables memory reuse and expands the supported training scale. In the next iteration of the training process, the above process is repeated, thereby improving training efficiency.

[0069] Step 2: Based on the improved hybrid parallel strategy, a model partitioning and task placement scheme based on dynamic programming is constructed.

[0070] To support the actual deployment and execution of the above hybrid parallel strategy, the problem will be formally modeled, and a model partitioning and task placement scheme will be designed to generate a parallel training plan that conforms to the above hybrid parallel strategy.

[0071] In the embodiment of the present invention, Figure 4 As shown, taking stage3 as an example, the total execution time of a stage can be divided into three parts, including the time of the start stage, the time of the execution stage and the time of the gradient synchronization stage. First, stage3 needs to wait for the data from stage2 to be transmitted before it can execute the calculation. This time is the start time. When the device in stage3 receives the data sent by stage2, it can calculate until the calculation of a single batch of data is completed. This time is the execution time. When the calculation is completed, the device in stage3 performs gradient synchronization. This time is the synchronization time. When the gradient synchronization of all stages is completed, the next iteration begins, and the above process is repeated until all data calculations are completed. We select the stage with the shortest device idle time (fewest bubbles) during execution and define it as the pivot stage. As shown in Figure 4 As shown, stage3 is defined as the pivot stage.

[0072] In this embodiment of the present invention, step 2 includes:

[0073] For synchronous training strategies, the iteration time of a single batch of data is used as the performance indicator; the single iteration time is minimized while considering all solution spaces of data parallelism and pipeline parallelism;

[0074] In pipeline parallel training, the model is divided into multiple stages and distributed to different devices for processing in a pipeline manner. During the training process, the total iteration time of a stage s in the pipeline is recorded as T , which contains 3 parts, namely the time of the start phase , execution phase time and the time of the final gradient synchronization phase The total iteration time of stage s is expressed as:

[0075] (1)

[0076] Minimize the total iteration time of stage s. The expression of the optimization objective is:

[0077] (2)

[0078] Among them, the time of the start of stage s , and estimate it, the expression is:

[0079] (3)

[0080] in, and Respectively i The computation time of each stage during the forward propagation process and the communication time to the next stage;

[0081] All-Reduce is used for gradient synchronization. For stage s, the gradient synchronization time is , whose expression is:

[0082] (4)

[0083] in, For a data parallel device group, the communication volume in the All-Reduce phase is ,The communication bandwidth depends on the minimum connection bandwidth between all ,devices;

[0084] Next, determine the execution time of each stage. Define the stage with the fewest bubbles in the pipeline as the pivot stage. Based on the execution time of the pivot stage, deduce the execution time of other stages.

[0085] For the execution time of the pivot stage, the expression is:

[0086] (5)

[0087] in, Mis the number of micro-batches processed in one iteration; and The forward and backward propagation calculation times of stage s are respectively; the execution time of other stages is calculated by the execution time of the pivot stage, and its expression is:

[0088] (6)

[0089] Based on the above formula, we can get 、 、 ,and , that is, to obtain the iteration time of each stage, when i When the value is s, the expressions are:

[0090]

[0091] in, The batch size allocated to device d; and In the l At the layer, the batch size processed by device d is , the execution time of forward and backward propagation; and The batch size processed by device d is When , the magnitude of the activation value transmitted by forward propagation and the gradient transmitted by backward propagation in stage s;

[0092] For each device, the The size of , design the algorithm as follows:

[0093] enter:

[0094] (the set of devices in stage s);

[0095] B (total batch size);

[0096] Output:

[0097] (batch size allocated to each device);

[0098] ;

[0099] ;

[0100] ;

[0101] ;

[0102] end for;

[0103] return β ;

[0104] ;

[0105] ;

[0106] (Measure the execution time of each device in the current batch);

[0107] end for;

[0108] do;

[0109] Select the device with the slowest and fastest execution time;

[0110] Transfer the number of samples from the slowest device to the fastest device and recalculate the execution time;

[0111] Reselect new fastest and slowest devices;

[0112] While the computing time of all devices is relatively balanced;

[0113] return β ;

[0114] InitialAcollate();

[0115] DynamicBalance();

[0116] The algorithm's primary goal is to optimally distribute the number of batches across heterogeneous devices, minimizing data parallel execution time while remaining within the memory capacity of each device. The algorithm is divided into two phases: initial batch allocation based on the device group's memory capacity, followed by dynamic adjustments based on the device's execution time. By migrating samples from the slowest device to the fastest, the time difference between devices is gradually reduced until a relative equilibrium is achieved.

[0117] In order to find the optimal model partitioning and task placement solution between heterogeneous devices and network models, a dynamic programming method is proposed: It is represented as dividing the L-layer continuous neural network into S stages and mapping them to the parallel planning delay on N heterogeneous devices, with the goal of minimizing , the expression is:

[0118] (8)

[0119] in, It has the optimal substructure property and is decomposed into two subproblems: (1) an independent stage, which contains l Layer neural network, placed on n devices, using data parallelism, its time is given by Represents; (2) an optimal sub-configuration , which consists of the last Layer neural network, divided into stage, and mapped to On each device, the optimal time for this subproblem is recorded as ;

[0120] Based on the recursive relationship, the algorithm process is as follows:

[0121] (Traverse the possible stages, from 1 to the minimum of the number of layers and the number of devices);

[0122] (Traverse the number of devices to consider device division);

[0123] (Traverse the model layers to try different layer division methods);

[0124] (For the current device partition, traverse the sub-partitioning scheme);

[0125] (For the current layer partitioning, traverse the sub-partitioning scheme);

[0126] Get and 、 and (Calculate the forward / backward computation and communication time of each stage under the current partition);

[0127] (update the key position of the current stage);

[0128] Get , , and (Calculate the start phase time, execution phase time, and gradient synchronization phase time under this partition);

[0129] (Get the total iteration time under the current partition);

[0130] Update (Update the final model partitioning and task placement scheme, and solve the sub-problems);

[0131] During the execution of the algorithm, the model partitioning points, the device grouping strategy, and the number of training samples allocated to the devices are recorded. Finally, the results are backtracked to identify the optimal model partitioning and task placement scheme.

[0132] In an embodiment of the present invention, the pivot stage is determined through iterative optimization. First, each stage is evaluated as a candidate pivot stage. For each candidate pivot stage, the total iteration time of the entire training process is calculated. The candidate pivot stage with the shortest iteration time is used as the pivot stage for this training. Through this iterative process, the impact of each stage as the pivot stage on the overall training time is dynamically evaluated, and the stage that minimizes the training iteration time is selected. Once the pivot stage is determined, other related variables can be determined accordingly.

[0133] Step 3: Based on the improved hybrid parallel strategy and the model partitioning and task placement scheme based on dynamic programming, an adaptive weighted semi-synchronous hybrid training mechanism is designed.

[0134] In the actual heterogeneous network training environment, the network conditions are often not ideal and the connection status may fluctuate, which may lead to the problem of laggards in the training process, such as Figure 5 As shown in Figure 2, this problem significantly prolongs the execution time of each iteration due to waiting for lagging nodes in the gradient synchronization group, reducing the overall training efficiency. To this end, a hybrid training mechanism called adaptive weighted semi-synchronous SSAW is proposed. This method combines the stability of the synchronous mechanism with the flexibility of the asynchronous mechanism, thereby significantly improving training efficiency while maintaining effective model convergence.

[0135] In this embodiment of the present invention, step 3 includes:

[0136] To improve overall training efficiency, a selective synchronization strategy is designed in the adaptive weighted semi-synchronous SSAW. For each iteration, gradient synchronization does not require all nodes to participate. Instead, a two-stage mechanism of local synchronization and global synchronization is introduced. Local synchronization is a lightweight gradient synchronization operation for some nodes. In each iteration, the running status of each node is dynamically evaluated, and the node that completes the current iteration quickly is preferentially selected to perform an All-Reduce operation. To ensure the convergence of the training process and the consistency of model parameters, SSAW will periodically perform a global synchronization operation after several rounds of local synchronization. Global synchronization forces all nodes to uniformly execute a complete All-Reduce operation, so that all nodes are realigned in terms of model parameters to prevent excessive model offset due to long-term asynchronous updates. At the same time, gradient accumulation is introduced for nodes that do not participate in local synchronization.

[0137] In the embodiment of the present invention, Figure 5 As shown in the figure, taking device GPU2 as an example, during the gradient synchronization in the second iteration, the All-Reduce synchronization method will wait for the lagging GPU2, thus slowing down the training time. In contrast, in SSAW, the gradient synchronization of the lagging node GPU2 in the second iteration is skipped to improve the training efficiency. To ensure the stable convergence of the training process,

[0138] When synchronizing gradients, each node is given a weight This weight is dynamically adjusted according to its historical synchronization times and is used to measure its contribution to the current synchronization. i No. j The gradient information at the iteration is recorded as , when it participates in gradient synchronization, the expression of weighted gradient is:

[0139] (9)

[0140] in, Indicates the set of devices participating in gradient synchronization in the current data parallel process;

[0141] Weight The expression is:

[0142] (10)

[0143] in, The number of times the current node has been synchronized locally in history.

[0144] Through the SSAW method, the training efficiency is improved while ensuring the effective convergence of the model.

[0145] In the technical solution provided by the present invention, the method includes designing an improved hybrid parallel strategy; constructing a model partitioning and task placement scheme based on dynamic programming based on the improved hybrid parallel strategy; designing an adaptive weighted semi-synchronous hybrid training mechanism based on the improved hybrid parallel strategy and the model partitioning and task placement scheme based on dynamic programming. This method realizes the reasonable distribution of communication load, improves hardware utilization and overall training efficiency.

[0146] Each step of the embodiment of the present invention may be performed by an electronic device, including but not limited to a tablet computer, a portable PC, a desktop computer, etc.

[0147] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the electronic device where the computer-readable storage medium is located is controlled to execute the above-mentioned embodiment of the deep learning model training method applicable to heterogeneous network and device environments.

[0148] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the electronic device 21 includes: a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, the deep learning model training method applicable to heterogeneous network and device environments in the embodiment is implemented. To avoid repetition, they are not described here one by one.

[0149] The electronic device 21 includes, but is not limited to, a processor 211 and a memory 212. Those skilled in the art will understand that Figure 6 It is only an example of the electronic device 21 and does not constitute a limitation of the electronic device 21. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0150] The processor 211 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0151] The memory 212 can be an internal storage unit of the electronic device 21, such as the hard drive or memory of the electronic device 21. The memory 212 can also be an external storage device of the electronic device 21, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 21. Furthermore, the memory 212 can include both the internal storage unit of the electronic device 21 and an external storage device. The memory 212 is used to store computer programs and other programs and data required by the network device. The memory 212 can also be used to temporarily store data that has been output or is about to be output.

[0152] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0153] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A deep learning model training method suitable for heterogeneous network and device environments, characterized in that: The method comprises: Step 1: Design an improved hybrid parallel strategy; Step 2: Based on the improved hybrid parallel strategy, a model partitioning and task placement scheme based on dynamic programming is constructed; Step 3: Based on the improved hybrid parallel strategy and the model partitioning and task placement scheme based on dynamic programming, an adaptive weighted semi-synchronous hybrid training mechanism is designed; The step 1 comprises: The improved hybrid parallel strategy, PatchPipe, allocates batches within a stage based on the memory capacity and computing power of heterogeneous devices. Between stages, multiple low-end heterogeneous devices are combined into a group, using data parallelism within the group and pipeline parallelism between groups to jointly process a complete training batch, achieving a balance in memory capacity with high-end devices. PatchPipe also schedules micro-batch backpropagation in advance to promptly release the memory activated by the current micro-batch forward propagation, freeing up memory for the next injected micro-batch. The step 3 includes: A selective synchronization strategy is designed in the adaptive weighted semi-synchronous SSAW. For each iteration, gradient synchronization introduces a two-stage mechanism of local synchronization and global synchronization. Local synchronization is a lightweight gradient synchronization operation for some nodes. In each iteration, the running status of each node is dynamically evaluated, and the node that completes the current iteration quickly is selected to perform an All-Reduce operation. After several rounds of local synchronization, SSAW periodically performs a global synchronization operation. Global synchronization forces all nodes to uniformly execute a complete All-Reduce operation, so that all nodes are realigned on the model parameters. At the same time, gradient accumulation is introduced for nodes that do not participate in local synchronization. When synchronizing gradients, each node is given a weight This weight is dynamically adjusted according to its historical synchronization times and is used to measure its contribution to the current synchronization. i No. j The gradient information at the iteration is recorded as , when it participates in gradient synchronization, the expression of weighted gradient is: （9） in, Indicates the set of devices participating in gradient synchronization in the current data parallel process; Weight The expression is: （10） in, The number of times the current node has been synchronized locally in history.

2. The method according to claim 1, characterized in that The step 2 includes: For synchronous training strategies, the iteration time of a single batch of data is used as the performance indicator; the single iteration time is minimized while considering all solution spaces of data parallelism and pipeline parallelism; In pipeline parallel training, the model is divided into multiple stages and distributed to different devices for processing in a pipeline manner. During the training process, the total iteration time of a stage s in the pipeline is recorded as T , which contains 3 parts, namely the time of the start phase , execution phase time and the time of the final gradient synchronization phase The total iteration time of stage s is expressed as: （1） Minimize the total iteration time of stage s. The expression of the optimization objective is: （2） Among them, the time of the start of stage s , and estimate it, the expression is: （3） in, and Respectively i The computation time of each stage during the forward propagation process and the communication time to the next stage; All-Reduce is used for gradient synchronization. For stage s, the gradient synchronization time is , whose expression is: （4） in, For a data parallel device group, the communication volume in the All-Reduce phase is ,The communication bandwidth depends on the minimum connection bandwidth between all ,devices; Next, determine the execution time of each stage. Define the stage with the fewest bubbles in the pipeline as the pivot stage. Based on the execution time of the pivot stage, the execution time of other stages can be deduced. The expression for the execution time of the pivot stage is: （5） in, M is the number of micro-batches processed in one iteration; and The forward and backward propagation calculation times of stage s are respectively; the execution time of other stages is calculated by the execution time of the pivot stage, and its expression is: （6） Based on the above formula, we can get 、 、 ,and , that is, to obtain the iteration time of each stage, when i When the value is s, the expressions are: in, The batch size allocated to device d; and In the l At the layer, the batch size processed by device d is , the execution time of forward and backward propagation; and The batch size processed by device d is When , the magnitude of the activation value transmitted by forward propagation and the gradient transmitted by backward propagation in stage s; Will It is represented as dividing the L-layer continuous neural network into S stages and mapping them to the parallel planning delay on N heterogeneous devices, whose goal is to minimize , the expression is: （8） in, It has the optimal substructure property and is decomposed into two subproblems: (1) an independent stage, which contains l Layer neural network, placed on n devices, using data parallelism, its time is given by Represents; (2) an optimal sub-configuration , which consists of the last Layer neural network, divided into stage, and mapped to On each device, the optimal time for this subproblem is recorded as ; During the execution of the algorithm, the model partitioning points, the device grouping strategy, and the number of training samples allocated to the devices are recorded. Finally, the results are backtracked to identify the optimal model partitioning and task placement scheme.

3. The method according to claim 2, characterized in that Determine the pivot stage through iterative optimization: First, evaluate each stage as a candidate pivot stage. For each candidate pivot stage, calculate the total iteration time of the entire training process. The candidate pivot stage with the shortest iteration time is selected as the pivot stage for this training. Through the above iterative process, the impact of each stage as the pivot stage on the overall training time is dynamically evaluated, and the stage that minimizes the training iteration time is selected.

4. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein, when the program is running, the device where the computer-readable storage medium is located is controlled to execute the deep learning model training method applicable to a heterogeneous network and device environment according to any one of claims 1 to 3.

5. An electronic device, characterized in that: include: one or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the device, enable the device to execute the deep learning model training method suitable for a heterogeneous network and device environment as described in any one of claims 1 to 3.

Images