A pipeline parallel training method suitable for heterogeneous devices

Abstract

The present invention relates to the field of deep learning technology, and in particular, provides a pipelined parallel training method suitable for heterogeneous devices. The method includes constructing a joint training architecture for heterogeneous devices; optimizing the scheduling strategy for micro-batches on the pipeline based on the constructed joint training architecture; and designing a communication mode between heterogeneous devices based on the optimized scheduling strategy. This method fully utilizes the characteristics of heterogeneous hardware, improves training efficiency, and optimizes the communication strategy.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a pipeline parallel training method suitable for heterogeneous devices. Background Art

[0002] Deep learning has made rapid progress in recent years, particularly in the field of large language models, which has sparked a wave of research. From the early BERT and GPT series to the latest large language models such as GPT-4, Claude, and DeepSeek, they have demonstrated outstanding performance in a variety of fields, including natural language processing and computer vision. The success of these models is largely due to the growth in model parameters and the expansion of training data. As the scale of deep learning models continues to expand, a single device is no longer sufficient to meet training requirements.

[0003] In existing technologies, methods for training base models on heterogeneous devices have certain limitations:

[0004] First, there are the limitations of a single training strategy: Traditional training strategies are primarily designed for homogeneous equipment environments and fail to fully consider the numerous heterogeneous devices in a data center and the performance differences between them. This makes it difficult to fully utilize heterogeneous hardware resources and does not fully consider the speed of device updates. Furthermore, both model training and fine-tuning require larger training scales, especially larger batch sizes, which significantly impact the model's generalization capabilities.

[0005] Secondly, inefficient training strategies: Inefficient in heterogeneous environments, especially when dealing with devices with significantly different video memory capacities, devices with smaller video memory often become a bottleneck for training scale; during training, the need for devices to store multiple activation values ​​leads to a video memory bottleneck, affecting training efficiency. In particular, when training heterogeneous devices together, the hardware differences between the devices are not fully considered, resulting in further performance degradation. Figure 1 As shown in the figure, heterogeneous devices, such as the NVIDIA A100 and NVIDIA T4, have significant differences in video memory capacity and computing power. The NVIDIA A100 has 40GB of video memory, while the NVIDIA T4 has only 16GB. When training on the NVIDIA T4, as the model parameter size increases, the video memory limitation will continue to compress the available training scale, resulting in insufficient model generalization and unsatisfactory training results.

[0006] Finally, inefficient communication strategies: Existing heterogeneous training methods typically use multiple peer-to-peer communication strategies to exchange data between devices, resulting in inefficient communication. For example, during training, computational results from multiple NVIDIA T4 devices must be transferred to the NVIDIA A100 device. The NVIDIA A100 then aggregates the data and performs computations based on the different communication groups. This repetitive nature of data reception and aggregation results in communication delays during forward and backward propagation, significantly reducing overall training efficiency. Summary of the Invention

[0007] In view of this, the present invention provides a pipeline parallel training method suitable for heterogeneous devices, which is used to fully utilize the characteristics of heterogeneous hardware, improve training efficiency and optimize communication strategies.

[0008] In a first aspect, the present invention provides a pipeline parallel training method applicable to heterogeneous devices, the method comprising:

[0009] Step 1: Build a joint training architecture for heterogeneous devices;

[0010] Step 2: Based on the constructed heterogeneous device joint training architecture, optimize the scheduling strategy of micro-batches on the pipeline;

[0011] Step 3: Based on the optimized scheduling strategy, design the communication mode between heterogeneous devices.

[0012] Optionally, step 1 includes:

[0013] At the physical view level, a hybrid parallel strategy of data parallelism and pipeline parallelism is adopted. Based on data parallelism, the mini-batch dataset is divided into multiple data subsets based on the performance differences of heterogeneous devices, and assigned to different devices for parallel computing. Each device processes the assigned data subset based on its computing performance. Based on pipeline parallelism, the model is divided into multiple stages. Each stage has the same computing task, and a stage is jointly executed by a different number of devices based on the performance differences of the devices. During the forward propagation process, data flows through the devices in sequence, and within each stage, the devices process the corresponding computing tasks based on their own performance. Between different stages, data transmission between different types of devices constitutes a heterogeneous device communication group. During backward propagation, gradient information is transmitted in opposite directions between heterogeneous devices or homogeneous devices.

[0014] Logical view level: To address the performance differences between heterogeneous devices, a heterogeneous pipeline parallel GPipe load balancing strategy is adopted. In the design of heterogeneous pipeline parallelism, for devices with small video memory or weak computing power, multiple devices are combined into computing groups to jointly complete the computing tasks of a stage. Within each stage, multiple devices are homogeneous, while heterogeneous combinations exist across stages. Among them, based on the computing performance differences between multiple devices, the heterogeneous combinations across stages divide the input data into multiple data subsets and assign them to different devices for parallel computing.

[0015] Optionally, step 2 includes:

[0016] Het-1F1B uses a forward-backward parallel strategy for heterogeneous devices. First, by scheduling micro-batch backpropagation before optimization, the memory activated by the micro-batch forward propagation is released. This memory is then released after the corresponding backpropagation and reused by the next injected micro-batch, thus expanding the training scale. Second, as the model parameter scale gradually increases, the memory ratio between A100 and T4 will gradually increase from the original 3:1 to 5:1. Het-1F1B uses A100 as the first node to alleviate the pressure on the T4 device.

[0017] To address the deadlock problem caused by Het-1F1B using the same communication stream to both receive and send, a dedicated communicator is created based on forward and backward propagation to achieve a near-asynchronous effect. A synchronous method is used to update the model weights, and the pipeline is periodically refreshed to ensure that the same version of the weights is used in all stages and all micro-batches in the training iteration.

[0018] Optionally, step 3 includes:

[0019] In a pipeline parallel design, if the devices communicating between adjacent stages are homogeneous, data is sent in a point-to-point manner. If the devices communicating between adjacent stages are heterogeneous, multiple homogeneous low-performance devices are combined into a computing unit to jointly complete the computing tasks of a single pipeline stage to match the high-performance devices. To complete data transmission between adjacent stages, two communication modes between heterogeneous devices are designed:

[0020] For scenarios where multiple low-performance devices transmit data to high-performance devices, a heterogeneous device data aggregation (Het-Gather) communication mode is designed. After completing its computational task, each low-performance device transmits the result to the high-performance device through a dedicated communication group. To ensure efficient data transmission, a data aggregation mechanism is designed on the high-performance device side of the heterogeneous communication group. This mechanism implements asynchronous data reception and dynamic merging by pre-allocating memory buffers.

[0021] For scenarios where a high-performance device distributes data to multiple low-performance devices, a Heterogeneous Device Data Distribution (Het-Scatter) communication mode is designed. After a high-performance device completes its computing task, it first shards and divides the large-scale data. The data shards are then distributed in parallel to different low-performance devices through independent communication channels. Each low-performance device processes the data after receiving it. In a second aspect, an embodiment of the present invention provides a computer-readable storage medium, the computer-readable storage medium including a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the pipeline parallel training method applicable to heterogeneous devices in the first aspect or any possible implementation of the first aspect.

[0022] In a third aspect, an embodiment of the present invention provides an electronic device comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, enable the device to execute the pipelined parallel training method applicable to heterogeneous devices in the first aspect or any possible implementation of the first aspect.

[0023] In the technical solution provided by the present invention, the method includes constructing a joint training architecture for heterogeneous devices; optimizing the scheduling strategy of micro-batches on the pipeline based on the constructed joint training architecture for heterogeneous devices; and designing a communication mode between heterogeneous devices based on the optimized scheduling strategy. This method fully utilizes the characteristics of heterogeneous hardware, improves training efficiency, and optimizes the communication strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A schematic diagram illustrating the differences in memory capacity and computing power between heterogeneous devices provided by an embodiment of the present invention;

[0026] Figure 2 A flowchart of a pipeline parallel training method applicable to heterogeneous devices provided by an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of the physical view of the heterogeneous device joint training architecture provided by an embodiment of the present invention;

[0028] Figure 4A schematic diagram of the logical view of the heterogeneous device joint training architecture provided by an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of an optimized scheduling strategy provided by an embodiment of the present invention;

[0030] Figure 6 Schematic diagram of the communication mechanism provided by an embodiment of the present invention, wherein (a) is homogeneous device communication; (b) is heterogeneous device communication;

[0031] Figure 7 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0033] It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0034] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0035] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the associated objects.

[0036] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0037] Figure 2 The flowchart of the pipeline parallel training method applicable to heterogeneous devices provided by the embodiment of the present invention is as follows: Figure 2 As shown, the method includes:

[0038] Step 1: Build a heterogeneous device joint training architecture.

[0039] In the embodiment of the present invention, Figure 3 and Figure 4 As shown, step 1 includes:

[0040] The heterogeneous device joint training architecture is based on the physical view level and the logical view level, making full use of the performance of different devices to perform different computing tasks, aiming to achieve performance balance among heterogeneous devices while ensuring efficient use of computing resources.

[0041] The training process of a deep neural network consists of two main phases: forward propagation and backpropagation. During the forward propagation phase, input data flows sequentially through each layer of the network, undergoing a series of linear transformations and nonlinear activation functions to generate a prediction. The loss between the prediction and the true label is then calculated to quantify the model's prediction error. During the backpropagation phase, the chain rule is used to calculate the gradient of the loss function with respect to the weights layer by layer, propagating the error forward from the output layer to the first layer of the network. The model weights are updated based on the learning rate and gradient to gradually reduce the error and improve model performance. This process is iterated until the model converges or the preset number of training rounds is reached. In pipeline parallelism, the idle time when devices wait for each other is called a bubble.

[0042] At the physical view level: a hybrid parallel strategy of data parallelism and pipeline parallelism is adopted. Based on data parallelism, data parallelism is carried out between devices in each stage of Stage 1, Stage 2, and Stage 3. Data parallelism is also carried out between A100 and A100, between T4 and T4, and between A100 and T4 within each stage. The mini-batch data set is divided into multiple data subsets based on the performance differences of heterogeneous devices and assigned to different devices for parallel computing. Each device processes the assigned data subset based on its computing performance. Data parallelism effectively compensates for the performance differences of heterogeneous devices and achieves near-linear expansion of the training scale of heterogeneous devices through fine-grained division of data parallelism. Based on pipeline parallelism, Stage1, Stage2, and Stage3 are pipelined in parallel across devices across stages. Pipeline parallelism is also used between A100 and A100, between T4 and T4, and between A100 and T4 across stages. By dividing the model into multiple stages, the computing tasks of each stage are the same. Different numbers of devices are combined to execute a stage based on the performance differences of the devices. Therefore, there may be different numbers of devices in each stage. During forward propagation, data flows through the devices in sequence. In each stage, the devices process the corresponding computing tasks based on their own performance. Between different stages, data transmission between different types of devices constitutes a heterogeneous device communication group. During reverse propagation, gradient information is transmitted in opposite directions between heterogeneous devices or homogeneous devices.

[0043] Logical view level: In view of the performance differences between heterogeneous devices, a GPipe load balancing strategy based on heterogeneous pipeline parallelism is adopted; in the design of heterogeneous pipeline parallelism, for devices with small video memory or weak computing power (such as T4), multiple devices are combined into computing groups to jointly complete the computing tasks of a stage. In each stage, multiple devices are homogeneous, and there will be heterogeneous combinations across stages; among them, according to the computing performance differences between multiple devices, the heterogeneous combination across stages divides the input data into multiple data subsets and assigns them to different devices for parallel computing. Compared with just using A100 as an ordinary high-end T4, GPipe based on heterogeneous versions effectively reduces the idle time of high-end devices, thereby effectively balancing the performance gap with devices with larger video memory. Such as Figure 4 As shown, Stage 2 and Stage 3 form a heterogeneous communication combination. During forward propagation, the A100 mini-batch needs to be further split according to the computing power characteristics of the T4. During backward propagation, the gradients of multiple T4 devices need to be aggregated to the A100 device. In Stage 3 and Stage 4 of the pipeline, three T4 devices are used to receive data sent by the A100, thereby balancing the performance gap between A100 and T4.

[0044] Step 2: Based on the constructed heterogeneous device joint training architecture, optimize the scheduling strategy of micro-batches on the pipeline.

[0045] The heterogeneous device joint training architecture in the present invention adopts a GPipe load balancing strategy based on heterogeneous pipeline parallelism, which effectively alleviates the contradiction caused by the performance difference between high-performance devices and low-performance devices. However, this scheduling method still has problems for the heterogeneous device collaborative training base model. First, the training scale will be limited by the economical equipment with weaker performance. For example, when using T4 for training, as the scale of model parameters increases, the graphics memory limitation will continue to compress the available training scale. The smaller batch size leads to insufficient model generalization and unsatisfactory training results. Secondly, in the GPipe pipeline parallel algorithm, all activations generated by micro-batch forward propagation are retained in memory and are not released until the corresponding back-propagation task is completed, resulting in memory requirements proportional to the number of micro-batches scheduled simultaneously, further exacerbating the fact that GPUs with smaller memory in heterogeneous clusters are more likely to become bottlenecks, limiting the scalability of training scale, and significantly increasing the memory overhead of computing resources.

[0046] In the embodiment of the present invention, Figure 5 As shown, step 2 includes:

[0047] To address the new challenges introduced by GPipe in heterogeneous device environments, a heterogeneous one forward one backward parallelism (Het-1F1B) strategy is adopted. First, by scheduling micro-batch backpropagation before optimization, the memory activated by the forward propagation of the micro-batch is released. This memory is then released after the corresponding backpropagation and reused by the next injected micro-batch, thus expanding the training scale. Second, to address the specific challenges caused by device heterogeneity, namely, as the scale of model parameters gradually increases, the memory ratio between A100 and T4 will gradually increase from the original 3:1 to 5:1. Het-1F1B uses A100 as the first node to alleviate the pressure on the T4 device.

[0048] In this embodiment of the present invention, taking Stage 4 as an example, micro-batches 1 and 1' represent two batches of data that are data-parallel. After the forward propagation of micro-batches 1 and 1' is completed, the backward propagation of micro-batches 1 and 1' is started immediately, thereby releasing the activated memory generated in the forward propagation process. When the backward propagation calculation of all micro-batches is completed, the parameters are updated and the next iteration begins.

[0049] In the embodiment of the present invention, Figure 5All-Reduce (AR) is a communication operation in parallel computing and distributed computing, which is used to aggregate data between multiple computing nodes (such as GPUs or CPUs). It is used in the distributed training process in deep learning to integrate the gradient information on each node and perform parameter updates.

[0050] To address the deadlock issue caused by Het-1F1B using the same communication stream for both receiving and sending, a dedicated communicator was created for forward and backpropagation, achieving a near-asynchronous effect. A synchronous approach was used to update the model weights, periodically refreshing the pipeline to ensure that the same version of the weights is used across all stages and micro-batches in a training iteration, without introducing stale weight versions. From an algorithmic perspective, this synchronous approach is equivalent to the standard and well-proven mini-batch stochastic gradient descent algorithm, thus ensuring convergence during model training.

[0051] Step 3: Based on the optimized scheduling strategy, design the communication mode between heterogeneous devices.

[0052] In the embodiment of the present invention, Figure 6 As shown, step 3 includes:

[0053] In the pipeline parallel design, if the devices communicating between adjacent stages are homogeneous, point-to-point transmission is adopted, such as Figure 6 As shown in (a) in the figure, point-to-point communication is used between homogeneous devices. In stages i to i+1, whether it is communication between A100 GPUs or between T4 GPUs, data can be directly transferred because the computing power and memory capacity of the devices are comparable. This method is simple and efficient and is suitable for communication between homogeneous GPUs. If the devices communicating between adjacent stages are heterogeneous, multiple homogeneous low-performance devices (such as multiple T4 GPUs) are combined into a computing unit to jointly complete the computing tasks of a single pipeline stage to match high-performance devices (such as A100 GPUs). Using traditional point-to-point communication is prone to repeated reception and transmission, resulting in inefficiency. Figure 6 As shown in (b) in the figure, in order to complete data transmission between adjacent stages, two communication modes between heterogeneous devices are designed:

[0054] For scenarios where multiple low-performance devices transmit data to high-performance devices, a heterogeneous device data gathering (Het-Gather) communication mode was designed. After completing its computational task, each low-performance device transmits the result to the high-performance device through a dedicated communication group. Taking the A100 and T4 as an example, after completing its computational task, each T4 transmits the result to the A100 GPU through a dedicated communication group. To ensure efficient data transmission, a data aggregation mechanism is designed on the high-performance device (A100 GPU) side of the heterogeneous communication group. This mechanism implements asynchronous data reception and dynamic merging by pre-allocating memory buffers, thereby reducing latency and memory overhead during data transmission.

[0055] For scenarios where high-performance devices distribute data to multiple low-performance devices, a heterogeneous device data scatter (Het-Scatter) communication mode is designed. After the high-performance device (A100 GPU) completes its computing task, it first shards and divides the large-scale data, and then distributes the data shards in parallel to different low-performance devices (T4 GPUs) through independent communication channels. Each low-performance device (T4 GPU) processes the data after receiving it.

[0056] In the technical solution provided by the present invention, the method includes constructing a joint training architecture for heterogeneous devices; optimizing the scheduling strategy of micro-batches on the pipeline based on the constructed joint training architecture for heterogeneous devices; and designing a communication mode between heterogeneous devices based on the optimized scheduling strategy. This method fully utilizes the characteristics of heterogeneous hardware, improves training efficiency, and optimizes the communication strategy.

[0057] Each step of the embodiment of the present invention may be performed by an electronic device, including but not limited to a tablet computer, a portable PC, a desktop computer, etc.

[0058] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is running, the electronic device where the computer-readable storage medium is located is controlled to execute the above-mentioned embodiment of the pipeline parallel training method applicable to heterogeneous devices.

[0059] Figure 7 A schematic diagram of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 7As shown, the electronic device 21 includes: a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, the pipeline parallel training method applicable to heterogeneous devices in the embodiment is implemented. To avoid repetition, they are not described here one by one.

[0060] The electronic device 21 includes, but is not limited to, a processor 211 and a memory 212. Those skilled in the art will understand that Figure 7 It is only an example of the electronic device 21 and does not constitute a limitation of the electronic device 21. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0061] The processor 211 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0062] The memory 212 can be an internal storage unit of the electronic device 21, such as the hard drive or memory of the electronic device 21. The memory 212 can also be an external storage device of the electronic device 21, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 21. Furthermore, the memory 212 can include both the internal storage unit of the electronic device 21 and an external storage device. The memory 212 is used to store computer programs and other programs and data required by the network device. The memory 212 can also be used to temporarily store data that has been output or is about to be output.

[0063] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0064] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A pipeline parallel training method suitable for heterogeneous devices, characterized in that: The method comprises: Step 1: Build a joint training architecture for heterogeneous devices; Step 2: Based on the constructed heterogeneous device joint training architecture, optimize the scheduling strategy of micro-batches on the pipeline; Step 3: Based on the optimized scheduling strategy, design the communication mode between heterogeneous devices; The step 1 comprises: At the physical view level, a hybrid parallel strategy of data parallelism and pipeline parallelism is adopted. Based on data parallelism, the mini-batch dataset is divided into multiple data subsets based on the performance differences of heterogeneous devices, and assigned to different devices for parallel computing. Each device processes the assigned data subset based on its computing performance. Based on pipeline parallelism, the model is divided into multiple stages. Each stage has the same computing task, and a stage is jointly executed by a different number of devices based on the performance differences of the devices. During the forward propagation process, data flows through the devices in sequence, and within each stage, the devices process the corresponding computing tasks based on their own performance. Between different stages, data transmission between different types of devices constitutes a heterogeneous device communication group. During backward propagation, gradient information is transmitted in opposite directions between heterogeneous devices or homogeneous devices. Logical view level: To address performance differences between heterogeneous devices, a GPipe load balancing strategy based on heterogeneous pipeline parallelism is adopted. In the heterogeneous pipeline parallel design, for devices with small video memory or weak computing power, multiple devices are combined into computing groups to jointly complete the computing tasks of a stage. Within each stage, multiple devices are homogeneous, but heterogeneous combinations exist across stages. Specifically, based on the computing performance differences between multiple devices, the heterogeneous combination across stages divides the input data into multiple data subsets and assigns them to different devices for parallel computing. The step 2 includes: Het-1F1B uses a forward-backward parallel strategy for heterogeneous devices. First, by scheduling micro-batch backpropagation before optimization, the memory activated by the micro-batch forward propagation is released. This memory is then released after the corresponding backpropagation and reused by the next injected micro-batch, thus expanding the training scale. Second, as the model parameter scale gradually increases, the memory ratio between A100 and T4 will gradually increase from the original 3:1 to 5:

1. Het-1F1B uses A100 as the first node to alleviate the pressure on the T4 device. To address the deadlock problem caused by Het-1F1B using the same communication stream to both receive and send, a dedicated communicator is created based on forward and backward propagation to achieve a near-asynchronous effect. A synchronous method is used to update the model weights, and the pipeline is periodically refreshed to ensure that the same version of the weights is used in all stages and all micro-batches in the training iteration.

2. The method according to claim 1, characterized in that The step 3 includes: In a pipeline parallel design, if the devices communicating between adjacent stages are homogeneous, data is sent in a point-to-point manner. If the devices communicating between adjacent stages are heterogeneous, multiple homogeneous low-performance devices are combined into a computing unit to jointly complete the computing tasks of a single pipeline stage to match the high-performance devices. To complete data transmission between adjacent stages, two communication modes between heterogeneous devices are designed: For scenarios where multiple low-performance devices transmit data to high-performance devices, a heterogeneous device data aggregation (Het-Gather) communication mode is designed. After completing its computational task, each low-performance device transmits the result to the high-performance device through a dedicated communication group. To ensure efficient data transmission, a data aggregation mechanism is designed on the high-performance device side of the heterogeneous communication group. This mechanism implements asynchronous data reception and dynamic merging by pre-allocating memory buffers. For scenarios where high-performance devices distribute data to multiple low-performance devices, a Heterogeneous Device Data Distribution (Het-Scatter) communication mode is designed. After a high-performance device completes its computing task, it first shards and divides the large-scale data, and then distributes the data shards in parallel to different low-performance devices through independent communication channels. Each low-performance device processes the data after receiving it.

3. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the pipeline parallel training method applicable to heterogeneous devices according to any one of claims 1 to 2.

4. An electronic device, characterized in that: include: one or more processors; Memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, enable the device to perform the pipeline parallel training method suitable for heterogeneous devices as described in any one of claims 1 to 2.

Images