Pipeline parallel distributed training method, device and system for deep neural network

Abstract

The application provides a pipeline parallel distributed training method, device and system for a deep neural network. The method comprises: sequentially transmitting each group of small batch training data constituting a current batch of training data to an edge server via an optical network, so that the edge server and the cloud server cooperatively perform asynchronous parallel cooperative training on different sub-task models constituting the deep neural network in a communication and training decoupling manner, and the edge server sequentially outputs the gradients corresponding to each group of small batch training data; and sequentially receiving the gradients of each group of small batch training data. The application can ensure correct transmission of data during model training, reduce communication overhead during training, achieve load balancing between devices, improve model training efficiency, effectiveness and resource utilization of devices participating in training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model training, and particularly relates to a pipeline parallel distributed training method, device and system for a deep neural network. BACKGROUND

[0002] With the rapid development of information technology, such as digital twin, Chat GPT and large model, the number of applications with high requirements for computing and latency is increasing. At the same time, the progress of artificial intelligence, especially the key role of deep neural network DNN in solving complex problems such as image recognition, target tracking and language conversion, makes the demand for powerful computing capacity grow. However, with the diversification of applications and the surge in data volume, the size of DNN models is also expanding, which requires more powerful computing resources to support the training of these large models.

[0003] To address this challenge, the European Telecommunications Standards Institute ETSI proposes the concept of mobile edge computing MEC, which allows computing tasks to be transferred to edge servers for timely processing. However, the computing resources of edge servers are limited, while cloud servers can provide sufficient computing and storage resources, but may produce large communication delay due to their distance from users, which does not meet the strict requirements of low latency applications. Therefore, in order to meet these needs, a system must be established that has sufficient computing resources and can provide high bandwidth and low latency communication links between cloud and edge servers. One possible solution is to use cloud-edge collaborative optical networks to improve the efficiency of communication and computing. Optical networks are known for their large capacity, high reliability and long distance transmission capabilities. In cloud-edge collaborative optical networks, in order to reduce the training delay of multi-layer DNN tasks, the tasks can be decomposed, with one part executed on the edge server and the other part handled by the cloud server, which not only reduces the delay of data transmission, but also reduces the computing burden of the central cloud.

[0004] In the field of computer science, how to train large neural network models on hardware accelerators with multiple interconnection capabilities, such as graphics processing units (GPUs), has attracted widespread attention. Common solutions include data parallelism, model parallelism, and pipeline parallelism, among other parallel training techniques that can improve training efficiency. Distributed parallel training can distribute different stages of model training to different computing nodes for concurrent execution, reducing waiting time and improving resource utilization. Different parallel training methods have different effects on performance, efficiency, and model accuracy. However, these studies are usually conducted in data center environments and may not fully consider the impact of communication delays. In cloud-edge collaborative optical networks, communication delays become an important factor due to the long physical distance between computing nodes. Therefore, studying how to efficiently deploy distributed model training in cloud-edge collaborative optical networks is crucial for accelerating AI applications in optical networks.

[0005] However, although current execution of existing asynchronous training methods such as pipeline parallelism in cloud-edge collaborative optical networks is faster than synchronous training, it often results in a decrease in result quality because the gradients used in asynchronous training may be calculated based on outdated parameters, which can lead to unstable training; at the same time, the additional communication overhead required for data exchange between different stages also affects training efficiency; in addition, if the computational complexity of each stage is inconsistent, it may cause some devices to be idle while others are busy computing, resulting in load imbalance. SUMMARY

[0006] In view of this, the embodiments of the present application provide a pipeline parallel distributed training method, device and system for deep neural networks to eliminate or improve one or more defects in the prior art.

[0007] One aspect of the present application provides a pipeline parallel distributed training method for deep neural networks, comprising:

[0008] For the current iteration round of the deep neural network, a preset pipeline parallel distributed training step is performed, wherein the pipeline parallel distributed training step includes: sequentially transmitting each group of mini-batch training data constituting the current batch training data to an edge server via an optical network, so that the edge server and a cloud server communicatively connected to the edge server via the optical network perform asynchronous parallel collaborative training on different sub-task models constituting the deep neural network based on a preset communication and training decoupling manner, and the edge server sequentially outputs the gradients corresponding to each group of mini-batch training data.

[0009] The gradients corresponding to each group of mini-batch training data returned by the edge server via the optical network are sequentially received.

[0010] In some embodiments of the present application, further comprising:

[0011] If the deep neural network is not currently converged, the current batch of training data has not been used for model training, or the current iteration round is not the last preset iteration round, when the gradient corresponding to a batch of small batch training data is first received in the current iteration round, performing the pipeline parallel distributed training step for the next iteration round of the deep neural network.

[0012] In some embodiments of the present application, the edge server is configured to perform the following:

[0013] Receiving each group of small batch training data in turn, and starting to perform a preset first asynchronous training step when a group of small batch training data is first received in the current iteration round, to obtain the intermediate activation value corresponding to each group of small batch training data respectively; and transmitting the intermediate activation value corresponding to each group of small batch training data to the cloud server through the optical network each time the intermediate activation value corresponding to a group of small batch training data is obtained.

[0014] The first asynchronous training step comprises: performing forward propagation calculation for the first sub-task model corresponding to the deep neural network based on each group of small batch training data respectively, so that the first sub-task model outputs the intermediate activation value corresponding to each group of small batch training data in turn.

[0015] In some embodiments of the present application, the cloud server is configured to perform the following:

[0016] Receiving the intermediate activation value corresponding to each group of small batch training data in turn, and starting to perform a preset second asynchronous training step when the intermediate activation value corresponding to a group of small batch training data is first received in the current iteration round, to obtain the intermediate loss value corresponding to each group of small batch training data respectively; and transmitting the intermediate loss value corresponding to each group of small batch training data to the edge server through the optical network each time the intermediate loss value corresponding to a group of small batch training data is obtained.

[0017] The second asynchronous training step comprises: sequentially performing a preset calculation step for the intermediate activation value corresponding to each group of small batch training data, and the calculation step comprises: sequentially performing forward propagation calculation and backward propagation calculation for the second sub-task model corresponding to the deep neural network based on the intermediate activation value corresponding to the current group of small batch training data, to obtain the intermediate loss value corresponding to the group of small batch training data.

[0018] In some embodiments of the present application, the edge server is further configured to perform the following:

[0019] receiving the intermediate loss value corresponding to each group of the small batch training data in sequence, and starting to perform a preset third asynchronous training step to obtain the gradient corresponding to each group of the small batch training data when the intermediate loss value corresponding to a group of the small batch training data is received for the first time in the current iteration round; and returning the gradient corresponding to each group of the small batch training data to the optical network each time the gradient corresponding to a group of the small batch training data is obtained; wherein, if the current iteration round is not the first iteration round, the edge server alternately performs the third asynchronous training step and the first asynchronous training step.

[0020] The third asynchronous training step comprises: performing back propagation calculation for the first sub-task model based on the intermediate loss value corresponding to each group of the small batch training data respectively, so that the first sub-task model outputs the gradient corresponding to each group of the small batch training data in sequence.

[0021] In some embodiments of the present application, before performing the preset pipeline parallel distributed training step in the current iteration round of the deep neural network, the method further comprises:

[0022] submitting a training task for the deep neural network to an optical network controller, so that the optical network controller obtains a target partition point for the deep neural network according to task requirement data corresponding to the training task and current resource states of the edge server and the cloud server respectively, the optical network controller divides the deep neural network into a first sub-task model and a second sub-task model according to the target partition point, the optical network controller allocates the first sub-task model to the edge server and allocates the second sub-task model to the cloud server, and then returns a corresponding model task partition message;

[0023] receiving the model task partition message returned by the optical network controller.

[0024] Another aspect of the present application provides a pipeline parallel distributed training device for a deep neural network, comprising:

[0025] a pipeline parallel distributed training module configured to perform a preset pipeline parallel distributed training step for a current iteration round of the deep neural network, wherein the pipeline parallel distributed training step comprises: sequentially transmitting each group of mini-batch training data constituting a current batch of training data to the edge server via the optical network, so that the edge server and the cloud server communicatively connected to the edge server via the optical network perform asynchronous parallel collaborative training on different sub-task models constituting the deep neural network based on a preset communication and training decoupling manner, and sequentially output the gradients corresponding to each group of the mini-batch training data by the edge server.

[0026] a gradient receiving module configured to sequentially receive the gradients corresponding to each group of the mini-batch training data returned by the edge server via the optical network.

[0027] A third aspect of the present application provides a pipeline parallel distributed training system for a deep neural network, comprising: a terminal device, an edge server and a cloud server sequentially communicatively connected via an optical network.

[0028] The terminal device is provided with a pipeline parallel distributed training device for a deep neural network, which is configured to perform the pipeline parallel distributed training method for a deep neural network.

[0029] A fourth aspect of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the pipeline parallel distributed training method for a deep neural network.

[0030] A fifth aspect of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the pipeline parallel distributed training method for a deep neural network.

[0031] A sixth aspect of the present application provides a computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the pipeline parallel distributed training method for a deep neural network.

[0032] The pipeline parallel distributed training method for the deep neural network provided in the application is executed for the preset pipeline parallel distributed training step of the current iteration round of the deep neural network, wherein the pipeline parallel distributed training step comprises: sequentially transmitting each group of small batch training data constituting the current batch training data to an edge server through an optical network, so that the edge server and the cloud server connected with the edge server through the optical network communication connection perform asynchronous parallel collaborative training on different sub-task models for constituting the deep neural network based on the preset communication and training decoupling mode, and the edge server sequentially outputs the gradient corresponding to each group of small batch training data; and sequentially receiving the gradient corresponding to each group of small batch training data returned by the edge server through the optical network, which can ensure correct transmission of data in the deep neural network training process, reduce communication overhead in the training process, realize load balancing between devices participating in training, improve the training efficiency and effectiveness of the deep neural network, and improve the resource utilization rate of the terminal device, the edge server and the cloud server participating in the deep neural network training.

[0033] Additional advantages, objects, and features of the application will be set forth in part in the description which follows, and in part will become apparent to those having ordinary skill in the art upon examination of the following or can be learned from practice of the application. The objects and other advantages of the application can be realized and attained by the structure particularly pointed out in the specification and claims hereof as well as the appended drawings.

[0034] It will be understood by those skilled in the art that the objects and advantages of the present application can be realized and attained by the structure particularly pointed out in the specification and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0035] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. The components in the drawings are not drawn to scale, but are merely intended to illustrate the principles of the application. Some parts of the drawings can be enlarged for the purpose of illustrating and describing some parts of the application, i.e., can become larger than other parts in the exemplary devices actually manufactured according to the application. In the drawings:

[0036] Figure 1 The first flowchart of the pipeline parallel distributed training method for the deep neural network in an embodiment of the application.

[0037] Figure 2 The second flowchart of the pipeline parallel distributed training method for the deep neural network in an embodiment of the application.

[0038] Figure 3A schematic diagram of partitioning and training logic for a deep neural network in an example of the present application.

[0039] Figure 4 A schematic diagram of a timeline for asynchronous parallel training in another example of the present application.

[0040] Figure 5 A schematic diagram of the structure of a pipelined parallel distributed training apparatus for a deep neural network in an embodiment of the present application.

[0041] Figure 6 A schematic diagram of the structure of a pipelined parallel distributed training system for a deep neural network in an embodiment of the present application.

[0042] Figure 7 A schematic diagram of a process for determining DNN task partitioning in an application example of the present application.

[0043] Figure 8 A schematic diagram of a specific process for DNN task training in an application example of the present application. DETAILED DESCRIPTION

[0044] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the embodiments and the accompanying drawings. Herein, the illustrative embodiments of the present application and their descriptions are used to explain the present application, but are not used as limitations to the present application.

[0045] It should also be noted that, in order to avoid the present application being obscured by unnecessary details, only structures and / or processing steps closely related to the solutions according to the present application are shown in the accompanying drawings, and other details not closely related to the present application are omitted.

[0046] It should be emphasized that the term "comprises / comprising" when used in this text means the presence of the stated features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0047] It should also be noted that, if not specifically stated, the term "connected" in this text can mean not only direct connection, but also indirect connection in the presence of an intermediate.

[0048] In the following, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0049] It should be noted that in the existing pipeline parallel training scheme executed in the cloud edge collaborative optical network, the model is partitioned across multiple GPUs, and each GPU is only responsible for a part of the model. Model parallelization can usually achieve faster training time than data parallelization, because using a large mini-batch size can improve statistical efficiency. Model parallel DNN training results in a serious underutilization of GPU resources. Therefore, pipeline parallelism is introduced on the basis of model parallelism. The core idea of pipeline parallelism is: on the basis of model parallelism, further introduce data parallelism, that is, divide the original data into several batch data batches, and send them to the GPU for training. The data before division can be called mini-batch training data, which can be written as mini-batch or minibatch. The training data divided on the mini-batch is called micro-batch. That is, each mini-batch of training data appearing in one or more embodiments of the present application refers to each micro-batch formed by dividing a batch of data in the training data.

[0050] Specifically, the model layers to be trained are divided into multiple stages, each stage containing a consecutive set of layers in the model. Each stage is mapped to a separate GPU that performs forward and backward passes for all layers in the stage. A simple pipeline parallel assignment splits the DNN across four machines. In the simplest case, only one minibatch is active in the system, just like in traditional model-parallel training. In an example of a configuration timeline of computations, there are four machines (first machine to fourth machine) and one active minibatch in the configuration. In the forward stage, each stage performs a forward pass of the minibatch for the layers in the stage and sends the results to the next stage. The output stage computes the loss for the minibatch after completing its forward pass. In the backward stage, each stage performs a backward pass and propagates the loss to the previous stage. Only one minibatch is active, and at most one GPU is active at any given point in time. To ensure that no GPU is idle at any point in time, multiple minibatches are injected into the pipeline one after the other, thus enhancing model-parallel training through pipelining. The asynchronous communication of forward output activations and cross-stage backward gradients after completing a minibatch's forward pass results in a large overlap between communication and the computation of the subsequent minibatch, thus achieving better hardware efficiency transfer compared to BSP, where each stage asynchronously sends output activations to the next stage while starting to process another minibatch.

[0051] It can be understood that the existing pipeline parallel training scheme executed in the cloud edge collaborative optical network has the following problems:

[0052] (1) Communication overhead: The data exchange between different stages of pipeline parallelism requires additional communication overhead, which is not fully considered in the prior art and can become a performance bottleneck. Moreover, in cloud-edge collaborative optical networks, the distance between the cloud and the edge server is usually far, and the communication time cannot be ignored.

[0053] (2) Load balancing: If the computational complexity of each stage in the pipeline parallelism scheme is inconsistent, it may cause some devices to be idle while others are busy with computation, resulting in load imbalance.

[0054] (3) Asynchronous pacing problem: The parameters and gradients used for updating in existing pipeline parallelism schemes do not come from the same iteration. The gradient used for updating may be calculated by the parameters several steps ago. This asynchronous pacing is difficult to unify. If the asynchronous pacing cannot be accurately controlled to be consistent, it is difficult to ensure correct data transmission.

[0055] Therefore, in order to solve the above problems, the embodiments of the present application respectively provide a pipeline parallel distributed training method for a deep neural network, a pipeline parallel distributed training device for a deep neural network for executing the pipeline parallel distributed training method for a deep neural network, a pipeline parallel distributed training system for a deep neural network, an entity device, a computer readable storage medium and a computer program product, which aims to accelerate the training of artificial intelligence models by jointly computing and communicating resources, reduce the waste of server computing or communication resources caused by the waiting time of intermediate results in service. At the same time, improve the training stability, and ensure high efficiency and low delay in long-distance transmission and large-scale computation. The scheme improves the computing efficiency and reduces the delay of data transmission by performing different stages of model training in parallel on multiple computing nodes. In cloud-edge collaborative optical networks, considering the deterministic delay advantage of optical network transmission, the asynchronous parallel training mode is used to accelerate model training, ensure the stability of asynchronous pacing, and reduce the risk of model divergence to meet the low-delay-sensitive applications.

[0056] The embodiments are specifically described as follows.

[0057] Therefore, the embodiments of the present application provide a pipeline parallel distributed training method for a deep neural network, which can be implemented by a pipeline parallel distributed training device for a deep neural network, as shown in Figure 1 , the pipeline parallel distributed training method for a deep neural network specifically includes the following contents:

[0058] Step 100: performing a preset pipeline parallel distributed training step for a current iteration round of the deep neural network, wherein the pipeline parallel distributed training step comprises: sequentially transmitting each group of minibatch training data constituting a current batch training data to an edge server via an optical network, so that the edge server and a cloud server in communication connection with the edge server via the optical network perform asynchronous parallel collaborative training on different subtask models constituting the deep neural network based on a preset communication and training decoupling manner, and sequentially output the gradients corresponding to each group of minibatch training data by the edge server.

[0059] In one or more embodiments of the present application, the data used for training the deep neural network can be pre-divided into batches to form each group of batch training data, and further, each group of batch training data can be divided into multiple minibatches to form each group of batch training data corresponding to each minibatch training data. It can be understood that the number of data samples contained in the minibatch training data is less than the number of data samples in the batch training data. The type of data sample can be set according to the application requirements of the deep neural network, for example, if the deep neural network is finally used for image classification, the data sample is image data, which can be set according to actual application needs, and the present application does not limit this.

[0060] It can be understood that the communication and training decoupling manner means that the devices participating in the training, such as the edge server and the cloud server, decouple the communication and the computing process, and process them asynchronously. For example, the edge server is performing model training on the local subtask model based on the first group of minibatch training data of the current batch received in advance while receiving the second group of minibatch training data of the current batch. The edge server can use one or more of the following.

[0061] It can be understood that the edge server and the cloud server in communication connection with the edge server via the optical network in the step 100 perform asynchronous parallel collaborative training on different subtask models constituting the deep neural network based on a preset communication and training decoupling manner, and sequentially output the gradients corresponding to each group of minibatch training data by the edge server. The specific implementation process can include the following contents:

[0062] (1) the edge server receives each group of the small batch training data in turn, and when a group of the small batch training data is first received in the current iteration round, starts to perform a preset first asynchronous training step to obtain an intermediate activation value corresponding to each group of the small batch training data respectively; and when the intermediate activation value corresponding to each group of the small batch training data is obtained each time, the intermediate activation value corresponding to the group of the small batch training data is transmitted to the cloud server through an optical network;

[0063] (2) the cloud server receives the intermediate activation value corresponding to each group of the small batch training data in turn, and when the intermediate activation value corresponding to a group of the small batch training data is first received in the current iteration round, starts to perform a preset second asynchronous training step to obtain an intermediate loss value corresponding to each group of the small batch training data respectively; and when the intermediate loss value corresponding to each group of the small batch training data is obtained each time, the intermediate loss value corresponding to the group of the small batch training data is transmitted to the edge server through an optical network;

[0064] (3) the edge server receives the intermediate loss value corresponding to each group of the small batch training data in turn, and when the intermediate loss value corresponding to a group of the small batch training data is first received in the current iteration round, starts to perform a preset third asynchronous training step to obtain a gradient corresponding to each group of the small batch training data respectively; and when the gradient corresponding to each group of the small batch training data is obtained each time, the gradient corresponding to the group of the small batch training data is returned to the pipeline parallel distributed training device for the deep neural network through an optical network; wherein, if the current iteration round is not the first iteration round, the edge server alternately performs the third asynchronous training step and the first asynchronous training step.

[0065] Step 200: receiving the gradient corresponding to each group of the small batch training data returned by the edge server through the optical network in turn.

[0066] From the above description, it can be known that the pipeline parallel distributed training method for the deep neural network provided by the embodiments of the present application can ensure the correct transmission of data in the deep neural network training process, reduce the communication overhead in the training process, realize the load balancing between the devices participating in the training, and improve the training efficiency, effectiveness of the deep neural network, and resource utilization of the terminal device, the edge server and the cloud server participating in the training of the deep neural network.

[0067] In order to further improve the training efficiency of the deep neural network, in a pipeline parallel distributed training method for the deep neural network provided by an embodiment of the present application, referring to Figure 2 , the pipeline parallel distributed training method for the deep neural network further comprises the following content after step 200:

[0068] Step 300: if the deep neural network is not currently converged, there is currently the batch training data that has not been used for model training, or the current iteration round is not the last preset iteration round, when the gradient corresponding to one of the mini-batch training data is first received in the current iteration round, perform the pipeline parallel distributed training step for the next iteration round of the deep neural network.

[0069] It can be understood that in the foregoing step 100, if the current iteration round is the first iteration round for the deep learning network, the pipeline parallel distributed training device for the deep neural network will wait for a period of time after sequentially transmitting each group of mini-batch training data corresponding to the current batch training data to the edge server through the optical network, and then execute step 200 to start receiving the gradient corresponding to each group of the mini-batch training data returned by the edge server through the optical network corresponding to the current iteration round, and then execute step 300: if the deep neural network is not currently converged, there is currently the batch training data that has not been used for model training, or the current iteration round is not the last preset iteration round, when the gradient corresponding to one of the mini-batch training data is first received in the current iteration round, perform the pipeline parallel distributed training step for the next iteration round of the deep neural network.

[0070] In the step 100, if the current iteration round is not the first iteration round of the deep learning network, the pipeline parallel distributed training device for the deep neural network receives the gradient corresponding to the first batch of mini-batch training data of the previous iteration round first, and then starts to execute the step 100 based on the gradient, transmits the first batch of mini-batch training data corresponding to the current batch of training data to the edge server through the optical network, and then receives the gradient corresponding to the second batch of mini-batch training data of the previous iteration round, and then transmits the second batch of mini-batch training data corresponding to the current batch of training data to the edge server through the optical network based on the gradient, and so on. The gradient of the previous iteration round and the transmission of the mini-batch training data of the current iteration round are alternately executed. After the transmission of all the mini-batch training data corresponding to the current batch of training data is completed, the step 200 is executed to start to receive the gradient corresponding to each batch of mini-batch training data of the current iteration round returned by the edge server through the optical network, and then the step 300 is executed: if the deep neural network does not converge, the batch of training data has not been used for model training, or the current iteration round is not the last preset iteration round, when the gradient corresponding to one batch of mini-batch training data is received for the first time in the current iteration round, the pipeline parallel distributed training step is executed for the next iteration round of the deep neural network.

[0071] That is, the embodiments of the present application do not necessarily have a time sequence relationship between the steps, but can execute the next step at the same time as the previous step, or start to execute the next step during the execution of the previous step. The specific need is determined according to the condition limitation recorded in the step.

[0072] In order to further improve the model training speed and improve the resource utilization rate of the edge server, in the pipeline parallel distributed training method for the deep neural network provided by the embodiments of the present application, the edge server is used to execute the following content:

[0073] Step 01: sequentially receive each batch of mini-batch training data, and when a batch of mini-batch training data is received for the first time in the current iteration round, start to execute a preset first asynchronous training step to obtain the intermediate activation value corresponding to each batch of mini-batch training data respectively; and each time a batch of mini-batch training data is obtained, the intermediate activation value corresponding to the batch of mini-batch training data is transmitted to the cloud server through the optical network;

[0074] The first asynchronous training step comprises: performing forward propagation calculation of the first sub-task model corresponding to the deep neural network based on each group of the small batch training data respectively, so that the first sub-task model outputs the intermediate activation value corresponding to each group of the small batch training data respectively in turn.

[0075] In order to further improve the model training speed and improve the resource utilization rate of the cloud server, in the pipeline parallel distributed training method for the deep neural network provided in the embodiment of the present application, the cloud server is configured to perform the following content:

[0076] Step 02: sequentially receiving the intermediate activation value corresponding to each group of the small batch training data, and starting to perform the preset second asynchronous training step when the intermediate activation value corresponding to a group of the small batch training data is first received in the current iteration round, so as to obtain the intermediate loss value corresponding to each group of the small batch training data respectively; and transmitting the intermediate loss value corresponding to each group of the small batch training data to the edge server through the optical network each time the intermediate loss value corresponding to a group of the small batch training data is obtained.

[0077] The second asynchronous training step comprises: sequentially performing a preset calculation step on the intermediate activation value corresponding to each group of the small batch training data, and the calculation step comprises: sequentially performing forward propagation calculation and backward propagation calculation of the second sub-task model corresponding to the deep neural network based on the intermediate activation value corresponding to the current group of the small batch training data, so as to obtain the intermediate loss value corresponding to the group of the small batch training data.

[0078] In order to further improve the model training speed and improve the resource utilization rate of the edge server, in the pipeline parallel distributed training method for the deep neural network provided in the embodiment of the present application, the edge server is further configured to perform the following content:

[0079] Step 03: sequentially receiving the intermediate loss value corresponding to each group of the small batch training data, and starting to perform the preset third asynchronous training step when the intermediate loss value corresponding to a group of the small batch training data is first received in the current iteration round, so as to obtain the gradient corresponding to each group of the small batch training data respectively; and returning the gradient corresponding to each group of the small batch training data through the optical network each time the gradient corresponding to a group of the small batch training data is obtained; wherein, if the current iteration round is not the first iteration round, the edge server alternately performs the third asynchronous training step and the first asynchronous training step.

[0080] The third asynchronous training step includes: performing back propagation calculation for the first sub-task model based on the intermediate loss value corresponding to each group of the small batch training data respectively, so that the first sub-task model outputs the gradient corresponding to each group of the small batch training data in turn.

[0081] It can be understood that the steps 01 to 03 are sequentially executed to constitute a complete pipeline parallel distributed training step.

[0082] Specifically, the embodiment of the present application designs a cloud edge cooperative optical network composed of a cloud server and multiple edge servers. The training request of the deep neural network can be transmitted through the optical link, forwarded by the router, and processed on multiple servers. Referring to Figure 3 , the DNN request model can be divided into two parts, Batch1, Batch2, Batch3 and Batch4 represent different batches of training data. The first part of the deep neural network is deployed on the edge server, and the second part is deployed on the cloud server. Therefore, the training process of the deep neural network can be summarized as four stages:

[0083] 1. The training process of the deep neural network; data transmission from the terminal device to the source edge server.

[0084] 2. Source edge server preprocessing.

[0085] 3. Intermediate data transmission from the source edge server to the target server.

[0086] 4. Target server post-processing.

[0087] Among them, the training process of the deep neural network mainly includes three stages of forward propagation, back propagation and parameter update, which are iterated.

[0088] In the traditional asynchronous parallel mode, each batch updates the parameters immediately as needed without waiting for the results of the previous batch. However, the variability of each batch time block duration can lead to overly aggressive parameter updates, resulting in an unstable training process. Therefore, maintaining a uniform time block is crucial to ensure data consistency and computational consistency between all servers. In Figure 3 The pipeline asynchronous update process shown in the figure trains each batch of data in order, ensuring the stability of the asynchronous step and reducing the risk of divergence. In the proposed pipeline workflow structure, computation and communication are alternated. Among them, the computation time includes the process of DNN training forward propagation and back propagation. Each stage can be described as follows:

[0089] Stage 1: Transmission time between terminal device and edge server.

[0090] Stage 2: Compute time of the edge server, where DNN layers before the partition point are deployed.

[0091] Stage 3: Transmission time between the edge server and the cloud server.

[0092] Stage 4: Compute time of the cloud server, where DNN layers after the partition point are deployed.

[0093] Referring to Figure 4 , the incoming batch is divided into mini-batch training data. When the forward propagation of one mini-batch training data is completed, the output activation is sent to the next stage asynchronously by each stage while processing another mini-batch training data. After the forward propagation is completed, the last layer immediately starts to backward propagate one mini-batch. After completing the backward propagation, each stage sends the gradient to the previous stage asynchronously while calculating the next mini-batch. The calculation and communication in asynchronous parallel are highly parallel. Asynchronous communication across forward activation and backward gradient stages results in significant overlap between communication and subsequent mini-batch calculation. This greatly reduces the waiting time. It should be noted that the internal storage of the server, which contains a large amount of data such as parameters, will be released when it is no longer needed for future calculations. In this way, the performance of the server can be ensured. In Figure 4 , each small block represents a time block.

[0094] In order to further ensure the correct transmission of data in the deep neural network training process and reduce the communication overhead in the training process, in an embodiment of the pipeline parallel distributed training method for deep neural networks provided in the present application, referring to Figure 2 , the step 100 in the pipeline parallel distributed training method for deep neural networks further specifically comprises the following content before the step 100:

[0095] Step 010: submit a training task for a deep neural network to an optical network controller, so that the optical network controller obtains a target partition point for the deep neural network according to task requirement data corresponding to the training task and current respective resource states of the edge server and the cloud server, the optical network controller divides the deep neural network into a first sub-task model and a second sub-task model according to the target partition point, the optical network controller allocates the first sub-task model to the edge server and allocates the second sub-task model to the cloud server, and then returns a corresponding model task partition message.

[0096] The target partition point is the optimal partition point calculated by the optical network controller.

[0097] Step 020: receive the model task partition message returned by the optical network controller.

[0098] It can be understood that the optical network controller can be a software-defined optical network controller (which can be abbreviated as SDON controller or SDON).

[0099] From the software level, the application also provides a pipeline parallel distributed training device for a deep neural network for performing all or part of the pipeline parallel distributed training method for a deep neural network, which is described in detail with reference to Figure 5 , and the pipeline parallel distributed training device for a deep neural network specifically includes the following contents:

[0100] The pipeline parallel distributed training module 10 is configured to perform a preset pipeline parallel distributed training step for a current iteration round of the deep neural network, wherein the pipeline parallel distributed training step includes: sequentially transmitting each group of mini-batch training data constituting the current batch training data to the edge server through the optical network, so that the edge server and the cloud server connected to the edge server through the optical network perform asynchronous parallel collaborative training on different sub-task models constituting the deep neural network based on a preset communication and training decoupling mode, and the edge server sequentially outputs the gradients corresponding to each group of mini-batch training data.

[0101] The gradient receiving module 20 is configured to sequentially receive the gradients corresponding to each group of mini-batch training data returned by the edge server through the optical network.

[0102] The embodiments of the pipeline parallel distributed training device for a deep neural network provided by the application can be used to perform the processing procedures of the embodiments of the pipeline parallel distributed training method for a deep neural network described above, and the functions thereof will not be repeated here. For details, please refer to the detailed description of the embodiments of the pipeline parallel distributed training method for a deep neural network.

[0103] The pipeline parallel distributed training device for a deep neural network can be completed in a client device. Specifically, the processing capability of the client device and the use scenario of the user can be considered for selection. The application does not limit this. If all operations are completed in the client device, the client device can further include a processor for specific processing of the pipeline parallel distributed training for a deep neural network.

[0104] The client device (which can also be referred to as a terminal device) described above can have a communication module (i.e., a communication unit) that can be in communication connection with a remote edge server to realize data transmission with the edge server. The edge server can include a server on the side of a task scheduling center, and in other implementation scenarios, can also include a server of an intermediate platform, such as a server of a third-party server platform that is in communication link with the server of the task scheduling center. The edge server can include a single computer device, or can include a server cluster composed of multiple servers, or a server structure of a distributed device.

[0105] The edge server and the client device can use any suitable optical network protocol to communicate, including an optical network protocol that has not been developed as of the filing date of the present application.

[0106] As can be known from the foregoing description, the pipeline parallel distributed training apparatus for a deep neural network provided by the embodiments of the present application can ensure correct data transmission during deep neural network training, and can reduce communication overhead during training, can realize load balancing among devices participating in training, and can improve deep neural network training efficiency, effectiveness, and resource utilization of terminal devices, edge servers, and cloud servers participating in deep neural network training.

[0107] Based on the foregoing embodiments of the pipeline parallel distributed training method for a deep neural network and / or the pipeline parallel distributed training apparatus for a deep neural network, the present application further provides an embodiment of a pipeline parallel distributed training system for a deep neural network, as shown in Figure 6 , the pipeline parallel distributed training system for a deep neural network specifically includes the following content:

[0108] The terminal device, the edge server, and the cloud server are sequentially connected by an optical network.

[0109] The terminal device is provided with a pipeline parallel distributed training apparatus for a deep neural network, which is used to execute the pipeline parallel distributed training method for a deep neural network provided by the foregoing embodiments.

[0110] It can be understood that the pipeline parallel distributed training apparatus for a deep neural network is also in communication connection with an optical network controller.

[0111] To further illustrate the above embodiments, this application also provides a specific application example of a pipelined parallel distributed training method for deep neural networks. This method belongs to a pipelined parallel distributed training scheme oriented towards joint computing and optical network resources. By leveraging the deterministic latency advantage of optical network transmission, it fully considers the impact of computation and communication latency on task training. The deterministic latency effectively maintains the pace stability in asynchronous parallel training, thereby improving convergence speed and reducing the risk of training divergence. Typically, the efficiency of a DNN training task is determined by the partitioning points and resource deployment. The computation time of each training task is related to the computing resources provided on the deployment node and the task partitioning method, while the communication time is related to the provided bandwidth resources and the amount of communication data. The parallel mode also affects the overall efficiency of task training and the network. By decoupling computation and communication, computational and communication tasks can be performed independently, reducing their mutual influence and waiting time. This greatly accelerates model training and improves resource utilization.

[0112] To achieve cloud-edge collaborative training, this application example outlines a detailed DNN task training process. First, the DNN consists of a series of stacked layers, with the output of one layer fed into the input of the next. For a DNN training request {r|r∈R}, each DNN request r={l1,l2,…,l…} N}, where N represents the total number of layers. For a partition point a of a DNN request r, the previous layers {l1,l2,…,l} a Deployed on edge servers s Top, back layer {l a+1 ,l a+2 ,…,l N Deployed on cloud server s c The time complexity of DNN training is mainly concentrated in the forward and backward propagation stages, and is a linear complexity related to the number of network layers and the computational cost per layer. The parameter update process involves simple parameter update operations, so the update time can be ignored. This application focuses on the application examples to study the time complexity of DNN forward and backward propagation.

[0113] Forward propagation refers to starting with the input data and calculating the output value of each layer through the neural network until the final result is obtained. For example, in an edge server... s The first layer l1~l is calculated above a After the layer, the output value OF is transmitted via a fiber optic link. la Transmitted to cloud server s c Calculate the next layer l a+1 ~l Nlayer, obtaining the final model loss. Backpropagation refers to the calculation of the loss function gradient of each parameter, returning from the output layer to the input layer layer by layer to adjust and optimize the parameters. For each data sample, the backward stage (i.e., backward propagation of random gradient obtained by utilizing loss) starts from the last layer of the DNN. That is, the cloud server s c performs subsequent l a+1 ~l N layer backward propagation stage, the intermediate data OF la+1 is sent to the edge server s s performs previous l1~l a layer backward propagation stage. The application examples have the ability to optimize the computational workload and transmission of various tasks by adjusting the sample size of the input data (especially the batch size b).

[0114] Before the DNN task training, the optimal partition point of the DNN needs to be determined. As Figure 7 shown, the terminal first submits a training task to the SDON controller. After receiving the task, the SDON analyzes the task requirements and queries the resource status of the current cloud server and edge server. Each computing node of the DNN is heavily dependent on data from the front-end and back-end nodes. However, there is a gap between different time blocks, which means that if the processing time of different stages differs greatly, it will lead to idle time, affecting the utilization of computing and communication resources. To solve this problem, it is important to balance the time consumed by different stages. In the pipeline workflow model, the training time of the DNN depends on the maximum time t block of all stages.

[0115]

[0116] According to formula (1), t block , where l i represents the i-th layer of the DNN task, N represents the total number of layers of the DNN task, a represents the model partition point of the DNN task, represents the output data size of 1 sample processed by l i layer in the forward propagation stage; represents the output data size of 1 sample processed by l i layer in the backward propagation stage; all of the above belong to task-related parameters; in addition, B1 represents the bandwidth from the terminal device to the edge server; B2 represents the bandwidth from the edge server to the cloud server; represents the forward propagation time of 1 sample processed by l i layer on the server s represents the backward propagation time of 1 sample processed by l i layer on the server s ; all of the above belong to network resource status parameters.

[0117] SDON determines the optimal partition point according to the available resources and task requirements. Then the generated sub-task model is assigned to the appropriate cloud and edge servers. Once the model is successfully deployed, the training begins. Again following the pipeline structure, each minibatch is trained iteratively, alternating between computation and communication. This process continues until the model converges, completing the training. Theoretically, the final training time T Figure 8 all can be calculated according to formula (2), where D is the total training data of the DNN task, and b is the data size of the minibatch.

[0118]

[0119] Therefore, the technical scheme provided by the application example makes up for the shortage of insufficient computing resources of a single server, and simultaneously utilizes the advantage of large bandwidth of an optical network to realize cloud-edge collaborative optical network to jointly accelerate model training. Meanwhile, an asynchronous distributed parallel training mode decoupling computation and communication is designed to reduce the waiting time. Thus, the training efficiency and resource utilization of the model are greatly improved.

[0120] The application example fully utilizes the deterministic delay of optical network transmission, and fully considers the influence of computation and communication on task training. The deterministic delay effectively stabilizes the steps in asynchronous parallel, thereby accelerating the convergence and reducing the risk of training divergence. On this basis, a specific implementation scheme of pipeline parallel combining computation and communication is given. The model training speed is improved, and the resource utilization is greatly increased.

[0121] The application example also provides an electronic device, which can include a processor, a memory, a receiver and a transmitter. The processor is used to execute the pipeline parallel distributed training method for a deep neural network mentioned in the above embodiments. The processor and the memory can be connected through a bus or other means to be connected through the bus. The receiver can be connected with the processor and the memory through wired or wireless means.

[0122] ​The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, or a combination thereof.

[0123] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the pipeline parallel distributed training method for deep neural networks in the embodiments of the present application. The processor executes various functions and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implements the pipeline parallel distributed training method for deep neural networks in the method embodiments.

[0124] The memory can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; and the data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0125] The one or more modules are stored in the memory, and when executed by the processor, implement the pipeline parallel distributed training method for deep neural networks in the embodiments.

[0126] In some embodiments of the present application, the user equipment can include a processor, a memory and a transceiver unit which can include a receiver and a transmitter, the processor, the memory, the receiver and the transmitter can be connected through a bus system, the memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to transceive signals.

[0127] As an implementation manner, the functions of the receiver and the transmitter in the present application can be implemented by a transceiver circuit or a transceiver dedicated chip, and the processor can be implemented by a dedicated processing chip, a processing circuit or a general-purpose chip.

[0128] As another implementation manner, the server provided by the embodiments of the present application can be implemented by using a general-purpose computer. That is, the program codes for implementing the functions of the processor, the receiver and the transmitter are stored in the memory, and the general-purpose processor implements the functions of the processor, the receiver and the transmitter by executing the codes in the memory.

[0129] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps of the aforementioned pipeline parallel distributed training method for a deep neural network.

[0130] The embodiments of the present application further provide a computer program product, which contains a computer program. The computer program is executed by a processor to implement the steps of the aforementioned pipeline parallel distributed training method for a deep neural network.

[0131] Those skilled in the art should understand that the exemplary components, systems and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software or a combination thereof. The actual implementation depends on the specific application and design constraints imposed on the overall system. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are program or code segments used to perform the required tasks. The program or code segments can be stored in a machine readable medium or transmitted through a data signal carried in a carrier wave in a transmission medium or communication link.

[0132] It is to be expressly understood that the application is not limited to the described and illustrated particular configurations and processes. For the sake of clarity, detailed descriptions of known methods are omitted. In the above described embodiments, several specific steps are described and illustrated as examples. However, the method processes of the application are not limited to the specific steps described and illustrated, and the skilled person can make various changes, modifications and additions, or change the order of the steps, after having understood the spirit of the application.

[0133] In this application, features described and / or illustrated with respect to one embodiment can be used in the same or similar manner in one or more other embodiments and / or combined with or substituted for features of other embodiments.

[0134] The above only describes the preferred embodiments of the application, and is not intended to limit the application. The skilled person can make various changes and changes to the embodiments of the application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A pipelined parallel distributed training method for a deep neural network, characterized in that, Comprise: For the current iteration round of the deep neural network, a preset pipeline parallel distributed training step is performed, wherein the pipeline parallel distributed training step comprises: sequentially transmitting each group of mini-batch training data constituting the current batch training data to the edge server through the optical network, so that the edge server and the cloud server connected with the edge server through the optical network communicate, and the cloud server performs asynchronous parallel collaborative training on different sub-task models of the deep neural network based on a preset communication and training decoupling mode, and the edge server sequentially outputs the gradients corresponding to each group of mini-batch training data; sequentially receiving the gradients corresponding to each group of mini-batch training data returned by the edge server through the optical network; The edge server is used to perform the following: sequentially receive each group of mini-batch training data, and when a group of mini-batch training data is first received in the current iteration round, start performing a preset first asynchronous training step to obtain intermediate activation values corresponding to each group of mini-batch training data respectively; and when the intermediate activation values corresponding to each group of mini-batch training data are obtained, the intermediate activation values corresponding to each group of mini-batch training data are transmitted to the cloud server through the optical network; Wherein, the first asynchronous training step comprises: performing forward propagation calculation on the first sub-task model corresponding to the deep neural network based on each group of mini-batch training data respectively, so that the first sub-task model sequentially outputs the intermediate activation values corresponding to each group of mini-batch training data respectively; The cloud server is used to perform the following: sequentially receive the intermediate activation values corresponding to each group of mini-batch training data, and when the intermediate activation values corresponding to a group of mini-batch training data are first received in the current iteration round, start performing a preset second asynchronous training step to obtain intermediate loss values corresponding to each group of mini-batch training data respectively; and when the intermediate loss values corresponding to each group of mini-batch training data are obtained, the intermediate loss values corresponding to each group of mini-batch training data are transmitted to the edge server through the optical network; Wherein, the second asynchronous training step comprises: sequentially performing a preset calculation step on the intermediate activation values corresponding to each group of mini-batch training data, and the calculation step comprises: sequentially performing forward propagation calculation and backward propagation calculation on the second sub-task model corresponding to the deep neural network based on the intermediate activation values corresponding to the current group of mini-batch training data to obtain the intermediate loss values corresponding to the group of mini-batch training data. 2.The method for pipelined parallel distributed training of deep neural networks according to claim 1, wherein, Also include: If the deep neural network is not currently convergent, there is still batch training data not used for model training currently, or the current iteration round is not the last preset iteration round, when the gradient corresponding to a group of mini-batch training data is first received in the current iteration round, perform the pipeline parallel distributed training step for the next iteration round of the deep neural network. 3.The method for pipelined parallel distributed training of deep neural networks of claim 1, wherein, The edge server is also used to perform the following: receive the intermediate loss value corresponding to each group of the small batch training data in sequence, and when the intermediate loss value corresponding to a group of the small batch training data is first received in the current iteration round, start to perform a preset third asynchronous training step to obtain the gradient corresponding to each group of the small batch training data respectively; and when the gradient corresponding to each group of the small batch training data is obtained, return the gradient corresponding to the small batch training data to the optical network; wherein, if the current iteration round is not the first iteration round, the edge server alternately performs the third asynchronous training step and the first asynchronous training step; wherein, the third asynchronous training step comprises: performing back propagation calculation for the first sub-task model based on the intermediate loss value corresponding to each group of the small batch training data respectively, so that the first sub-task model outputs the gradient corresponding to each group of the small batch training data in sequence.

4. The method of claim 1 to 3, wherein, Before performing the preset pipeline parallel distributed training step for the current iteration round of the deep neural network, further comprising: submitting a training task for the deep neural network to an optical network controller, so that the optical network controller obtains a target partition point for the deep neural network according to task requirement data corresponding to the training task and current resource states of the edge server and the cloud server, the optical network controller divides the deep neural network into a first sub-task model and a second sub-task model according to the target partition point, the optical network controller allocates the first sub-task model to the edge server and allocates the second sub-task model to the cloud server, and then returns a corresponding model task partition message; receiving the model task partition message returned by the optical network controller.

5. A pipelined parallel distributed training device for deep neural networks, characterized in that, comprising: a pipeline parallel distributed training module, configured to perform a preset pipeline parallel distributed training step for the current iteration round of the deep neural network, wherein the pipeline parallel distributed training step comprises: transmitting each group of small batch training data constituting the current batch training data to the edge server in sequence through the optical network, so that the edge server and the cloud server which is communicatively connected to the edge server through the optical network perform asynchronous parallel collaborative training on different sub-task models of the deep neural network based on a preset communication and training decoupling mode, and the edge server outputs the gradient corresponding to each group of the small batch training data in sequence; a gradient receiving module, configured to receive the gradient corresponding to each group of the small batch training data returned by the edge server through the optical network in sequence; the edge server is configured to perform the following content: receive each group of the small batch training data in sequence, and when a group of the small batch training data is first received in the current iteration round, start to perform a preset first asynchronous training step to obtain the intermediate activation value corresponding to each group of the small batch training data respectively; and when the intermediate activation value corresponding to each group of the small batch training data is obtained, transmit the intermediate activation value corresponding to the small batch training data to the cloud server through the optical network. The first asynchronous training step comprises: performing forward propagation calculation of the first sub-task model corresponding to the deep neural network based on each group of the small batch training data respectively, so that the first sub-task model outputs the intermediate activation value corresponding to each group of the small batch training data respectively in turn. The cloud server is configured to perform the following: The cloud server is configured to perform the following: The second asynchronous training step comprises: performing the preset calculation step in turn for the intermediate activation value corresponding to each group of the small batch training data, and the calculation step comprises: performing forward propagation calculation and backward propagation calculation of the second sub-task model corresponding to the deep neural network based on the intermediate activation value corresponding to the current group of the small batch training data in turn, so as to obtain the intermediate loss value corresponding to the group of the small batch training data.

6. A pipelined parallel distributed training system for a deep neural network, the system comprising: The second asynchronous training step comprises: performing the preset calculation step in turn for the intermediate activation value corresponding to each group of the small batch training data, and the calculation step comprises: performing forward propagation calculation and backward propagation calculation of the second sub-task model corresponding to the deep neural network based on the intermediate activation value corresponding to the current group of the small batch training data in turn, so as to obtain the intermediate loss value corresponding to the group of the small batch training data. Comprise: The terminal device, the edge server and the cloud server are connected in turn through the optical network; 7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The terminal device is provided with a pipeline parallel distributed training device for the deep neural network, and the pipeline parallel distributed training device for the deep neural network is configured to perform the pipeline parallel distributed training method for the deep neural network according to any one of claims 1 to 4.

8. A computer readable storage medium having stored thereon a computer program, characterized in that, The processor executes the computer program to realize the pipeline parallel distributed training method for the deep neural network according to any one of claims 1 to 4. The computer program is executed by the processor to realize the pipeline parallel distributed training method for the deep neural network according to any one of claims 1 to 4.

Images