A flexible communication method for data-parallel distributed deep learning training acceleration
By pausing synchronous communication during the backpropagation and parameter update phases and performing unfinished parameter synchronization in parallel in the forward propagation phase of the next loop in the distributed deep learning training system, the idle problem of computing and communication operations is solved, significantly improving the Training speed.
Patent Information
- Application Number
- CN202210651078.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-09
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-06-09
AI Technical Summary
The existing data parallel distributed deep learning training system has idle computing operations and communication operations during the training process, resulting in inefficient training, especially the lack of effective parallelism in the forward propagation and backpropagation stages, which affects the training speed.
During the backpropagation and parameter update phase of the distributed training loop, synchronous communication operations are paused, communication status is saved, and unfinished parameter synchronization communication operations in the previous training loop are performed in parallel in the forward propagation phase of the next training loop, which improves the Calculate and communicate parallelism.
By performing activation calculation operations and parameter synchronous communication operations in parallel, the speed of data parallel distributed deep learning training is significantly improved and training efficiency is improved.
Smart Images

Figure CN114970830B_ABST
Abstract
Description
Technical Field
[0001] The present invention mainly relates to the field of distributed deep learning training technology, and in particular to a flexible communication method for accelerating data parallel distributed deep learning training. Background Art
[0002] Deep learning technology has a profound impact on our daily lives and plays an invaluable role in areas such as language recognition, machine translation, and image classification. Deep learning consists of two steps: training and reasoning. The size of the training dataset and the complexity of the deep neural network (DNN) model are two important factors that affect the accuracy of deep learning reasoning results.
[0003] In recent years, with the development of deep learning algorithms, DNN models have become increasingly complex. At the same time, the development of the Internet has made it easy to obtain large-scale data sets for training DNN models. The surge in model parameters and training data has improved the inference accuracy of DNN models. However, this also places higher demands on the computing and memory resources of deep learning training systems. In this context, it can be foreseen that if a single accelerator system is continuously used for training, the future will inevitably face the problem of inefficiency, which will be far from meeting the needs of academia and industry.
[0004] In order to solve the problem of low training efficiency caused by the surge in model parameters and training data, some practitioners have proposed a method of using a distributed deep learning training system to train DNN models. In a distributed training system, training data and DNN models can be stored in multiple single accelerator systems, which then perform training tasks together. This training method not only reduces the storage and computing power requirements of a single accelerator system, but also significantly shortens the time it takes to train a DNN model through their collaborative training.
[0005] Data parallelism is one of the most important methods of distributed training, and its characteristics are as follows:
[0006] First, the randomly selected small batch training data set needs to be evenly divided and each part of the data is allocated to each single accelerator system in the distributed training system.
[0007] Second, each single-accelerator system stores a copy of the complete DNN model and trains it using a local dataset.
[0008] At present, the traditional process of training DNN models using data parallelism is as follows:
[0009] First, in the forward propagation stage, each single-accelerator system computes the local DNN model activations layer by layer from forward to backward.
[0010] Then, in the back-propagation phase, the gradients of the neural network parameters (including weights and biases) of each layer are calculated layer by layer from back to front, and the local DNN model is updated according to the gradients.
[0011] Finally, when the parameters of any n-th layer of the neural network are updated, the parameter update phase begins, that is, the parameters are synchronized in the distributed training system through communication operations, the global parameters are obtained, and the DNN model is updated. After the model is updated, the next training cycle begins.
[0012] However, there are still some technical problems in the training process of the above traditional technologies, which directly affect the training efficiency:
[0013] First: In the forward propagation stage of training, there are only neural network computing operations in the distributed training system, and there are no communication operations between single accelerator systems, which causes the functional modules responsible for communication to be idle.
[0014] Second: During the back propagation and parameter update phase of training, when the parameter calculation of the first layer of the neural network is completed, the synchronous communication operation of the parameters cannot be started immediately. Instead, it is necessary to wait for the communication of the parameters from the last layer of the neural network to the second layer of the neural network to be completed. During this period of time, the functional module responsible for the neural network calculation in the distributed training system is idle.
[0015] In summary, there is an urgent need for methods that can improve the parallelism of computing operations and communication operations in data parallel distributed training, thereby improving the speed of data parallel distributed training. Summary of the invention
[0016] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a flexible communication method for accelerating data parallel distributed deep learning training, which is simple in principle, easy to implement, and has a significant improvement in the speed of data parallel distributed training.
[0017] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0018] A flexible communication method for data parallel distributed deep learning training acceleration, comprising:
[0019] In the back-propagation phase and parameter update phase of the distributed training loop: when the parameter update of the local DNN model of all computing nodes is completed, the ongoing communication operation of synchronizing the DNN model parameters of all nodes is suspended and the communication status is saved;
[0020] After the communication status is saved, the communication operation of the first layer neural network parameters of all nodes is synchronized, the first layer neural network is updated, and then the next training cycle begins;
[0021] In the forward propagation phase of the next training cycle, while starting to calculate the activation of the first layer of the neural network, the synchronous communication operations of the neural network parameters from the second layer to the last layer that were not completed in the previous training cycle are started in sequence to update each layer of the neural network;
[0022] After the activation calculation of the nth layer of the neural network is completed and the n+1th layer of the neural network is updated, the activation calculation of the n+1th layer of the neural network begins; where 1≤n<the number of DNN model layers.
[0023] As a further improvement of the method of the present invention: the distributed training loop includes a forward propagation stage, a back propagation stage and a parameter update stage; the forward propagation stage is used to complete the calculation operation of activating the local DNN model of the single accelerator system; in the back propagation stage, it is used to calculate and update the parameters of the local DNN model; the parameter update stage is used to synchronize the local parameters of all single accelerator systems through communication operations and update the DNN model.
[0024] As a further improvement of the method of the present invention: the parameters include weights and biases.
[0025] As a further improvement of the method of the present invention: in the back propagation stage and the parameter updating stage, the parameters of the DNN model are calculated sequentially from the last layer to the first layer. When the model parameters of any layer are calculated, the synchronous communication operation of the model parameters of this layer is started after waiting for the ongoing synchronous communication operation in the system to be completed.
[0026] As a further improvement of the method of the present invention: in the forward propagation stage, before calculating the activation of any layer of the neural network, the synchronous communication operation of the neural network parameters of this layer in the previous training cycle has been completed, that is, the model of this layer has been updated.
[0027] As a further improvement of the method of the present invention: in the back propagation stage and the parameter updating stage, when the parameter calculation of the first layer of the neural network is completed, the ongoing synchronous communication operation of the DNN model parameters in the system is suspended and the communication status is saved.
[0028] As a further improvement of the method of the present invention: when the communication is suspended, the synchronous communication operation of the first layer neural network parameters is started and completed, and the first layer neural network is updated.
[0029] As a further improvement of the method of the present invention: after the first layer of the neural network is updated, the next training cycle is started; in the forward propagation stage, the activation of the first layer of the neural network is calculated; at the same time, the synchronous communication operation of the second layer of the neural network parameters that was not completed in the previous training cycle is started, and if any, the second layer of the neural network is updated.
[0030] As a further improvement of the method of the present invention: after the synchronous communication operation of the parameters of the second layer of neural network is completed, the synchronous communication operation of the parameters of the third layer of neural network that was not completed in the previous training cycle is started, and the third layer of neural network is updated; and so on, the synchronous communication operation of all model parameters that were not completed in the previous training cycle is completed, and each layer of neural network is updated.
[0031] As a further improvement of the method of the present invention: when the activation calculation of the first layer of the neural network is completed and the second layer of the neural network is updated, the activation calculation of the second layer of the neural network begins; and so on, the activation calculation of each layer of the neural network is completed.
[0032] Compared with the prior art, the advantages of the present invention are:
[0033] The flexible communication method for accelerating data-parallel distributed deep learning training of the present invention is simple in principle, easy to implement, and can significantly improve the speed of data-parallel distributed training. The present invention greatly improves the speed of data-parallel distributed deep learning training by executing the activated computing operation and the synchronous communication operation of the parameters in the previous training cycle in parallel. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 It is a schematic diagram of the training process of the forward propagation stage of the first training cycle in a specific application example of the present invention.
[0035] Figure 2 It is a schematic diagram of the training process of the forward propagation stage from the second training cycle to the last training cycle in a specific application example of the present invention.
[0036] Figure 3 It is a schematic diagram of the training process from the first training cycle to the penultimate training cycle back propagation and parameter synchronization stage in a specific application example of the present invention.
[0037] Figure 4 It is a schematic diagram of the training process of the last training cycle back propagation and parameter synchronization stage in a specific application example of the present invention.
[0038] Figure 5 It is a schematic diagram of the process of the method of the present invention in a specific application. DETAILED DESCRIPTION
[0039] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0040] like Figure 5 As shown, a flexible communication method for data parallel distributed deep learning training acceleration of the present invention includes:
[0041] In the back-propagation phase and parameter update phase of the distributed training loop: when the parameters (including weights and biases) of the local DNN models of all computing nodes are updated, the ongoing communication operation of synchronizing the DNN model parameters of all nodes is suspended and the communication status is saved;
[0042] After the communication status is saved, the communication operation of synchronizing the parameters of the first layer of neural network of all nodes is started and completed, the first layer of neural network is updated, and then the next training cycle is started;
[0043] In the forward propagation phase of the next training cycle, while starting to calculate the activation of the first layer of the neural network, the synchronous communication operations of the neural network parameters from the second layer to the last layer that were not completed in the previous training cycle are started in sequence to update each layer of the neural network;
[0044] After the activation calculation of the nth (1≤n<number of DNN model layers) layer of neural network is completed and the n+1th layer of neural network is updated, the calculation of the activation of the n+1th layer of neural network begins.
[0045] In a specific application example, a distributed training loop is formed in a distributed training system composed of multiple single accelerator systems, which performs data parallel distributed training in three stages, namely: forward propagation stage, back propagation stage and parameter update stage, and the parameters include weights and biases. The forward propagation stage is used to complete the calculation operation of activating the local DNN model of the single accelerator system; the back propagation stage is used to calculate and update the parameters of the local DNN model; the parameter update stage is used to synchronize the local parameters of all single accelerator systems through communication operations and update the DNN model.
[0046] In a specific application example, in the back propagation phase and parameter update phase of the distributed training loop, the parameters of the DNN model are calculated sequentially from the last layer to the first layer. When the model parameters of any layer are calculated, the synchronous communication operation of the model parameters of this layer is started after waiting for the ongoing synchronous communication operation in the system to be completed.
[0047] In a specific application example, in the forward propagation phase of a distributed training loop, before calculating the activation of any layer of the neural network, the synchronous communication operation of the neural network parameters of this layer in the previous training loop has been completed, that is, the model of this layer has been updated.
[0048] In a specific application example, in the back propagation phase and parameter update phase of the distributed training loop, when the parameter calculation of the first layer of the neural network is completed, the ongoing synchronous communication operation of the DNN model parameters in the system is suspended and the communication status is saved.
[0049] In a specific application example, when the communication is suspended, the synchronization communication operation of the first layer neural network parameters is started and completed, and the first layer neural network is updated.
[0050] In a specific application example, after the first layer of the neural network is updated, the next training cycle begins. In the forward propagation phase, the activation of the first layer of the neural network begins to be calculated; at the same time, the synchronization communication operation of the second layer of the neural network parameters that was not completed in the previous training cycle (if any) is started, and the second layer of the neural network is updated.
[0051] In a specific application example, when the synchronous communication operation of the parameters of the second layer of neural network is completed, the synchronous communication operation of the parameters of the third layer of neural network that was not completed in the previous training cycle (if any) is started, and the third layer of neural network is updated. Similarly, the synchronous communication operation of all model parameters that were not completed in the previous training cycle (if any) is completed, and each layer of neural network is updated.
[0052] In a specific application example, when the activation calculation of the first layer of the neural network is completed and the second layer of the neural network is updated, the activation calculation of the second layer of the neural network begins. And so on, the activation calculation of each layer of the neural network is completed.
[0053] In a specific application example, the detailed process of the present invention is as follows:
[0054] like Figure 1 As shown, in the specific application example, the training process of the forward propagation stage of the first training cycle is: activation is calculated layer by layer from the first layer of the neural network to the last layer of the neural network, and there is no synchronous communication operation during the training process.
[0055] like Figure 2 As shown, in a specific application example, the training process from the second training cycle to the last training cycle forward propagation stage is:
[0056] For communication operations: First, start the synchronous communication operation of the unfinished parameters of the second layer of neural network in the previous training cycle, and update the second layer of neural network. Then start the synchronous communication operation of the parameters of the third layer of neural network, and update the third layer of neural network. And so on, complete the synchronous communication operation of all unfinished parameters in the previous training cycle, and update each layer of neural network.
[0057] For calculation operations: First, calculate the activation of the first layer of the neural network. Then, after the activation calculation is completed and the second layer of the neural network is updated, start calculating the activation of the second layer of the neural network. Then, calculate the activation in this way, that is, after the activation calculation of the nth (1<n<number of DNN model layers) layer of the neural network is completed and the n+1th layer of the neural network is updated, start calculating the activation of the n+1th layer of the neural network, and finally complete the calculation of the activation of each layer of the DNN model.
[0058] like Figure 3 As shown in the figure, in a specific application example, the training process from the first training cycle to the penultimate training cycle back propagation and parameter synchronization stage is:
[0059] For calculation operations: the local DNN model parameters are calculated layer by layer from the last layer of the neural network to the first layer of the neural network.
[0060] For communication operations: when the parameters of the local nth (1<n≤number of DNN model layers) layer neural network are calculated, and the ongoing synchronous communication operation in the current system is completed, start the synchronous communication operation of the nth layer neural network parameters; when the parameters of the local 1st layer neural network are calculated, pause the ongoing communication operation in the system and save the communication status; after the communication status is saved, start and complete the synchronous communication operation of the 1st layer neural network parameters, and update the 1st layer neural network.
[0061] like Figure 4 As shown in the specific application example, the training process of the last training cycle back propagation and parameter synchronization stage is:
[0062] For calculation operations: the local DNN model parameters are calculated layer by layer from the last layer of the neural network to the first layer of the neural network.
[0063] For communication operations: After the parameter calculation of the local nth (1≤n≤number of DNN model layers) layer neural network is completed and the ongoing synchronous communication operation in the current system is completed, the synchronous communication operation of the nth layer neural network parameters is started, and finally the update of the DNN model parameters is completed.
[0064] like Figure 5 As shown in the figure, in the forward propagation stage, the communication of the neural network parameters from the second layer to the last layer that were not completed in the previous training cycle is carried out in sequence. In the backward propagation and parameter synchronization stage, when the parameters of any layer of the neural network of the local DNN model are updated, the synchronous communication operation of the parameters of this layer can be started; when the parameters of the first layer of the neural network are updated, the communication is suspended and the communication status is saved; finally, the synchronous communication operation of the parameters of the first layer is carried out.
[0065] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.
Claims
1. A flexible communication method for data parallel distributed deep learning training acceleration, characterized in that: include: In the back-propagation phase and parameter update phase of the distributed training loop: when the parameter update of the local DNN model of all computing nodes is completed, the ongoing communication operation of synchronizing the DNN model parameters of all nodes is suspended and the communication status is saved; After the communication status is saved, the communication operation of the first layer neural network parameters of all nodes is synchronized, the first layer neural network is updated, and then the next training cycle begins; In the forward propagation phase of the next training cycle, while starting to calculate the activation of the first layer of the neural network, the synchronization communication operation of the neural network parameters from the second layer to the last layer that were not completed in the previous training cycle is started in sequence to update each layer of the neural network; After the activation calculation of the nth layer of the neural network is completed and the n+1th layer of the neural network is updated, the activation calculation of the n+1th layer of the neural network begins; where 1≤n<the number of DNN model layers.
2. The flexible communication method for data parallel distributed deep learning training acceleration according to claim 1, characterized in that: The distributed training cycle includes a forward propagation phase, a back-propagation phase and a parameter update phase; the forward propagation phase is used to complete the calculation operation of activating the local DNN model of the single accelerator system; the back-propagation phase is used to calculate and update the parameters of the local DNN model; the parameter update phase is used to synchronize the local parameters of all single accelerator systems through communication operations and update the DNN model.
3. The flexible communication method for data parallel distributed deep learning training acceleration according to claim 2, characterized in that: The parameters include weights and biases.
4. The flexible communication method for data parallel distributed deep learning training acceleration according to any one of claims 1 to 3, characterized in that: In the back propagation phase and the parameter updating phase, the parameters of the DNN model are calculated sequentially from the last layer to the first layer. When the model parameters of any layer are calculated, the synchronous communication operation of the model parameters of this layer is started after waiting for the ongoing synchronous communication operation in the system to be completed.
5. The flexible communication method for data parallel distributed deep learning training acceleration according to any one of claims 1 to 3, characterized in that: In the forward propagation stage, before calculating the activation of any layer of the neural network, the synchronous communication operation of the neural network parameters of this layer in the previous training cycle has been completed, that is, the model of this layer has been updated.
6. The flexible communication method for data parallel distributed deep learning training acceleration according to any one of claims 1 to 3, characterized in that: In the back propagation phase and the parameter updating phase, when the parameter calculation of the first layer of the neural network is completed, the ongoing synchronous communication operation of the DNN model parameters in the system is suspended and the communication state is saved.
7. The flexible communication method for data parallel distributed deep learning training acceleration according to claim 6, characterized in that: When the communication is suspended, the synchronization communication operation of the first layer neural network parameters is started and completed, and the first layer neural network is updated.
8. The flexible communication method for data parallel distributed deep learning training acceleration according to claim 7, characterized in that: After the first layer of the neural network is updated, the next training cycle begins; In the forward propagation phase, the activation of the first layer of the neural network is calculated; at the same time, the synchronization communication operation of the second layer of the neural network parameters that was not completed in the previous training cycle is started, and if any, the second layer of the neural network is updated.
9. The flexible communication method for data parallel distributed deep learning training acceleration according to claim 8, characterized in that: When the synchronous communication operation of the parameters of the second layer of neural network is completed, the synchronous communication operation of the parameters of the third layer of neural network that was not completed in the previous training cycle is started, and the third layer of neural network is updated; and so on, the synchronous communication operation of all model parameters that were not completed in the previous training cycle is completed, and each layer of neural network is updated.
10. The flexible communication method for data parallel distributed deep learning training acceleration according to claim 9, characterized in that: When the activation calculation of the first layer of the neural network is completed and the second layer of the neural network is updated, the activation calculation of the second layer of the neural network begins; and so on, the activation calculation of each layer of the neural network is completed.
