A memory scheduling method for pipeline parallel training

Abstract

The present invention discloses a memory scheduling method for pipeline parallel training, including: deployment phase: segmenting a neural network model with network layers as units to form different network partitions; deploying network partitions on different computing nodes; training phase: each computing node is only responsible for the calculation of the deployed network partition; each computing node communicates data with upstream and downstream computing nodes; data transfer mechanism: unloading activation data in the computing node to the CPU memory, waiting for the data to be retrieved when it needs to be used again; data processing phase: compressing the activation data and then sending it to the downstream computing node or CPU memory; after the downstream computing node receives the activation data, or after the original computing node retrieves the activation data from the CPU memory, the computing node decompresses the activation data once. The present invention has the advantages of simple principle, easy implementation, and can reduce storage load and communication overhead.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of neural network application, and in particular to a memory scheduling method for pipeline parallel training. Background Art

[0002] At present, deep learning technology is developing rapidly. It has been widely used in many fields and has also become a hot research field in academia. Among deep learning technologies, the deep neural network model has received the most attention. Deep neural networks have shown good results in many artificial intelligence applications, including computer vision, natural language processing, machine translation, image recognition and other fields. However, the trend of neural network models developing towards deeper and deeper has led to a sharp increase in storage load and computing power, especially for neural network training.

[0003] The increase in the size of neural networks makes training more difficult, and training methods based on a single hardware device are becoming increasingly unsustainable. In order to solve the problem of difficulty in training neural network models, some researchers have proposed the concept of "distributed parallel training". The so-called "distributed parallel solution" is to deploy multiple general-purpose accelerators to form a distributed parallel system, which combines the physical memory and core computing power of multiple accelerators to achieve the purpose of training large-scale network models and accelerating the training process.

[0004] GPU is one of the representatives of neural network accelerators and is currently the most widely used neural network accelerator. It is an open source neural network accelerator platform launched by NVIDIA for deep learning reasoning processes. Most of the existing distributed parallel systems are deployed on multi-GPU platforms.

[0005] Pipeline parallelism is a type of distributed parallelism solution. It has a significant speed advantage over other solutions in terms of training speed, but it still has two technical problems:

[0006] First, the storage load is too large. Generally speaking, if a parallel system has several pipelines, the corresponding memory and bandwidth requirements will increase several times because it needs to save multiple versions of data and program states. In the pipeline parallel system, the computing node that stores the most data needs to store the same amount of data as the number of pipeline stages, resulting in excessive storage load during the pipeline parallel training process.

[0007] Second, the communication overhead is too high. When training a pipeline parallel system, adjacent computing nodes need to frequently exchange activations and gradients, which leads to huge communication overhead of the pipeline and makes it impossible for the pipeline to achieve the ideal speedup ratio corresponding to the number of pipeline stages.

[0008] In summary, there is an urgent need for methods that can handle the problem of excessive storage and communication loads when pipelines perform training tasks in parallel. Summary of the invention

[0009] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a memory scheduling method for pipeline parallel training which is simple in principle, easy to implement, and can reduce storage load and communication overhead.

[0010] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0011] A memory scheduling method for pipeline parallel training, comprising:

[0012] Deployment phase: Split the neural network model into network layers to form different network partitions; deploy different network partitions to different computing nodes;

[0013] Training phase: Each computing node is only responsible for computing the deployed network partition; each computing node communicates data with upstream and downstream computing nodes; receives and sends activation data during forward propagation, and receives and sends gradient data during backpropagation;

[0014] Data transfer mechanism: offloads the activation data in the compute node to the CPU memory and waits to retrieve the data when it needs to be used again;

[0015] Data processing stage: The activation data is compressed and then sent to the downstream computing node or CPU memory. After the downstream computing node receives the activation data, or after the original computing node retrieves the activation data from the CPU memory, the computing node decompresses the activation data.

[0016] As a further improvement of the present invention: in the training phase, in the forward propagation phase, activation data from an upstream computing node is received, forward calculation is completed, and then the activation data thereof is sent to a downstream computing node.

[0017] As a further improvement of the present invention: in the training phase, in the back propagation phase, the gradient data from the downstream computing node is received, the back calculation is completed, and then the own gradient data is sent to the upstream computing node.

[0018] As a further improvement of the present invention: in the training phase, each computing node continuously switches the computing channels of forward propagation and backward propagation, so that small batch training data can be continuously input; the training processes of different small batches are mixed together, and each computing node saves multiple versions of activation data and weight parameters.

[0019] As a further improvement of the present invention: in the training phase, there is a time interval between the forward calculation and the reverse calculation of a small batch. Before the reverse calculation of the small batch comes, the activation data it generates and the weight parameters it uses are not used during this time interval.

[0020] As a further improvement of the present invention: in the training phase, after the computing node completes a small batch of forward calculations, the generated activation data is sent to the downstream computing node; the activation data is then unloaded to other storage devices; before the reverse calculation of this small batch comes, the activation data is retrieved from the external storage device for reverse calculation.

[0021] As a further improvement of the present invention: in the data processing stage, a zero-value compression algorithm is implemented by configuring a decompression and compression module.

[0022] As a further improvement of the present invention: in the training phase, the activation data communication operation of the computing node during forward propagation includes:

[0023] The activation data generated by the computing node is compressed by the decompression and compression module, and then the compressed data is sent to the downstream computing node;

[0024] After the downstream computing node receives the compressed activation data, the decompression and compression unit decompresses the data and then uses it for forward calculation.

[0025] As a further improvement of the present invention: in the training phase, the operation of the computing node unloading activation data includes:

[0026] The internal computing core of the computing node compresses the generated activation data through a decompression and compression unit;

[0027] Offload compressed activation data to CPU memory.

[0028] As a further improvement of the present invention: in the training phase, the operation of the computing node retrieving activation data includes:

[0029] The compute node retrieves the compressed activation data from the CPU memory;

[0030] The decompression and compression unit decompresses the data;

[0031] The decompressed activation data is used for the back calculation.

[0032] Compared with the prior art, the advantages of the present invention are:

[0033] The memory scheduling method for pipeline parallel training of the present invention is a pipeline parallel solution deployed on a multi-GPU platform, which transfers part of the activation data in the pipeline GPU node to the CPU memory, and waits for the activation data to be retrieved when it is needed again, thereby reducing the storage load of the pipeline. At the same time, the memory scheduling method of the present invention sets a decompression and compression unit in the GPU node, compresses the activation data before transmitting, and decompresses it after receiving, which can reduce the communication overhead of transmitting the activation data. The final pipeline parallel solution can greatly reduce the storage load during training, and control the training speed loss within a reasonable range. The present invention has the advantages of simple principle, easy implementation, and reducing the storage load and communication overhead of the pipeline parallel solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the topological structure principle of the present invention after specific application.

[0035] Figure 2 It is a schematic diagram of the principle of the pipeline parallel solution in a specific application example of the present invention.

[0036] Figure 3 It is a schematic diagram of the principle of realizing data transfer by pipeline parallel scheme in a specific application example of the present invention.

[0037] Figure 4 It is a schematic diagram of the principle of the decompression and compression unit in a specific application example of the present invention.

[0038] Figure 5 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] like Figure 1 and Figure 5 As shown, the memory scheduling method for pipeline parallel training of the present invention includes:

[0041] Deployment phase: Split the neural network model into network layers to form different network partitions; deploy different network partitions to different computing nodes;

[0042] Training phase: Each computing node is only responsible for computing the deployed network partition. In a specific application example, in the forward propagation phase, it receives activation data from the upstream computing node, completes the forward calculation, and then sends its own activation data to the downstream computing node. In the reverse propagation phase, it receives gradient data from the downstream computing node, completes the reverse calculation, and then sends its own gradient data to the upstream computing node.

[0043] Data transfer mechanism: During the training process of the pipeline parallel scheme, multiple versions of activation data need to be saved for calculating gradients during back propagation, which is the main reason for high storage load. The memory scheduling method of the present invention unloads the activation data in the computing node to the CPU memory and waits for the data to be retrieved when it needs to be used again, avoiding data accumulation in the computing node.

[0044] Data processing stage: activation data unloading and retrieval operations, and activation data sending and receiving operations between nodes all have a lot of communication overhead, which affects the training speed. The memory scheduling method of the present invention sets a compression unit in the computing node to compress the activation data and then send it to the downstream computing node or CPU memory. After the downstream computing node receives the activation data, or the original computing node retrieves the activation data from the CPU memory, the decompression and compression unit inside the node decompresses the activation data once.

[0045] In a specific application example, the training process in the training phase uses a special tandem scheduling method to streamline the training process. During the training process, each computing node continuously switches the forward propagation and backward propagation computing channels to continuously input small batches of training data. The training processes of different small batches are mixed together, requiring each computing node to save multiple versions of activation data and weight parameters.

[0046] In a specific application example, in the data processing stage, a zero-value compression algorithm is implemented by configuring a decompression and compression module.

[0047] After the deployment phase is completed, the network model has been segmented, and the network partitions have been deployed on different computing nodes. In a specific application example, the memory scheduling process in the training phase includes:

[0048] Step S1: The data set is input from the first computing node. During forward computing, the computing node first receives the compressed activation data from the upstream computing node, then the decompression and compression unit decompresses the data, and finally sends it to the computing core for forward computing.

[0049] Step S2: After the computing node completes the forward calculation, the decompression and compression unit compresses the activation data generated by itself and then sends it to the downstream computing node and CPU memory.

[0050] Step S3: Before the reverse calculation of the computing node comes, the computing node first retrieves the compressed activation data from the CPU memory and also receives the gradient data from the downstream node. Then the decompression and compression unit decompresses the activation data and finally sends it to the computing core for reverse calculation.

[0051] Step S4: After the computing node completes the reverse calculation, it sends the gradient data to the upstream computing node.

[0052] like Figure 2 As shown in a specific application example, the model parameters of network partition, that is, weight parameters, are Figure 2 The W block shown on the right. The subscript number 1 of the W block is the network partition ID, and the subscript number 2 is the version ID of the weight parameter. Since the training of multiple small batches of the pipeline parallel scheme is mixed together, it is necessary to save multiple versions of the weight parameters.

[0053] like Figure 2 As shown, in a specific application example, the activation data is as follows Figure 2 The A block on the right. The number 1 in the subscript of the A block is the ID of the network partition to which the activation data belongs, and the number 2 in the table below is the version ID of the activation data. Since the storage load is too high during pipeline parallel training, the recalculation strategy is used by default. Under the recalculation strategy, each network partition only retains the activation data of the upper and lower boundary layers, and the activation data of other layers are recalculated at the time of back propagation. Therefore, one version of the activation data contains the output of the upper and lower boundary layers.

[0054] like Figure 2 As shown, in a specific application example, the forward calculation channel is as follows Figure 2 The blank squares in the figure are the IDs of the small batches during training. The reverse calculation channel is as follows: Figure 2 The grid squares in the figure are the IDs of the small batches during training.

[0055] like Figure 2 As shown in the figure, in a specific application example, the computing node uses a scheduling method in which the computing channels are switched one after another. When starting training, the computing node needs a preparation phase. After completing the reverse calculation of the first small batch, it will enter the scheduling state of one after another. This scheduling method allows the data set to be input continuously in small batches.

[0056] In a specific application example, weights need to be updated during the training phase, and the weight update strategy is: each time a computing node completes a reverse calculation, the weight is updated immediately. The specific process includes:

[0057] Step S10: The data set is continuously input in small batches. The forward phase of the computing node needs to receive the activation data of the upstream computing node and then send its own activation data to the downstream computing node.

[0058] Step S20: After the computing node completes the forward calculation of a small batch, it is necessary to retain the weight parameters used in this small batch and the activation data generated.

[0059] Step S30: After the computing node completes the forward calculation of a small batch, it immediately switches to the reverse calculation of another small batch. In the reverse calculation stage, the computing node receives the gradient data from the downstream, then selects the previously saved activation data for reverse calculation, and updates the previously saved weight parameters. Finally, it sends its own gradient data to the upstream computing node.

[0060] Step S40: The computing node reversely calculates the updated weight parameters for the forward calculation of the next small batch. In a specific application, for example, small batch 5 uses the weights updated by small batch 1; small batch 6 uses the weights updated by small batch 2.

[0061] like Figure 3 As shown, in a specific application example, the data transfer process of the data transfer mechanism of the present invention includes:

[0062] Step S100: After the computing node completes the forward computation of a small batch, it sends the activation data of the lower boundary layer to the downstream computing node, and then unloads the full version of the activation data to the CPU memory;

[0063] Step S200: When the back propagation of this small batch comes, the computing node retrieves the activation data from the CPU to calculate the gradient, and then selects the corresponding version of the weight parameter for update.

[0064] like Figure 4 As shown, it is a schematic diagram of the decompression and compression unit in the present invention in a specific computing example, using the decompression and compression unit provided by the memory controller in the GPU. In the specific computing example, all the above communication operations require the following steps:

[0065] Step S1000: After the GPU core (stream processor) calculates the activation data or gradient data, it is compressed by the compression unit of the GPU memory controller;

[0066] Step S2000: Send the compressed data to the downstream GPU node or CPU memory. The compressed data is sent to other GPU nodes via the PCIe channel and sent to the CPU memory via the PCIe channel and the CPU memory controller.

[0067] Step S3000: The GPU node receives activation data from the upstream GPU node in the forward calculation phase; receives gradient data from the downstream GPU node and activation data from the CPU memory in the reverse calculation phase. After receiving the data, the decompression and compression unit of the GPU node memory controller decompresses the received activation data and finally sends it to the GPU core for calculation.

[0068] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A memory scheduling method for pipeline parallel training, characterized in that: include: Deployment phase: The neural network model is divided into units of network layers to form different network partitions; Deploy different network partitions to different computing nodes; Training phase: Each computing node is only responsible for computing the deployed network partition; each computing node communicates data with upstream and downstream computing nodes; Activation data is received and sent during the forward propagation process, and gradient data is received and sent during the back-propagation process; Data transfer mechanism: offloads the activation data in the compute node to the CPU memory and waits to retrieve the data when it needs to be used again; Data processing stage: compress the activation data and then send it to the downstream computing node or CPU memory; After the downstream computing node receives the activation data, or after the original computing node retrieves the activation data from the CPU memory, the computing node decompresses the activation data once.

2. The memory scheduling method for pipeline parallel training according to claim 1, characterized in that: In the training phase, in the forward propagation phase, activation data from an upstream computing node is received, forward calculation is completed, and then the activation data of the node itself is sent to a downstream computing node.

3. The memory scheduling method for pipeline parallel training according to claim 1, characterized in that: In the training phase, in the back propagation phase, the gradient data from the downstream computing node is received, the reverse calculation is completed, and then the own gradient data is sent to the upstream computing node.

4. The memory scheduling method for pipeline parallel training according to claim 1, 2 or 3, characterized in that: During the training phase, each computing node continuously switches the computing channels of forward propagation and back propagation, so that small batches of training data are continuously input; the training processes of different small batches are mixed together, and each computing node saves multiple versions of activation data and weight parameters.

5. The memory scheduling method for pipeline parallel training according to claim 4, characterized in that: In the training phase, there is a time interval between the forward calculation and the backward calculation of a small batch. Before the backward calculation of a small batch comes, the activation data it generates and the weight parameters it uses are not used during this time interval.

6. The memory scheduling method for pipeline parallel training according to claim 1, 2 or 3, characterized in that: In the training phase, after the computing node completes a small batch of forward calculations, it sends the generated activation data to the downstream computing node; then it unloads the activation data to other storage devices; before the reverse calculation of this small batch comes, it retrieves the activation data from the external storage device for reverse calculation.

7. The memory scheduling method for pipeline parallel training according to claim 1, 2 or 3, characterized in that: In the data processing stage, a zero-value compression algorithm is implemented by configuring a decompression and compression module.

8. The memory scheduling method for pipeline parallel training according to claim 7, characterized in that: In the training phase, the activation data communication operation of the computing node during forward propagation includes: The activation data generated by the computing node is compressed by the decompression and compression module, and then the compressed data is sent to the downstream computing node; After the downstream computing node receives the compressed activation data, the decompression and compression unit decompresses the data and then uses it for forward calculation.

9. The memory scheduling method for pipeline parallel training according to claim 7, characterized in that: In the training phase, the operation of the computing node unloading activation data includes: The internal computing core of the computing node compresses the generated activation data through a decompression and compression unit; Offloads compressed activation data to CPU memory.

10. The memory scheduling method for pipeline parallel training according to claim 7, characterized in that: In the training phase, the operation of the computing node retrieving activation data includes: The compute node retrieves the compressed activation data from the CPU memory; The decompression and compression unit decompresses the data; The decompressed activation data is used for the backward computation.

Images