A data exchange method and system for improving deep learning training performance

Abstract

The application discloses a data exchange method and system for improving deep learning training performance, and comprises the following steps: S1: data loading, numbering all training samples in shared storage, and generating a random training sequence according to the current training round; the node comprises a GPU, a CPU, local storage and a storage engine; the training samples of the first round are loaded into the local storage of the corresponding node according to the training sequence; S2: judging whether the current round is the last round, if yes, executing S3; if not, executing S4; S3: the GPU of each node uses the training samples in the local storage to perform deep learning training of the current round according to the training sequence; S4: the GPU performs training of the current round according to the training sequence, while the CPU exchanges the training samples that have been trained in the current round with other nodes, and after completion, the step of S2 is repeated. In the training process, the training samples are exchanged to the node to which the next round belongs, so that cross-node access during training is avoided.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a data exchange method and system for improving deep learning training performance. Background Art

[0002] Deep learning training uses the stochastic gradient descent algorithm and generally requires several rounds. Each round traverses the entire dataset samples according to a pre-generated random sequence. The traversal is divided into multiple steps, and each step traverses a batch of samples. In distributed training, multiple compute nodes are used. For each batch of samples in each step, they are evenly distributed to each compute node. The samples assigned to each compute node are called local batches. Each compute node calculates the characteristics of the local batch samples according to the training model and updates the parameters. They then synchronize with each other to complete a training step. The compute nodes share a storage system based on a parallel file system, called shared storage. For the assigned local batch samples, each compute node reads the sample data from the shared storage for training.

[0003] Currently, deep learning training requires increasingly large datasets, and the number of compute nodes can scale to hundreds or even thousands. Increasing the number of nodes helps reduce computational time, but due to the limited bandwidth of shared storage, reading data from shared storage becomes a bottleneck that hinders training performance once the number of nodes reaches a certain level. To address this bottleneck, compute nodes are equipped with local high-speed storage, such as SSDs, which acts as a local cache. During training, data is first loaded from shared storage into the local cache, and subsequent training rounds read data from the local cache. However, because a single node cache cannot accommodate the entire dataset, the dataset is fragmented and stored in the caches of various nodes, forming a distributed cache system. Each node cache stores a portion of the overall dataset. During training, if the required data is not in the local node cache, it is retrieved from other nodes. However, cross-node access incurs higher latency and overhead than local access, making GPU reuse impossible.

[0004] The current solution to cross-node overhead is to reorder the random sequence based on the contents of the local cache to maximize the use of existing data in the local cache for training. This targeted reordering improves the local cache hit rate and reduces cross-node accesses, but it also affects the accuracy of the training results due to the change in the randomness of the training sequence. Another approach is to use batch access to obtain multiple samples to be used at once across nodes. This method can reduce the number of cross-node accesses, but it cannot completely avoid cross-node accesses during training. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention proposes a data exchange method and system for improving the performance of deep learning training. During each round of GPU training, the CPU is used to exchange the trained training samples to the node to which they belong in the next round, so that all sample data can be obtained locally in each round of training.

[0006] The specific technical solutions are as follows:

[0007] A data exchange method for improving deep learning training performance includes the following steps:

[0008] S1: Data loading: All training samples stored in the shared storage are numbered, and a random training sequence for the current round of deep learning training is generated according to the round number of the current round of deep learning training. For the first round of training, the training samples are loaded into the local storage of the corresponding node according to the first round of training sequence. The node includes: GPU, CPU, local storage, and storage engine.

[0009] S2: Determine whether the current round is the last round. If so, execute S3; if not, execute S4;

[0010] S3: Each node's GPU uses the training samples stored locally to perform the current round of deep learning training according to the training sequence.

[0011] S4: The GPU of each node uses the training samples on the local storage to perform the current round of deep learning training according to the training sequence. At the same time, the CPU exchanges the training samples that have been trained in this round with other nodes through the background thread. After a round of training is completed and the data exchange is completed, S2 is repeated.

[0012] Furthermore, the data exchange in S4 is carried out between nodes in pairs. The training samples that have been trained in this round are exchanged in the next round according to the nodes to which the training sequence belongs. A node can exchange data with any other node. After the two nodes establish a network connection, they exchange data with each other as requesting nodes and sending nodes. The requesting node negotiates with the sending node on the acceptable amount of data to be sent at that time based on the free space in the local storage. The amount of data sent by the sending node cannot exceed the acceptable amount of the requesting node.

[0013] Furthermore, the storage engine manages the training samples, deletes the exchanged training samples that have been trained in this round from the local storage of the node, stores the exchanged training samples in the local storage of the node, and can index and access them according to the sample sequence number.

[0014] Furthermore, in S3 and S4, during the GPU training process, the training framework sends the required sample serial number to the storage engine according to the training sequence and requests the corresponding data; the storage engine of each node uses the sample number to index and access the training samples loaded or exchanged to the local storage of the node, and returns the required training samples.

[0015] Furthermore, in S3 and S4, the GPU performs the current round of deep learning training, specifically: performing vector calculations on the training sample data using a given training model.

[0016] Furthermore, in S1, the training samples are numbered according to their traversal order.

[0017] A system for improving deep learning training performance, used to implement the data exchange method for improving deep learning training performance, characterized by comprising: a data loading module, a training module, and a data exchange module;

[0018] The data loading module is used to load the training samples from the shared storage into the local storage of the corresponding node according to the first round of training sequence;

[0019] The training module is used to perform the deep learning training in S3 or S4 on the training samples in each node according to the training sequence;

[0020] The data exchange module is used to perform the data exchange in S4 in each round of training except the last round.

[0021] Furthermore, before each round of training begins, the data required by each node has been saved to local storage through loading or swapping operations.

[0022] The beneficial effects of the present invention are:

[0023] (1) The method and system conceived by the present invention can exchange the trained training samples to the nodes belonging to the next round during a round of training.

[0024] (2) The method and system conceived by the present invention can obtain all required samples from local storage when using distributed cache for deep learning training, avoiding cross-node access during training.

[0025] (3) The method and system conceived by the present invention eliminate the consumption of the network during GPU training, and at the same time utilize CPU resources and network resources to exchange samples during GPU training, thereby maximizing the throughput of the entire system training. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1This is a flow chart of a data exchange method for improving deep learning training performance in the present invention.

[0027] Figure 2 Schematic diagram of loading a data set from shared storage to a local SSD cache of a computing node in an embodiment of the present invention.

[0028] Figure 3 Schematic diagram of parallel GPU training and CPU data exchange in an embodiment of the present invention.

[0029] Figure 4 Schematic diagram of data exchange between node A, node B, node C, and node D in an embodiment of the present invention.

[0030] Figure 5 It is a schematic diagram of a protocol for agreeing on the exchange volume when two nodes exchange data in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0032] In the following description, reference is made to “one specific embodiment”, which describes a subset of all possible embodiments, but it is understood that “one specific embodiment” describes the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0033] Unless otherwise defined, all technical terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0034] Before further describing the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are subject to the following explanations.

[0035] (1) CPU, short for Central Processing Unit, is the computing and control core of a computer system and the final execution unit for information processing and program execution. In this invention, it is used to read sample data, pre-process the samples, and then send them to the GPU for training.

[0036] (2) GPU, short for Graphic Process Unit, refers to a component that performs dedicated computational acceleration and is used to train samples in the present invention.

[0037] (3) Local storage, used to save sample data on the computing end so that sample data can be quickly read during training.

[0038] (4) Storage engine, used to manage sample data stored in local storage and index samples according to sample serial numbers.

[0039] (5) Training framework refers to common systems used for deep learning training, such as Pytorch, Tensorflow, etc.

[0040] Based on the above architecture, an embodiment of the present invention proposes a data exchange method for improving deep learning training performance. During each round of GPU training, the CPU is used to exchange the trained data to the node to which it belongs in the next round, so that all sample data can be obtained locally during each round of training.

[0041] like Figure 1 As shown, a data exchange method for improving deep learning training performance includes the following steps:

[0042] S1: Data loading. Figure 2 As shown in the figure, for a dataset stored in shared storage, all training samples in the dataset are numbered; this number serves as the sample sequence. Assuming a total of n rounds of deep learning training, a random training sequence is generated for each round, i = 1, 2, 3, ... n. The node to which each training sample belongs in a given round is determined by the training sequence for that round. For the first round, the training samples are loaded into the local SSD storage of the corresponding node according to the first round's training sequence, forming a distributed cache. After loading, each node contains a portion of the dataset.

[0043] Each node consists of a GPU, CPU, local storage, and a storage engine. The GPU is used for deep learning training, which is performed in batches. The CPU is used to retrieve data from local storage. The local storage is used to store training samples. The storage engine manages sample data (inserting or deleting) using sample numbers as indexes.

[0044] S2: Determine whether the current round (i.e., round i) is the last round of training. If it is the last round of training, execute S3; if not, execute S4.

[0045] S3: Each node's GPU uses the training samples on local storage to perform deep learning training for the current round (i.e., round i) according to the training sequence. That is, each GPU performs vector calculations on the training sample data using a given training model. During training, the training framework sends the required sample serial number to the storage engine based on the training sequence and requests the corresponding data. The storage engine of each node uses the sample serial number to index and access the training samples loaded or swapped to the node's local storage, and returns the required training samples. Since all the training samples required by the node are already stored in local storage at the beginning of the current round of training, the storage engine can directly obtain the data and return it to the training framework.

[0046] S4: Each node's GPU uses the training samples stored locally to perform the current round (i.e., round i) of deep learning training according to the training sequence. Specifically, each GPU performs vector calculations on the training sample data using the given training model. During the current round (i.e., round i) of GPU training, the CPU simultaneously exchanges data from the trained training samples of this round with other nodes through a background thread. After a round of training is completed and the data exchange is complete, the next round of training begins, repeating S2.

[0047] The training samples that have been trained in the current round are recorded as invalid samples, and the training samples that have not been trained are valid samples. Invalid samples will no longer be used in this round and can be used for data exchange. Data exchange should ensure that before the next round of training begins, all training samples required by the GPU of each node have been saved to the local storage of the corresponding node according to the training sequence, so that all samples required for the next round of training can be obtained from the local storage and the waiting time for the GPU to obtain data can be reduced. Data exchange is carried out simultaneously with the process of GPU deep learning training, such as Figure 3 As shown in the figure, the CPU and network are used to exchange data during the deep learning training process to maximize the utilization of system resources.

[0048] The data exchange process performed by the CPU is carried out between nodes in pairs. The failed samples are exchanged according to the node to which the training sequence belongs in the next round (i.e., the i+1th round). A node can exchange data with any other node. After the two nodes establish a network connection, they exchange data with each other as requesting nodes and sending nodes. The requesting node negotiates with the sending node on the acceptable amount of data to be sent at that time (i.e., the maximum exchange amount) based on the free space in the local storage. The amount of data sent by the sending node must not exceed the acceptable amount of the requesting node to ensure that there is enough space to store the data sent to the requesting node, and the frequency of training sample exchange can be optimized based on the available space in the local storage of the requesting node. The storage engine manages the training samples, deletes the failed samples that are exchanged out from the local storage of the node; stores the exchanged training samples in the local storage of the node, and can index and access them according to the sample sequence number.

[0049] To implement the above method, a system for improving deep learning training performance is constructed, comprising: a data loading module, a training module, and a data exchange module. The data loading module is used to load training samples from shared storage into the local storage of the corresponding node according to the first round of training sequence; the training module is used to perform deep learning training in S3 of the above method on the training samples in each node according to the training sequence; and the data exchange module is used to perform data exchange in S4 of the above method before each round of training after the first round.

[0050] In summary, during the data loading phase, the present invention loads training samples from shared storage into the local storage of the corresponding node according to the first round of training sequence. Afterwards, training rounds begin, with each node directly obtaining training samples from local storage. Before each round of training begins, a determination is made as to whether that round is the last. If not, samples are swapped during training so that all training samples required for the next round can be obtained from local storage. If it is the last round, no swapping is required, and the entire training process is complete upon completion of that round.

[0051] The difference between the present invention and cross-node prefetching is that the data exchange method used in the present invention exchanges with the corresponding node according to the node to which the training sample belongs in the next round during GPU training, and the data of this round of training can be obtained from local storage. Cross-node prefetching is to obtain multiple sample data required by the current node in batches from another node when the sample required by the current node is in another node during this round of training. Compared with cross-node prefetching, the data exchange method described in the present invention enables the data required by the training node to be obtained from the local node during the training process without the need for cross-node operations.

[0052] The following describes a data exchange method for improving deep learning training performance in detail through an embodiment of the present invention.

[0053] Example 1

[0054] S1: If Figure 2 As shown in the figure, for a data set stored in a shared storage, all training samples in the data set are numbered according to their traversal order; it is assumed that two rounds of deep learning training are required, and a random training sequence for the round of deep learning training is generated according to the round number of the i-th round, i = 1, 2. Figure 2 The dataset stored in shared storage contains 16 training samples and 4 nodes. For the first round, the 16 training samples are evenly distributed among the 4 nodes to form a distributed cache. The node's local storage is determined based on the first round training sequence. After loading, each node caches a portion of the original dataset. In this embodiment, node A performs deep learning training on four samples with sample numbers 3, 14, 10, and 12 according to the training sequence in the first round of training, and therefore these four samples are loaded into node A; node B performs deep learning training on four samples with sample numbers 11, 7, 16, and 8 according to the training sequence in the first round of training, and therefore these four samples are loaded into node B; node C performs deep learning training on four samples with sample numbers 4, 2, 1, and 15 according to the training sequence in the first round of training, and therefore these four samples are loaded into node C; node D performs deep learning training on four samples with sample numbers 9, 5, 6, and 13 according to the training sequence in the first round of training, and therefore these four samples are loaded into node D.

[0055] S2: Determine whether the current round (i.e., round i) is the last round of training. If it is the last round of training, execute step S3; if not, execute step S4.

[0056] S3: Each node's GPU uses the training samples stored locally to perform the current round of deep learning training according to the training sequence. Specifically, each GPU performs vector calculations on the training sample data using a given training model. Each node's storage engine uses the sample sequence number to index and access the training samples loaded into the node's local storage.

[0057] S4: The GPU of each node uses the training samples on the local storage to perform the current round (i.e., the i-th round) of deep learning training according to the training sequence. At the same time, the CPU exchanges data with other nodes for the training samples that have been trained in this round (i.e., failed samples) through the background thread. After one round of training is completed and the data exchange is completed, it is ready to start the next round of training, that is, repeat S2. When the first round of training is completed, all samples will be exchanged to the nodes to which they belong for the second round of training through data exchange. Since two rounds of training are set in this embodiment, the second round of training is the last round. Then, S2 and S3 will be executed during the second round, that is, only deep learning training will be performed, and no data exchange operation will be performed. After the second round of training is completed, the entire training process will also end.

[0058] like Figure 4 As shown, each grid represents a training sample. The training samples with white background are those that have not been trained in this round, the samples with dark background are invalid samples, and the samples with grid background are invalid samples of other nodes that are waiting to exchange data with node A. In the current training moment of the first round of training, there are 3 invalid samples in node A, namely samples 3, 14 and 10. These 3 invalid samples belong to node D, node B and node D respectively in the next round of training. Therefore, they need to be exchanged to the corresponding nodes before the start of the second round of training; in the current training moment of the first round of training, there are 3 invalid samples in node B, namely samples 11, 7 and 16. These 3 invalid samples belong to node A, node C and node D respectively in the next round of training. Therefore, they need to be exchanged to the corresponding nodes before the start of the second round of training. Point; In node C, there are 3 failed samples at the current training moment of the first round of training, namely samples 4, 2 and 1. These 3 failed samples belong to node B, node A and node C respectively in the next round of training, so they need to be exchanged to the corresponding nodes before the start of the second round of training; In node D, there are 3 failed samples at the current training moment of the first round of training, namely samples 9, 5 and 6. These 3 failed samples belong to node A, node A and node B respectively in the next round of training, so they need to be exchanged to the corresponding nodes before the start of the second round of training. Taking node A as an example, when a network connection is established between node A and other nodes for data exchange, while the failed samples required by the other node on node A are exchanged out, the training samples on other nodes that will belong to node A in the next round are also exchanged in; that is, since sample No. 11 on node B, sample No. 2 on node C, and samples No. 9 and No. 15 on node D belong to node A in the second round of training, they can be exchanged to node A for storage in the first round of training. At the same time, samples No. 3, No. 14, and No. 10 on node A that have completed training can be exchanged to the nodes to which they belong in the second round.

[0059] like Figure 5As shown, when two nodes that have established a network connection are exchanging data, the requesting node sends a request to the sending node based on the local free space and limits the acceptable amount. The amount of data sent by the sending node will be limited within the acceptable amount to ensure that there is enough space to store the data sent to the requesting node. During the first round of training, according to the second round of training sequence, it can be seen that node A needs training samples No. 2, No. 11, No. 9, and No. 5 in the second round of training sequence. Node A, as the requesting node, sends requests to node C, node B, and node D respectively. In this embodiment, when node A sends a request to node B, the acceptable amount is 10MB. Node B, as the sending node, sends the corresponding training samples. Node B has a total of 2MB of samples that can be exchanged with node A. If the data amount is within 10MB, all these samples will be sent. The same applies to the remaining nodes when they act as requesting nodes.

[0060] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0061] The foregoing description is merely a preferred embodiment of the present invention, which is provided to enable those skilled in the art to understand and implement the present invention and is not intended to limit the present invention. Various modifications and variations of the present invention are readily apparent to those skilled in the art, and modifications to these embodiments will be readily apparent to those skilled in the art. The general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Any modifications, equivalent substitutions, improvements, and the like made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A data exchange method for improving deep learning training performance, characterized in that: The following steps are involved: S1: Data loading: All training samples stored in the shared storage are numbered, and a random training sequence for the current round of deep learning training is generated according to the round number of the current round of deep learning training; For the first round of training, the training samples are loaded into the local storage of the corresponding node according to the first round of training sequence; the node includes: GPU, CPU, local storage, and storage engine; S2: Determine whether the current round is the last round. If so, execute S3; if not, execute S4; S3: Each node's GPU uses the training samples stored locally to perform the current round of deep learning training according to the training sequence. S4: The GPU of each node uses the training samples stored locally to perform the current round of deep learning training according to the training sequence. At the same time, the CPU uses a background thread to exchange data from the trained training samples of this round with other nodes. After a round of training is completed and the data exchange is complete, S2 is repeated. The data exchange in S4 is carried out between nodes in pairs. The training samples that have been trained in this round are exchanged with the nodes to which the training sequence belongs in the next round. A node can exchange data with any other node. After the two nodes establish a network connection, they exchange data with each other as requesting nodes and sending nodes. The requesting node negotiates with the sending node on the acceptable amount of data to be sent at that time based on the free space in the local storage. The amount of data sent by the sending node cannot exceed the acceptable amount of the requesting node.

2. The data exchange method for improving deep learning training performance according to claim 1, characterized in that: The storage engine manages the training samples, deletes the exchanged training samples that have been trained in this round from the local storage of the node, stores the exchanged training samples in the local storage of the node, and can index and access them according to the sample sequence number.

3. The data exchange method for improving deep learning training performance according to claim 1, characterized in that: In S3 and S4, during the GPU training process, the training framework sends the required sample serial number to the storage engine according to the training sequence and requests the corresponding data; the storage engine of each node uses the sample serial number to index and access the training samples loaded or exchanged to the local storage of the node, and returns the required training samples.

4. The data exchange method for improving deep learning training performance according to claim 1, characterized in that: In S3 and S4, the GPU performs the current round of deep learning training, specifically: performing vector calculations on the training sample data using a given training model.

5. The data exchange method for improving deep learning training performance according to claim 1, characterized in that: In S1, the training samples are numbered according to their traversal order.

6. A system for improving deep learning training performance, used to implement the data exchange method for improving deep learning training performance according to any one of claims 1 to 5, characterized in that: include: Data loading module, training module, data exchange module; The data loading module is used to load the training samples from the shared storage into the local storage of the corresponding node according to the first round of training sequence; The training module is used to perform the deep learning training in S3 or S4 on the training samples in each node according to the training sequence; The data exchange module is used to perform the data exchange in S4 in each round of training except the last round.

7. The system for improving deep learning training performance according to claim 6, characterized in that: Before each round of training begins, the data required by each node has been saved to local storage through loading or exchange operations.

Images