A method and device for storing distributed deep learning cache data
By dynamically migrating data in real-time and updating random sequence lists in distributed cache systems, the problems of insufficient concurrency in data ingestion and unbalanced load in deep learning training are solved, and data ingestion performance and resource utilization are improved.
Patent Information
- Application Number
- CN202411092481.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-09
AI Technical Summary
The existing distributed cache systems have problems such as insufficient data ingestion concurrency and unbalanced load in deep learning training, resulting in limited data ingestion performance.
Using an efficient data layout method, we can dynamically migrate data in a distributed cache system, allocate load balancing on the cache server, and use cache resources to improve the concurrency of data ingestion. Specific steps include initializing the deep learning training system, dynamically selecting the cache service process, exchanging data asynchronously, and updating the list of random sequences in each cycle to optimize the data layout.
The balancing distribution of the load of the cache service node is realized, the data intake concurrency and performance of the deep learning training task is improved, and the resources of the computing node are fully utilized.
Smart Images

Figure CN118642661B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to data storage technology, and more particularly to a method and device for distributed deep learning cache data storage. Background Art
[0002] Deep learning training is divided into two stages: data ingestion and model training. As computing power continues to grow rapidly, the speed of model training has increased significantly, but the speed of data ingestion has lagged behind and gradually become a performance bottleneck for deep learning training. Using a cache system to reduce the amount of data the model ingests from the backend storage system can speed up data ingestion.
[0003] The existing distributed cache system uses storage resources such as disks, SSDs or memory on each computing node to build a cache space, and evenly divides and maps the training data set to the local cache of each computing node according to the number of computing nodes. During model training, each rank training process selects a computing node to ingest data according to the data segmentation and mapping strategy. If the local cache of the computing node hits, the data is ingested from the cache. Otherwise, the data is ingested from the backend storage and stored in the local cache of the computing node.
[0004] This cache data layout scheme faces the challenge of insufficient data ingestion concurrency. Specifically, the deep learning training task randomly accesses the data set in each cycle, and each rank training process ingests data from the local cache of each computing node in parallel according to the cut random sequence fragments. Although this method of uniformly cutting the random sequence can ensure that the amount of data that all rank training processes need to ingest in each cycle is equivalent, and the number of cache data requests received by the cache service process of each computing node in each cycle is also the same, it often happens that multiple rank training processes ingest data from the local cache of the same computing node in a short period of time, and the cache services of other computing nodes are idle at this time. This load imbalance of the computing node cache service reduces the concurrency of data ingestion, thereby affecting the performance of data ingestion. A common method to improve the concurrency of data ingestion is to start multiple data loading processes for each rank training process. These training processes ingest data from the local cache of each computing node in parallel. However, each cycle data set is randomly accessed. Simply increasing the number of data loading processes only increases the number of processes that ingest data from the local cache of the same computing node within a certain period of time. The load on the cache server is still unbalanced. In this case, the local cache resources on the computing node are not fully utilized, and the performance of the distributed cache system is therefore limited. Efficient data layout methods can ensure load balancing on the cache server and improve the utilization of computing node resources. When the number of data loading processes started by the training process increases, the concurrency of the training process ingesting data from the cache system will increase accordingly, and the performance of data ingestion will also be significantly improved. Summary of the invention
[0005] In order to solve the above-mentioned shortcomings of the prior art, the present invention provides a method and device for distributed deep learning cache data storage.
[0006] The present invention utilizes efficient data layout methods and devices to dynamically migrate data in real time in a distributed cache system, ensures load balancing of the cache server at a fine-grained level, and fully utilizes the cache resources of the cache server, thereby facilitating the improvement of the concurrency of distributed deep learning training cache data ingestion, thereby improving the data ingestion performance of deep learning training tasks.
[0007] The first aspect of the present invention relates to a method for storing distributed deep learning cache data, which adopts the following technical solution:
[0008] Step 1: Initialize the deep learning training system, including preparing data in the backend storage system, initializing the distributed cache system, starting the cache service process, initializing the cache area and swap area of the cache space to empty, starting the training process, and generating a random sequence list;
[0009] Step 2: The training task of the current cycle starts, and all training processes execute the model training task in parallel, where each training process dynamically selects a cache service process to ingest data; when the training process ingests data, the cache service process dynamically exchanges data in an asynchronous manner to implement data layout for the next cycle of data ingestion;
[0010] Step 3: Synchronize the data ingestion progress of all training processes. The training process that completes the data ingestion of the current cycle first will be in a blocked state. When the last training process completes the data ingestion, the data ingestion process of the current cycle is completed. On the one hand, the data exchange of the cache service process in the current cycle is completed, and each cache service process updates the local cache space. On the other hand, the blocked training processes are awakened at the same time, and the random sequence list is updated. All training processes enter the next cycle at the same time;
[0011] Step 4: When the training process of all cycles is completed, the training task is completed, the model is output, the cache space is released, and the cache service process is destroyed;
[0012] Step 5: Input data into the trained model to obtain the output of the model;
[0013] Furthermore, the step 1 specifically includes the following sub-steps:
[0014] S1.1, the training data set is stored in the backend storage system, and the number of samples contained in the data set is recorded as ,During the training task execution, the dataset will be evenly cut and mapped to the local cache of the computing nodes;
[0015] S1.2, the cache space of the distributed cache system consists of the local storage of the computing nodes. The computing nodes that provide cache space are cache service nodes, and their number is recorded as , each computing node provides the same cache capacity, and a cache service process is started on each computing node, denoted as ,This process manages the local cache space of the computing node;
[0016] S1.3, the local cache space of each computing node is divided into a cache area and a swap area. The cache area is used to store the sample data to be used in the current cycle, and the swap area is used to receive data from each cache service process and these data will be used in the next cycle. The capacity of the cache area and the swap area is the same, and the maximum number of samples that can be stored in the cache area and the swap area is recorded as , and calculated by formula (1) ;When the training task starts, the cache area and swap area on each computing node are empty;
[0017]
[0018] S1.4, when the training task is initialized, all rank training processes are started, and the number of rank training processes is recorded as ,Each rank training process corresponds to a cache client. In addition, in order to improve the concurrency of data access, a rank training process can start multiple data loading processes for data ingestion;
[0019] S1.5, calculate and generate a random sequence list for the current cycle, denoted as , calculate and generate the random sequence list of the next cycle, recorded as ;
[0020] Furthermore, the step 2 comprises the following sub-steps:
[0021] S2.1, Training Process Random sequence list according to the current period Ingest data, and The sample data selected is Index in Formula (2) must be satisfied;
[0022]
[0023] Among them, start is the starting index value, end is the length of the random sequence list, and step is the step size;
[0024] S2.2, all training processes concurrently ingest data to train the process For example, the sample When , we first select the cache service process, that is, we perform a modulus operation according to formula (3) to calculate the cache service process number k, so as to determine the target cache service process as , then the training process To the cache service process Request Sample ;
[0025]
[0026] S2.3, cache service process After receiving the request, Retrieve samples from a local cache ,like If the cache hits, the cache service process The sample Remove from the buffer and return the data to the training process ;like If the cache does not hit, the cache service process Read samples from the backend storage , and then return the data to the training process ;
[0027] S2.4, when the sample After being returned to the training process, the cache service process Execute the sample asynchronously Data migration, first, calculate the sample The cache service process to which it belongs in the next cycle ,Sure The method is to list the random sequence in the next cycle Search ,Sure exist Index in , and perform modulus operation according to formula (4) to calculate The number m of the cache service process should be placed in the next cycle;
[0028]
[0029] Then, cache the service process The sample Push to cache service process , and by The sample Save in The exchange area completes the current cycle sample Layout;
[0030] S2.5, for the training process ,like , then the training process The next sample will be ingested, starting with the index of that sample , then the training process Go to S2.2 to ingest data , otherwise the training process Complete the data ingestion of the current cycle;
[0031] Furthermore, the step three includes the following sub-steps:
[0032] S3.1, there are differences in the progress of concurrent training processes ingesting data. The training process that completes data ingestion first will be blocked until the last training process completes data ingestion;
[0033] S3.2, when the last training process completes data ingestion, on the one hand, each cache service process updates the local cache space, that is, the cache service process moves all the data in its swap area to its cache area; on the other hand, it uses a random sequence list Replacement , then calculate and generate a new random sequence list and assign it to ;
[0034] S3.3, the blocked training processes are awakened at the same time, and all training processes enter the next cycle at the same time, starting a new round of training;
[0035] Furthermore, the cache space managed by the cache service process is divided into a cache area and a swap area, wherein the cache area stores sample data that will be ingested by the training process in the current cycle. When the current cycle ends, the sample data in the cache area will be consumed and become empty; the swap area successively receives sample data from each cache service process in the current cycle. When the current cycle is about to end, the swap area will be full of sample data.
[0036] Furthermore, the cache service process uses a hash list to organize the cache area and the swap area, thereby accelerating the efficiency of sample data retrieval in the cache space.
[0037] Furthermore, the present invention establishes an additional index for the random sequence list of the next cycle to accelerate the speed of searching the random sequence list.
[0038] Furthermore, the cache service nodes are interconnected using RDMA technology, so that the cache service process can use a fast network link to exchange sample data, and the data migration between cache service processes is performed asynchronously, which does not hinder the training process from ingesting data from the cache area.
[0039] A second aspect of the present invention relates to a device for distributed deep learning cache data storage, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a method for distributed deep learning cache data storage of the present invention.
[0040] The third aspect of the present invention relates to a computer-readable storage medium, characterized in that a program is stored thereon, and when the program is executed by a processor, a distributed deep learning cache data storage method of the present invention is implemented.
[0041] The beneficial effect of the present invention is that the present invention realizes the reasonable distribution of data on the cache service nodes through the data layout strategy in each cycle, so that the data ingestion load on these cache service nodes can be kept balanced in any time period, so that the storage and network resources on each cache node can be fully utilized, which is conducive to improving the concurrency of deep learning training task data ingestion, thereby speeding up the speed of data ingestion. In order to support the data layout strategy, the local storage space of each cache node is divided into a cache area and a swap area, wherein the cache area is used to store the data to be used by the cache service process in the current cycle, and the swap area is used to receive the data to be used by the cache service process in the next cycle. During the execution of the training task, each cache service process provides the data in the cache area to each cache client while receiving data from the cache service process in real time and saving it to the swap area. At the same time, each cache client dynamically selects the cache service process in real time and forwards the data ingestion request, and the cache service processes push data to each other in an asynchronous manner, thereby achieving load balancing of the cache service process. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is an architecture diagram of the deep learning training task of the present invention for parallel data ingestion.
[0043] Figure 2 It is a flowchart of concurrent data ingestion for deep learning training of the present invention.
[0044] Figure 3 It is a schematic diagram of exchanging and saving data between cache service processes of the present invention.
[0045] Figure 4 It is a schematic diagram of a method for a cache client to select a cache service process and a process for the cache service process to exchange data according to the present invention.
[0046] Figure 5 It is a schematic diagram of the flow of data in the local cache space of the first cycle cache service process of the present invention. DETAILED DESCRIPTION
[0047] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.
[0048] Embodiment 1
[0049] This embodiment relates to a method for distributed deep learning cache data layout. Figure 1 The architecture for ingesting data for deep learning training tasks is described. The backend storage in the figure is supported by mainstream distributed file systems such as Lustre, GPFS, etc., which store complete data sets and provide storage services to each computing node through mounting and other methods. Several training processes are started on the computing node. Each training process is regarded as a cache client, which selects the cache service process through the selector and sends data ingestion requests to it through the network card. A cache service process is started on each computing node that provides cache services. The cache service process manages the local storage space of the computing node. The local storage space is provided by the memory and is divided into a cache area and a swap area according to its function.
[0050] Figure 2 The basic process of ingesting data and training models using the method of improving the concurrency of deep learning training data ingestion of the present invention is described. A method of distributed deep learning cache data layout in this example includes the following steps:
[0051] 1. Initialize the deep learning training system, including preparing data in the backend storage system, initializing the distributed cache system, starting the cache service process, initializing the cache area and swap area of the cache space to empty, starting the training process, and generating a random sequence list;
[0052] 1.1. The backend storage of this example uses the GPFS file system, which stores the image dataset. The number of samples contained in the dataset is recorded as , for example, the number of pictures in this example is 6;
[0053] 1.2. The computing node is the client of GPFS, on which the GPFS directory is mounted. The computing node that provides cache resources is also called the cache service node. The cache space of the distributed cache system consists of the local storage of the cache service node. The local storage device in this example uses an SSD device. The number of cache service nodes is recorded as ,The number of cache service nodes in this example is 2;
[0054] 1.3. Each cache service node starts a cache service process, denoted as , which manages the cache space of this node; during model training, the data set will be evenly cut and mapped to each cache service node, and each cache service node provides the same cache capacity; the cache space of the cache service node is divided into a cache area and a swap area. The cache area stores the sample data to be used in the current cycle, and the swap area receives data from each cache service process and these data will be used in the next cycle. The cache area and swap area capacity of each cache service node are the same, and the maximum number of samples that can be stored in the cache area and swap area is recorded as , and calculated by formula (1) Therefore, in this example The value of is 3, that is, the capacity of the cache area and the swap area are both 3; when the training task starts, the cache area and the swap area on each cache service node are empty;
[0055]
[0056] 1.4. Initialize the model training task: The training model of this example uses ResNet50, and the training device uses the GPU on the computing node. Each GPU device corresponds to a rank training process, and the number of rank training processes is recorded as In this example, there are two rank training processes; each rank training process corresponds to a cache client. In order to improve the concurrency of data access, a rank training process can start multiple data loading processes or threads for data ingestion;
[0057] 1.5. Calculate and generate a random sequence list for the current period and a list of random sequences for the next cycle , for example in this example The value is <4, 2, 1, 3, 0, 5>, The value is <3, 4, 1, 2, 5, 0>;
[0058] 2. The training task of the current cycle starts, and all training processes execute the model training task in parallel. Each training process dynamically selects the cache service process to ingest data. When the training process ingests data, the cache service process dynamically exchanges data in an asynchronous manner to implement data layout for the next cycle of data ingestion. Figure 4 Describes the main steps of the rank training process ingesting data and the cache service process migrating data;
[0059] 2.1 Training Process Random sequence list according to the current period Ingest data, and The sample data selected is Index in The formula (2) must be satisfied:
[0060]
[0061] Among them, start is the starting index value, end is the length of the random sequence list, and step is the step size; for example, in this example ,then The data to be ingested in the current cycle is {4, 1, 0}, The data to be ingested in the current cycle is {2, 3, 5};
[0062] 2.2. All training processes concurrently ingest data to train the process For example, the sample When , we first select the cache service process, that is, we perform a modulus operation according to formula (3) to calculate the cache service process number k, so as to determine the target cache service process as , then the training process To the cache service process Request Sample ;For example The first sample ingested in the current cycle is "4", and accordingly, , then the cache service process number can be calculated , then the training process From the cache service process Sample "4" is taken, and at the same time, The cache service process Ingest sample "2";
[0063]
[0064] 2.3. Cache service process After receiving the request, Retrieve samples from a local cache ,like If the cache hits, the cache service process The sample Remove from the buffer and return the data to the training process ;like If the cache does not hit, the cache service process Read samples from the backend storage , and then return the data to the training process For example, in this example Retrieve from local cache , that is, sample "4". According to step 1.3, the cache area on the cache service node is empty when the training task starts, so sample "4" does not hit the cache area. Read sample "4" from the GPFS mount directory and return it to the training process ;
[0065] 2.4. When the sample After being returned to the training process, the cache service process Execute the sample asynchronously Data migration, first, calculate the sample The cache service process to which it belongs in the next cycle ,Sure The method is to list the random sequence in the next cycle Search ,Sure exist Index in , and perform modulus operation according to formula 4 to calculate In the next cycle, the cache service process number m should be placed; then, the cache service process The sample Push to cache service process , and by The sample Save in The exchange area completes the current cycle sample Layout; Figure 3 Describes a schematic diagram of cache service processes pushing data to each other, Figure 5 Describes the data movement of each cache service process in the current cycle cache space in this example; for example, the cache service process in this example Will , that is, sample "4", is sent to the training process Then, the sample "4" is searched in the random sequence list <3, 4, 1, 2,5, 0> to obtain its index , and then calculate the target cache service process number according to formula (4) , so the cache service process Push sample "4" to The exchange area;
[0066]
[0067] 2.5. For the training process ,like , then the training process The next sample will be ingested, starting with the index of that sample , then the training process Go to step 2.2 to ingest data , otherwise the training process Complete the data ingestion of the current cycle; for example, in this example After taking sample "4", because 0+2<6 is satisfied, Go to step 2.2 to take sample "1";
[0068] 3. Synchronize the data ingestion progress of all training processes. The training process that completes the data ingestion of the current cycle first will be in a blocked state. When the last training process completes the data ingestion, the data ingestion process of the current cycle is completed. On the one hand, the data exchange of the cache service process in the current cycle is completed, and each cache service process updates the local cache space. On the other hand, the blocked training processes are awakened at the same time, and the random sequence list is updated. All training processes enter the next cycle at the same time;
[0069] 3.1. There are differences in the progress of concurrent training processes ingesting data. The training process that completes data ingestion first will be blocked until the last training process completes data ingestion. For example, in this example, assume Before Complete data ingestion, so when After taking the last sample "0", Complete data ingestion in the current cycle, then train the process blocked;
[0070] 3.2. When the last training process completes data ingestion, each cache service process updates the local cache space, that is, the cache service process moves all the data in its swap area to its cache area, and then uses the random sequence list Replacement , then calculate and generate a new random sequence list and assign it to For example, in this example When data ingestion is complete, both training processes have completed data ingestion, and the cache service process and The data migration between the two is then completed. and The cache is empty, and The exchange area contains the sample data {1, 3, 5}, The exchange area contains the sample data {4,0, 2}, and Migrate the sample data in your exchange area to your own cache area, and then <4, 2, 1, 3,0, 5> is replaced by <3, 4, 1, 2, 5, 0>, and generate a new random sequence list and assign it to ;
[0071] 3.3. The blocked training processes are awakened at the same time, and all training processes enter the next cycle at the same time and start a new round of training; for example, in this example When data ingestion is complete, After being awakened, the two training processes enter the next cycle together and start a new round of training;
[0072] 4. When the training process of all cycles is completed, the training task is completed, the model is output, the cache space is released, and the cache service process is destroyed; for example, when the training of all cycles in this example is completed, the entire training task is completed and the model is output, and the cache service process is destroyed. and Release their cache area and swap area respectively, and then destroy the two cache service processes;
[0073] 5. The user inputs image data into the model, and the model outputs image classification results.
[0074] Embodiment 2
[0075] This embodiment relates to a method for distributed deep learning cache data layout, including the following steps:
[0076] 1. Initialize the deep learning training system, including preparing data in the backend storage system, initializing the distributed cache system, starting the cache service process, initializing the cache area and swap area of the cache space to empty, starting the training process, and generating a random sequence list;
[0077] 1.1. The backend storage of this example uses the GPFS file system, which stores the image dataset. The number of samples contained in the dataset is recorded as , for example, the number of pictures in this example is 6;
[0078] 1.2. Compute nodes are GPFS clients, on which GPFS directories are mounted. Compute nodes that provide cache resources are also called cache service nodes. The cache space of the distributed cache system consists of the local storage of the cache service nodes. In this example, the local storage device uses DRAM devices. The number of cache service nodes is recorded as ,The number of cache service nodes in this example is 2;
[0079] 1.3. Each cache service node starts a cache service process, denoted as , which manages the cache space of this node; during model training, the data set will be evenly cut and mapped to each cache service node, and each cache service node provides the same cache capacity; the cache space of the cache service node is divided into a cache area and a swap area. The cache area stores the sample data to be used in the current cycle, and the swap area receives data from each cache service process and these data will be used in the next cycle. The cache area and swap area capacity of each cache service node are the same, and the maximum number of samples that can be stored in the cache area and swap area is recorded as , and calculated by formula (1) Therefore, in this example The value of is 3, that is, the capacity of the cache area and the swap area are both 3; when the training task starts, the cache area and the swap area on each cache service node are empty;
[0080]
[0081] 1.4. Initialize the model training task: The training model of this example uses ResNet50, and the training device uses the GPU on the computing node. Each GPU device corresponds to a rank training process, and the number of rank training processes is recorded as In this example, there are two rank training processes; each rank training process corresponds to a cache client. In order to improve the concurrency of data access, a rank training process can start multiple data loading processes or threads for data ingestion;
[0082] 1.5. Calculate and generate a random sequence list for the current period and a list of random sequences for the next cycle , for example in this example The value is <4, 2, 1, 3, 0, 5>, The value is <3, 4, 1, 2, 5, 0>;
[0083] 2. The training task of the current cycle starts, and all training processes execute the model training task in parallel. Each training process dynamically selects the cache service process to ingest data. When the training process ingests data, the cache service process dynamically exchanges data in an asynchronous manner to implement data layout for the next cycle of data ingestion. Figure 4 Describes the main steps of the rank training process ingesting data and the cache service process migrating data;
[0084] 2.1 Training Process Random sequence list according to the current period Ingest data, and The sample data selected is Index in The formula (2) must be satisfied:
[0085]
[0086] Among them, start is the starting index value, end is the length of the random sequence list, and step is the step size; for example, in this example ,then The data to be ingested in the current cycle is {4, 1, 0}, The data to be ingested in the current cycle is {2, 3, 5};
[0087] 2.2. All training processes concurrently ingest data to train the process For example, the sample When , we first select the cache service process, that is, we perform a modulus operation according to formula (3) to calculate the cache service process number k, so as to determine the target cache service process as , then the training process To the cache service process Request Sample ;For example The first sample ingested in the current cycle is "4", and accordingly, , then the cache service process number can be calculated , then the training process From the cache service process Sample "4" is taken, and at the same time, The cache service process Ingest sample "2";
[0088]
[0089] 2.3. Cache service process After receiving the request, Retrieve samples from a local cache ,like If the cache hits, the cache service process The sample Remove from the buffer and return the data to the training process ;like If the cache does not hit, the cache service process Read samples from the backend storage , and then return the data to the training process For example, in this example Retrieve from local cache , that is, sample "4". According to step 1.3, the cache area on the cache service node is empty when the training task starts, so sample "4" does not hit the cache area. Read sample "4" from the GPFS mount directory and return it to the training process ;
[0090] 2.4. When the sample After being returned to the training process, the cache service process Execute the sample asynchronously Data migration, first, calculate the sample The cache service process to which it belongs in the next cycle ,Sure The method is to list the random sequence in the next cycle Search ,Sure exist Index in , and perform modulus operation according to formula 4 to calculate In the next cycle, the cache service process number m should be placed; then, the cache service process The sample Push to cache service process , and by The sample Save in The exchange area completes the current cycle sample Layout; Figure 3 Describes a schematic diagram of cache service processes pushing data to each other, Figure 5 Describes the data movement of each cache service process in the current cycle cache space in this example; for example, the cache service process in this example Will , that is, sample "4", is sent to the training process Then, the sample "4" is searched in the random sequence list <3, 4, 1, 2,5, 0> to obtain its index , and then calculate the target cache service process number according to formula (4) , so the cache service process Push sample "4" to The exchange area;
[0091]
[0092] 2.5. For the training process ,like , then the training process The next sample will be ingested, starting with the index of that sample , then the training process Go to step 2.2 to ingest data , otherwise the training process Complete the data ingestion of the current cycle; for example, in this example After taking sample "4", because 0+2<6 is satisfied, Go to step 2.2 to take sample "1";
[0093] 3. Synchronize the data ingestion progress of all training processes. The training process that completes the data ingestion of the current cycle first will be in a blocked state. When the last training process completes the data ingestion, the data ingestion process of the current cycle is completed. On the one hand, the data exchange of the cache service process in the current cycle is completed, and each cache service process updates the local cache space. On the other hand, the blocked training processes are awakened at the same time, and the random sequence list is updated. All training processes enter the next cycle at the same time;
[0094] 3.1. There are differences in the progress of concurrent training processes ingesting data. The training process that completes data ingestion first will be blocked until the last training process completes data ingestion. For example, in this example, assume Before After completing data ingestion, After taking the last sample "0", Complete data ingestion in the current cycle, then train the process blocked;
[0095] 3.2. When the last training process completes data ingestion, each cache service process updates the local cache space, that is, the cache service process moves all the data in its swap area to its cache area, and then uses the random sequence list Replacement , then calculate and generate a new random sequence list and assign it to For example, in this example When data ingestion is complete, both training processes have completed data ingestion, and the cache service process and The data migration between the two is then completed. and The cache is empty, and The exchange area contains the sample data {1, 3, 5}, The exchange area contains the sample data {4,0, 2}, and Migrate the sample data in your exchange area to your own cache area, and then <4, 2, 1, 3,0, 5> is replaced by <3, 4, 1, 2, 5, 0>, and generate a new random sequence list and assign it to ;
[0096] 3.3. The blocked training processes are awakened at the same time, and all training processes enter the next cycle at the same time and start a new round of training; for example, in this example When data ingestion is complete, After being awakened, the two training processes enter the next cycle together and start a new round of training;
[0097] 4. When the training process of all cycles is completed, the training task is completed, the model is output, the cache space is released, and the cache service process is destroyed; for example, when the training of all cycles in this example is completed, the entire training task is completed and the model is output, and the cache service process is destroyed. and Release their cache area and swap area respectively, and then destroy the two cache service processes;
[0098] 5. The user inputs image data into the model, and the model outputs image classification results.
[0099] The distributed cache system uses the local memory on the computing nodes to build a cache space. This example uses DRAM, or dynamic random access memory, as the storage medium of the distributed cache system.
[0100] The cache area and swap area of the cache service process are two independent cache spaces. In order to speed up the data insertion, deletion and retrieval of the cache space, the cache space uses a data structure that combines a hash table and a red-black tree. Specifically, the cache space is first a hash table, and then each item in the table is a red-black tree, and each node of the red-black tree stores a sample data. When retrieving the cache space, first search the hash table to find the root node of the target red-black tree, and then retrieve the target node in the red-black tree.
[0101] The cache service nodes are connected using RDMA technology. The swap area of the cache service process is mapped to the RDMA buffer. Then the cache service process can quickly push the sample data to the swap area of other cache service processes through RDMA.
[0102] The cache service process dynamically manages the local cache space, that is, the cache area and the swap area use the cache space on demand. When sample data needs to be inserted into the cache area or the swap area, the present invention applies for cache space in the cache area or the swap area on demand to save the sample data. When the sample data in the cache area or the swap area is deleted, the present invention releases the cache space in the cache area or the swap area occupied by the sample data.
[0103] Embodiment 3
[0104] This embodiment relates to a method for distributed deep learning cache data layout. A training process can start multiple data loading processes or threads to increase the concurrency of each training process ingesting data. In this example, the data required by each training process is obtained from the cache service process in parallel by its data loading process working in collaboration. The specific steps are as follows:
[0105] 1. Initialize the deep learning training system, including preparing data in the backend storage system, initializing the distributed cache system, starting the cache service process, initializing the cache area and swap area of the cache space to empty, starting the training process, and generating a random sequence list;
[0106] 1.1. The backend storage of this instance uses the Lustre file system, which stores the image dataset. The number of samples contained in the dataset is recorded as , for example, the number of pictures in this example is 8;
[0107] 1.2. Compute nodes are Lustre clients, on which Lustre storage directories are mounted. Compute nodes that provide cache resources are also called cache service nodes. The cache space of the distributed cache system consists of the local storage of the cache service nodes. In this example, the local storage device uses SSD devices. The number of cache service nodes is recorded as ,The number of cache service nodes in this example is 2;
[0108] 1.3. Each cache service node starts a cache service process, denoted as , which manages the cache space of this node; during model training, the data set will be evenly cut and mapped to each cache service node, and each cache service node provides the same cache capacity; the cache space of the cache service node is divided into a cache area and a swap area. The cache area stores the sample data to be used in the current cycle, and the swap area receives data from each cache service process and these data will be used in the next cycle. The cache area and swap area capacity of each cache service node are the same, and the maximum number of samples that can be stored in the cache area and swap area is recorded as , and calculated by formula (1) Therefore, in this example The value of is 4, that is, the capacity of the cache area and the swap area are both 4; when the training task starts, the cache area and the swap area on each cache service node are empty;
[0109]
[0110] 1.4. Initialize the model training task: The training model of this example uses ResNet50, and the training device uses the GPU on the computing node. Each GPU device corresponds to a rank training process, and the number of rank training processes is recorded as In this example, there are two rank training processes; each rank training process starts two data loading processes; each data loading process corresponds to a cache client, and the data required by the rank training process is obtained by its data loading process in collaboration;
[0111] 1.5. Calculate and generate a random sequence list for the current period and a list of random sequences for the next cycle , for example in this example The value is <1, 4, 7, 2, 3, 0, 6, 5>, The value is <2, 6, 3, 4, 5, 7, 1, 0>;
[0112] 2. The training task of the current cycle starts, and all training processes execute the model training task in parallel. Each training process dynamically selects the cache service process to ingest data. When the training process ingests data, the cache service process dynamically exchanges data in an asynchronous manner to implement data layout for the next cycle of data ingestion. The main steps of the rank training process ingesting data and the cache service process migrating data include:
[0113] 2.1 Training Process Random sequence list according to the current period Ingest data, and The sample data selected is Index in The formula (2) must be satisfied:
[0114]
[0115] Among them, start is the starting index value, end is the length of the random sequence list, and step is the step size; for example, in this example ,then The data to be ingested in the current cycle is {1, 7, 3, 6}, The data to be ingested in the current cycle is {4, 2, 0, 5}, where Start two data loading processes, denoted as and , Start two data loading processes, denoted as and ;
[0116] 2.2. All training processes concurrently ingest data to train the process For example, it ingests samples in batches through the data loading process: first, the cache service process is selected, that is, the modulus operation is performed according to formula (3) to calculate the cache service process number k, so as to determine the target cache service process as , then the training process Data loading process and Concurrently to the cache service process Request a sample; for example The first samples ingested in the current cycle are <1, 7>. For sample "1", the corresponding parameter value is , then the cache service process number can be calculated , for sample "7", the corresponding parameter value is , then the cache service process number can be calculated , then the data is loaded into the process From the cache service process Take sample "1" and return it to the training process , while data loading process From the cache service process Ingest sample "7" and return it to the training process ,at the same time, The first batch of samples ingested in the current cycle is <4, 2>. For sample "4", the corresponding parameter value is , then the cache service process number can be calculated , for sample "2", the corresponding parameter value is , then the cache service process number can be calculated , then the data is loaded into the process From the cache service process Take sample "4" and return it to the training process , while data loading process From the cache service process Take sample "2" and return it to the training process ;
[0117]
[0118] 2.3. Cache service process After receiving the request, Retrieve samples from a local cache ,like If the cache hits, the cache service process The sample Remove from the cache and return the data to the data loading process; if If the cache does not hit, the cache service process Read samples from the backend storage , and then returns the data to the data loading process; for example, in this example Retrieve from local cache , that is, sample "1". According to step 1.3, the cache area on the cache service node is empty when the training task starts, so sample "1" does not hit the cache area. Read sample "1" from the GPFS mount directory and return it to the data loading process ;
[0119] 2.4. When the sample After being returned to the training process, the cache service process Execute the sample asynchronously Data migration, first, calculate the sample The cache service process to which it belongs in the next cycle ,Sure The method is to list the random sequence in the next cycle Search ,Sure exist Index in , and perform modulus operation according to formula 4 to calculate In the next cycle, the cache service process number m should be placed; then, the cache service process The sample Push to cache service process , and by The sample Save in The exchange area completes the current cycle sample Layout; Figure 3 Describes a schematic diagram of the cache service process pushing data to each other; for example, in this example, the cache service process Will , i.e. sample "1", is sent to the data loading process Then, the sample "1" is retrieved from the random sequence list <2, 6, 3, 4, 5, 7, 1, 0> of the next cycle to obtain its index. , and then calculate the target cache service process number according to formula (4) , so the cache service process Push sample "1" to The exchange area;
[0120]
[0121] 2.5. For the training process , if the index of the last sample in the current batch satisfies , then the training process Go to step 2.2 to acquire the next sample, otherwise the training process Complete the data ingestion of the current cycle; for example, the training process in this example Through the data loading process and After taking samples "1" and "7", because the index of the last sample "7" in the current batch satisfies 2+2<6, Go to step 2.2 to acquire the next batch of samples <3,6>;
[0122] 3. Synchronize the data ingestion progress of all training processes. The training process and its data loading process that complete the data ingestion of the current cycle first will be in a blocked state. When the last training process completes data ingestion, the data ingestion process of the current cycle is completed. On the one hand, the data exchange of the cache service process in the current cycle is completed, and each cache service process updates the local cache space. On the other hand, the blocked training process and its data loading process are awakened at the same time, and the random sequence list is updated. All training processes enter the next cycle at the same time;
[0123] 3.1. There are differences in the progress of concurrent training processes ingesting data. The training process that completes data ingestion first and its data loading process will be blocked until the last training process completes data ingestion. For example, in this example, assume Before After data ingestion is completed, the training process After taking the last sample "6", Complete data ingestion in the current cycle, then train the process and its data loading process and blocked;
[0124] 3.2. When the last training process completes data ingestion, each cache service process updates the local cache space, that is, the cache service process moves all the data in its swap area to its cache area, and then uses the random sequence list Replacement , then calculate and generate a new random sequence list and assign it to For example, in this example When data ingestion is complete, both training processes have completed data ingestion, and the cache service process and The data migration between the two is then completed. and The cache is empty, and The exchange area contains the sample data {1, 2, 3, 5}, The swap area contains the sample data {7, 4, 6, 0}, and Migrate the sample data in your exchange area to your own cache area, and then the random sequence list of the current cycle <1, 4, 7, 2, 3, 0, 6, 5> is replaced by a random sequence list for the next cycle <2, 6, 3, 4, 5, 7, 1, 0>, and generate a new random sequence list and assign it to ;
[0125] 3.3. The blocked training process and its data loading process are awakened at the same time, and all training processes enter the next cycle at the same time and start a new round of training; for example, in this example, the training process When data ingestion is complete, the training process and its data loading process and After being awakened, the two training processes enter the next cycle together and start a new round of training;
[0126] 4. When the training process of all cycles is completed, the training task is completed, the model is output, the training process and its data loading process are destroyed, the cache space is released, and the cache service process is destroyed; for example, when the training of all cycles in this example is completed, the entire training task is completed and the model is output, and the training process is destroyed and , destroy the data loading process , cache service process and Release their cache area and swap area respectively, and then destroy the two cache service processes;
[0127] 5. The user inputs image data into the model, and the model outputs image classification results.
[0128] Embodiment 4
[0129] This embodiment relates to a distributed deep learning cache data storage device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a distributed deep learning cache data storage method of embodiment 1.
[0130] Embodiment 5
[0131] This embodiment relates to a computer-readable storage medium, characterized in that a program is stored thereon, and when the program is executed by a processor, a distributed deep learning cache data storage method of embodiment 1 is implemented.
[0132] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.
Claims
1. A method for distributed deep learning cache data storage, characterized in that: The steps include: Step 1: Initialize the deep learning training system, including preparing data in the backend storage system, initializing the distributed cache system, starting the cache service process, initializing the cache area and swap area of the cache space to empty, starting the training process, and generating a random sequence list; specifically, it includes the following sub-steps: S1.1, the training data set is stored in the backend storage system, and the number of samples contained in the data set is recorded as ,During the training task execution, the dataset will be evenly cut and mapped to the local cache of the computing nodes; S1.2, the cache space of the distributed cache system consists of the local storage of the computing nodes. The computing nodes that provide cache space are cache service nodes, and their number is recorded as , each computing node provides the same cache capacity, and a cache service process is started on each computing node, denoted as ,This process manages the local cache space of the computing node; S1.3, the local cache space of each computing node is divided into a cache area and a swap area. The cache area is used to store the sample data to be used in the current cycle, and the swap area is used to receive data from each cache service process and these data will be used in the next cycle. The capacity of the cache area and the swap area is the same, and the maximum number of samples that can be stored in the cache area and the swap area is recorded as , and calculated by formula (1) ;When the training task starts, the cache area and swap area on each computing node are empty; S1.4, when the training task is initialized, all rank training processes are started, and the number of rank training processes is recorded as ,Each rank training process corresponds to a cache client. In addition, in order to improve the concurrency of data access, a rank training process can start multiple data loading processes for data ingestion; S1.5, calculate and generate a random sequence list for the current cycle, denoted as , calculate and generate the random sequence list of the next cycle, recorded as ; Step 2: The training task of the current cycle starts, and all training processes execute the model training task in parallel, where each training process dynamically selects the cache service process to ingest data; when the training process ingests data, the cache service process dynamically exchanges data in an asynchronous manner to implement data layout for the next cycle of data ingestion; it includes the following sub-steps: S2.1, Training Process Random sequence list according to the current period Ingest data, and The sample data selected is Index in Formula (2) must be satisfied; Among them, start is the starting index value, end is the length of the random sequence list, and step is the step size; S2.2, all training processes ingest data concurrently, training process Ingest samples When , we first select the cache service process, that is, we perform a modulus operation according to formula (3) to calculate the cache service process number k, so as to determine the target cache service process as , then the training process To the cache service process Request Sample ; S2.3, cache service process After receiving the request, Retrieve samples from a local cache ,like If the cache hits, the cache service process The sample Remove from the buffer and return the data to the training process ;like If the cache does not hit, the cache service process Read samples from the backend storage , and then return the data to the training process ; S2.4, when the sample After being returned to the training process, the cache service process Execute the sample asynchronously Data migration, first, calculate the sample The cache service process to which it belongs in the next cycle ,Sure The method is to list the random sequence in the next cycle Search ,Sure exist Index in , and perform modulus operation according to formula (4) to calculate The number m of the cache service process should be placed in the next cycle; Then, cache the service process The sample Push to cache service process , and by The sample Save in The exchange area completes the current cycle sample Layout; S2.5, for the training process ,like , then the training process The next sample will be ingested, starting with the index of that sample , then the training process Go to step S2.2 to ingest data , otherwise the training process Complete the data ingestion of the current cycle; Step 3: Synchronize the data ingestion progress of all training processes. The training process that completes the data ingestion of the current cycle first will be in a blocked state. When the last training process completes the data ingestion, the data ingestion process of the current cycle is completed. On the one hand, the data exchange of the cache service process in the current cycle is completed, and each cache service process updates the local cache space. On the other hand, the blocked training processes are awakened at the same time, and the random sequence list is updated. All training processes enter the next cycle at the same time. It includes the following sub-steps: S3.1, there are differences in the progress of concurrent training processes ingesting data. The training process that completes data ingestion first will be blocked until the last training process completes data ingestion; S3.2, when the last training process completes data ingestion, on the one hand, each cache service process updates the local cache space, that is, the cache service process moves all the data in its swap area to its cache area; on the other hand, it uses a random sequence list Replacement , then calculate and generate a new random sequence list and assign it to ; S3.3, the blocked training processes are awakened at the same time, and all training processes enter the next cycle at the same time, starting a new round of training; Step 4: When the training process of all cycles is completed, the training task is completed, the model is output, the cache space is released, and the cache service process is destroyed; Step 5: Input data into the trained model to obtain the output of the model.
2. A distributed deep learning cache data storage method as claimed in claim 1, characterized in that: The cache space managed by the cache service process is divided into a cache area and a swap area. The cache area stores sample data that will be ingested by the training process in the current cycle. When the current cycle ends, the sample data in the cache area will be consumed and empty; the swap area successively receives sample data from each cache service process in the current cycle. When the current cycle is about to end, the swap area will be full of sample data.
3. A distributed deep learning cache data storage method as claimed in claim 1, characterized in that: The cache service process uses a hash list to organize the cache area and the exchange area to accelerate the efficiency of sample data retrieval in the cache space.
4. The method for storing distributed deep learning cache data according to claim 1, characterized in that: The random sequence list of the next cycle creates an additional index for the list to speed up the random sequence list retrieval speed.
5. The method for storing distributed deep learning cache data according to claim 1, characterized in that: The cache service nodes are interconnected using RDMA technology, so that the cache service process can use a fast network link to exchange sample data, and the data migration between cache service processes is performed asynchronously, which does not hinder the training process from ingesting data from the cache area.
6. A distributed deep learning cache data storage device, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a distributed deep learning cache data storage method according to any one of claims 1 to 5.
7. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a distributed deep learning cache data storage method according to any one of claims 1 to 5 is implemented.
