A DL training data reading method based on index shuffle

Abstract

The present invention discloses a method for reading DL training data based on index shuffle, which includes: persisting data to non-volatile memory and constructing array indexes; partitioning the array indexes and performing a multi-threaded lock-free parallel shuffle to obtain shuffled array indexes; traversing the array indexes and pre-reading the data from the non-volatile memory to DRAM based on the shuffled array indexes. By using the present invention, the file system can be simplified, the reading performance of the data set can be improved, and ultimately the training speed of the DNN can be increased. As a method for reading DL training data based on index shuffle, the present invention can be widely used in the field of computer systems.

Description

Technical Field

[0001] The present invention relates to the field of computer systems, and in particular to a DL training data reading method based on index shuffle. Background Art

[0002] Currently, the main shortcomings of the existing shuffle strategies for DNN training datasets are: 1) The index structure of the default file system is highly complex, which makes it scalable for large-scale, metadata-intensive DNN datasets. 2) The main drawback of the raw data shuffle is that it overloads the memory and CPU. 3) The main drawback of the metadata-based shuffle is that disk I / O is the main bottleneck. 4) The cache hit rate is extremely low, resulting in unexpected disk I / O and delaying the data reading process. 5) Single-threaded shuffles are inefficient, and multi-threaded shuffles involve lock overhead. Summary of the Invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a DL training data reading method based on index shuffle, which can simplify the file system, improve the reading performance of the data set, and ultimately increase the training speed of DNN.

[0004] The first technical solution adopted by the present invention is: a DL training data reading method based on index shuffle, comprising the following steps:

[0005] S1. Persist the data into non-volatile memory and build array indexes.

[0006] S2. Divide the array index and perform multi-threaded lock-free parallel shuffle to obtain the shuffled array index;

[0007] S3: Traverse the array index and pre-read the data from the non-volatile memory to DRAM based on the shuffled array index.

[0008] Furthermore, the step of persisting the data to the non-volatile memory and constructing the array index specifically includes:

[0009] S11. Obtain a dataset for a deep neural network;

[0010] S12, loading the data set into non-volatile memory and recording the address of each sample using an array, one data set corresponds to one array;

[0011] S13. Get the array index.

[0012] Furthermore, the step of dividing the array index and performing multi-threaded lock-free parallel shuffle to obtain the shuffled array index specifically includes:

[0013] S21, randomly partition the array index and generate multiple threads in the shuffle phase of each epoch in the deep neural network training;

[0014] S22. Based on the thread, shuffle the array according to the array index to obtain the shuffled array index.

[0015] Furthermore, it also includes:

[0016] S4: If it is determined that the target accuracy of the deep neural network training has not been achieved, the process goes to the next epoch and returns to step S21.

[0017] Furthermore, the step of randomly partitioning the array index and generating multiple threads in the shuffle phase of each epoch in the deep neural network training specifically includes:

[0018] S211, randomly divide the array index, generate random numbers and obtain multiple sub-arrays;

[0019] S212. In the shuffle phase of each epoch in the deep neural network training, a corresponding number of threads are generated according to the random number.

[0020] Furthermore, the random number corresponds to the number of sub-arrays, each thread is responsible for shuffling only one sub-array, and the threads are isolated from each other.

[0021] Furthermore, the array index is traversed in a snake-like manner, specifically:

[0022] In the first epoch, the data is read by traversing the index in the forward direction and the starting index of the cached data is recorded;

[0023] In the second epoch, reverse traversal index is adopted to read all samples in descending order of index.

[0024] In the third epoch, data is read by traversing the index in the forward direction;

[0025] The steps are traversed cyclically until the target accuracy of deep neural network training is reached, and the traversal order of each epoch is opposite to the previous traversal order.

[0026] The beneficial effects of the method of the present invention are as follows: in response to the problems of low shuffle and data reading efficiency in the DNN training computing framework, the present invention proposes a dedicated file system, designs a more suitable array index structure based on the characteristics of the DNN training data set and the data set access characteristics, further implements a multi-threaded parallel and efficient lock-free shuffle strategy based on the array structure, adopts a snake traversal method to prioritize cache reading and enable pre-reading, thereby improving read throughput, and ultimately aims to alleviate the bottleneck of the data reading stage in DNN training and improve the execution efficiency of DNN training jobs. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flowchart of the steps of a DL training data reading method based on index shuffle of the present invention; DETAILED DESCRIPTION

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0029] Our invention builds on the traditional deep neural network (DNN) computing framework by shifting the dataset shuffle operation from the application layer to the file system layer. We also introduce non-volatile memory (NVM) and leverage NVM's byte-addressable nature to design an efficient indexing structure. This simplifies metadata and enables multi-threaded, lock-free parallel shuffles, making the file system's functionality more compatible with the data access patterns of DNN training. Traditional DNN computing frameworks have dedicated modules for dataset shuffling.

[0030] like Figure 1 As shown, the present invention provides a DL training data reading method based on index shuffle, which includes the following steps:

[0031] S1. Persist the data into non-volatile memory and build array indexes.

[0032] S11. Obtain a dataset for a deep neural network;

[0033] S12, loading the data set into non-volatile memory and recording the address of each sample using an array, one data set corresponds to one array;

[0034] S13. Get the array index.

[0035] Specifically, the index structure of the present invention is designed as follows: while loading the data set into non-volatile memory, the address of each sample is recorded in an array. One data set corresponds to one array, so each sample only needs 4 bytes to store metadata, and the total space required for the entire index structure is the number of samples × 4 bytes. The address space of the array in the memory is continuous, and each element can be directly located by subscript, so there is no additional pointer overhead. Since there are no semantics such as parent-child nodes between elements, even if any two elements are swapped, the function of the index structure will not be destroyed. The biggest advantage this brings us is that we can directly shuffle the index itself, and there is no redundant metadata except for the index itself, which maximizes space utilization.

[0036] S2. Divide the array index and perform multi-threaded lock-free parallel shuffle to obtain the shuffled array index;

[0037] S21, randomly partition the array index and generate multiple threads in the shuffle phase of each epoch in the deep neural network training;

[0038] S211, randomly divide the array index, generate random numbers and obtain multiple sub-arrays;

[0039] S212. In the shuffle phase of each epoch in the deep neural network training, a corresponding number of threads are generated according to the random number.

[0040] S22. Based on the thread, shuffle the array according to the array index to obtain the shuffled array index.

[0041] Specifically, based on the above-mentioned index structure, the present invention designs a multi-threaded lock-free parallel shuffle, and the specific strategy is as follows: divide the array into multiple sub-arrays, start a shuffle thread for each sub-array, each thread is only responsible for the shuffle of one sub-array, and the threads are isolated from each other, and no message communication and data sharing are performed. Therefore, each element will only be accessed by one thread, and no locking is required during access. It is an extremely efficient parallel mode. The specific shuffle algorithm is: traverse the sub-arrays, and exchange each element with an element at a subsequent random position. The algorithm complexity is O(n). For a shuffle, this strategy is a pseudo-shuffle, because each element will only appear in the sub-array where it is located. Therefore, in order to better ensure the randomness of sample reading and thus ensure the ultimate accuracy of model training, the present invention adopts random division when dividing the sub-arrays. The specific approach is: when dividing the sub-arrays, generate a random number, which is the number of divided sub-arrays, that is, the number of threads, and then divide the array into two halves according to the random number, with adjacent elements as a group. In this way, the position that each element can reach in different epochs is different, so as long as the epoch number is large enough, each element has the chance to reach any position in the entire array.

[0042] During DNN training, the access pattern of the dataset in each epoch has the following two characteristics: first, the order in which each sample is accessed is random; second, each sample is accessed only once. In other words, each element in the index is read once in each epoch. Because the tree structure does not support in-place shuffles, it is impossible to read all elements in a traversal manner. Each access to a sample is a random read of the index, with a time complexity of O(logn). However, using an array, we can perform a full scan of all elements using the simplest array traversal method, reducing the time complexity to O(1).

[0043] S3: Traverse the array index and pre-read the data from the non-volatile memory to DRAM based on the shuffled array index.

[0044] S4: If it is determined that the target accuracy of the deep neural network training has not been achieved, the process goes to the next epoch and returns to step S21.

[0045] As a further preferred embodiment of the method, the array index is traversed in a snake-like manner, specifically:

[0046] In the first epoch, the data is read by traversing the index in the forward direction and the starting index of the cached data is recorded;

[0047] In the second epoch, reverse traversal index is adopted to read all samples in descending order of index.

[0048] In the third epoch, data is read by traversing the index in the forward direction;

[0049] The steps are traversed cyclically until the target accuracy of deep neural network training is reached, and the traversal order of each epoch is opposite to the previous traversal order.

[0050] Specifically, the read performance of DRAM is better than that of NVM. In order to fully utilize the advantages of DRAM, the present invention additionally stipulates the index traversal method when reading the data set. The system still uses the operating system's default LRU cache replacement strategy. In this way, in each epoch, the last set of samples accessed will be cached in the memory. In the next epoch, this set of cached samples will be read first. The specific method is as follows: in the first epoch, the index is traversed forward to read the data, and the starting index of the cached data is recorded. During shuffle, the data is divided into a cached part and a non-cached part, and the above-mentioned shuffle strategy is executed separately for these two parts; in the second epoch, since the samples with the largest index are all in the cache, we adopt a reverse traversal method to read all samples in descending order of the index; in the third epoch, the data in the cache is replaced with the sample last accessed in the second epoch, that is, the sample with the smallest index, so the forward traversal method is adopted; the traversal method of subsequent epochs is similar. It can be seen that the traversal order of each time is opposite to the previous one, forming a snake-like traversal. This allows us to prioritize access to cached data in each epoch, improving the cache hit rate and ultimately improving read performance. To enable more data to be retrieved from DRAM, the present invention further implements a pre-read strategy that prioritizes accessed data into DRAM based on the index. This ensures efficient GPU data retrieval, reduces GPU latency, and improves resource utilization.

[0051] This paper implements a file system with the aforementioned functionality. This system provides a shuffle(dataset) interface to upper-layer PyTorch-based deep learning applications, where `dataset` specifies the dataset used for this DNN training task. Developers simply call this function at the beginning of each epoch to shuffle the dataset.

[0052] In summary, to address the problems of low shuffle and data reading efficiency in the DNN training computing framework, the present invention proposes a dedicated file system, designs a more suitable array index structure based on the characteristics of the DNN training dataset and the dataset access characteristics, and further implements an efficient multi-threaded parallel lock-free shuffle strategy based on the array structure. It uses a snake-like traversal method to prioritize cache reading and enable pre-reading to improve read throughput. The ultimate goal is to alleviate the bottleneck of the data reading stage in DNN training and improve the execution efficiency of DNN training jobs.

[0053] The beneficial effects of the present invention specifically include as follows:

[0054] 1. Improved deep learning training speed: Without affecting the randomness of sample reading and ensuring the ultimate accuracy of DNN training, a lock-free parallel shuffle is implemented, exponentially improving shuffle efficiency. The performance improvement is related to the number of threads. By traversing the array to read data, the indexing time complexity is O(1), which is both convenient and fast. By introducing a priority read cache and pre-reading strategy, all data can be retrieved from DRAM, significantly improving the GPU's data acquisition speed and throughput. The shuffle and data reading phases are no longer bottlenecks, and the overall performance of DNN training will be improved.

[0055] 2. Save space resources: At the application level, there is no need to maintain the path information of each sample, only the dataset ID is needed; at the file system level, the array structure address space is continuous and no additional space is required to maintain pointers, which can save a lot of memory space.

[0056] 3. Improved resource utilization: With increased data reading speed, the idle time the GPU spends waiting for data is reduced, allowing GPU resources to be more fully utilized. Multi-threaded shuffles can effectively utilize multi-core CPUs, improving CPU resource utilization. In addition, saving memory space also means improved memory utilization.

[0057] 4. Improved system scalability: Metadata-based shuffles require less data, and the array structure has low time and space overhead. They can store large amounts of data, are easy to expand and maintain, and are well-suited to dataset expansion. The saved memory space and CPU cycles can be used for data preprocessing or storing intermediate data, making them more adaptable to the expansion of DNN models.

[0058] A DL training data reading device based on index shuffle:

[0059] at least one processor;

[0060] at least one memory for storing at least one program;

[0061] When the at least one program is executed by the at least one processor, the at least one processor implements the above-described method for reading DL training data based on index shuffle.

[0062] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0063] A storage medium storing processor-executable instructions, characterized in that the processor-executable instructions, when executed by the processor, are used to implement the above-mentioned index shuffle-based DL training data reading method.

[0064] The contents of the above method embodiments are all applicable to the present storage medium embodiment. The functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0065] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A DL training data reading method based on index shuffle, characterized in that: The following steps are involved: S1. Persist the data into non-volatile memory and build array indexes. S2. Divide the array index and perform multi-threaded lock-free parallel shuffle to obtain the shuffled array index; S3, traverse the array index and pre-read the data from the non-volatile memory to DRAM according to the shuffled array index; The step of persisting data to non-volatile memory and constructing an array index specifically includes: S11. Obtain a dataset for a deep neural network; S12, loading the data set into non-volatile memory and recording the address of each sample using an array, where one data set corresponds to one array; S13, get array index; The step of dividing the array index and performing multi-threaded lock-free parallel shuffle to obtain the shuffled array index specifically includes: S21, randomly partition the array index and generate multiple threads in the shuffle phase of each epoch in the deep neural network training; S22, based on the thread, shuffle the array according to the array index to obtain the shuffled array index; The step of randomly partitioning the array index and generating multiple threads during the shuffle phase of each epoch in the deep neural network training specifically includes: S211, randomly divide the array index, generate random numbers and obtain multiple sub-arrays; S212. In the shuffle phase of each epoch in the deep neural network training, generate a corresponding number of threads according to the random number; The method of traversing the array index is snake traversal, specifically: In the first epoch, the data is read by traversing the index in the forward direction and the starting index of the cached data is recorded; In the second epoch, reverse traversal index is adopted to read all samples in descending order of index. In the third epoch, data is read by traversing the index in the forward direction; The steps are traversed repeatedly until the target accuracy of deep neural network training is reached, and the traversal order of each epoch is opposite to the previous traversal order; The specific shuffle algorithm is: traverse the subarray and exchange each element with an element at a random position in the subsequent array. The algorithm complexity is O(n).

2. The method for reading DL training data based on index shuffle according to claim 1, characterized in that: Also includes: S4: If it is determined that the target accuracy of the deep neural network training has not been achieved, the process goes to the next epoch and returns to step S21.

3. The DL training data reading method based on index shuffle according to claim 2, characterized in that: The random number corresponds to the number of sub-arrays. Each thread is responsible for shuffling only one sub-array and the threads are isolated from each other.

Images