An IO acceleration system and data storage method suitable for deep learning training

Abstract

The present invention discloses an IO acceleration system and data storage method suitable for deep learning training. Before a training task begins, the system and method prepare replicas corresponding to the training rounds. The replicas are stored in a storage device in a random sequence corresponding to the training rounds, based on the characteristics of the storage device. Reading the replica data according to the random sequence complies with the sequential access characteristics of the storage device. Each training round reads data from the corresponding replica. The method converts random IO requests to the storage device during training into sequential IO requests, fully utilizing the performance of sequential storage devices such as hard disks.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning, and more specifically, relates to an IO acceleration system and data storage method suitable for deep learning training. Background Art

[0002] Deep learning has developed rapidly in recent years and has been widely used in various fields, such as weather forecasting, image and speech recognition, natural language processing, healthcare, etc. The original algorithm for deep learning training is as follows: (1) define a loss function with parameters and assume a set of initialization parameters; (2) substitute the dataset tensor into the loss function for calculation, i.e. forward propagation; (3) evaluate the quality of the parameters based on the calculation results and the dataset labels, and update the parameters, i.e. backward propagation; (4) perform another round of substitution calculation until the loss function converges to the minimum or acceptable range. The set of parameters that minimize the loss function is the training result output by the training model, and the dataset tensor is the input. Each completed parameter update is called a step, and each complete traversal of the dataset is called an epoch. In the original algorithm, each epoch has only one step, that is, each update traverses and calculates the entire dataset, which requires a huge amount of calculation to achieve convergence. In order to reduce the amount of calculation and speed up the convergence speed, deep learning training usually uses a modified version of the algorithm, namely the stochastic gradient descent algorithm. In each step, the algorithm randomly uses a sample from the dataset to perform calculations and update the parameters. It then continues with the next non-overlapping sample until the entire dataset has been traversed, completing one round. In stochastic gradient descent, the parameters are updated multiple times per round compared to the original algorithm, resulting in faster convergence. The samples used in each step of stochastic gradient descent are called a batch. The number of samples in a batch is a configurable parameter, with common values ranging from tens to hundreds.

[0003] As can be seen, each round of random traversal of the dataset during training requires a large amount of random I / O when the dataset cannot fit into memory. In deep learning training, the richer the training samples, the more accurate the training results. Therefore, deep learning training datasets are often very large. Historically, the development of deep learning was primarily constrained by computing power. However, with the development of computing accelerators such as GPUs and the continuous improvement of distributed algorithms, the bottleneck has shifted to the storage layer. Recent research shows that during deep learning training using computing clusters, GPU resources are often idle and underutilized, resulting in poor training performance. In short, the development of computing accelerators and the random I / O characteristics of deep learning have created a gap between computing power resources and storage performance. This underutilization of computing power increases the time and economic costs of deep learning training and hinders the development of deep learning research.

[0004] Currently, the primary approach to addressing these storage bottlenecks is to address caching. However, deep learning workloads are characterized by multiple rounds of random sequence iterations of the dataset, making traditional cache replacement strategies inadequate. A common caching strategy involves storing a fixed portion of data in the cache, with the hit rate generally dependent solely on the ratio of cache capacity to dataset size. Because deep learning workloads are often extremely large, caching is very limited in effectiveness. Another solution involves addressing storage, grouping and storing raw samples in batches. The basic unit of the random sequence for a training round no longer corresponds to a sample, but rather to a batch. Because this approach increases the granularity of the random sequence, the randomness of the samples in the training batch is reduced, sacrificing training accuracy. Summary of the Invention

[0005] In response to the shortcomings of existing storage methods in deep learning scenarios, the present invention proposes an IO acceleration system and data storage method suitable for deep learning training. It uses inexpensive, large-capacity sequential storage devices to store multiple copies of data sets. Its purpose is to convert random access at the application level in deep learning training into sequential access at the storage device level, thereby improving the IO efficiency during the training process and increasing the overall training speed.

[0006] To achieve the above objectives, according to one aspect of the present invention, an IO acceleration system suitable for deep learning training is provided, comprising: a random sequence generation module, a replica preparation module, a training framework, and a storage system;

[0007] The random sequence generation module generates a random sequence according to the round number and sample quantity transmitted by the replica preparation module and the training framework;

[0008] The replica preparation module traverses the original dataset based on the number of replicas input by the user and the path of the original dataset, obtains the sample number and sample number of the original dataset, and then passes the round number and sample number to the random sequence generation module; receives the random sequence generated by the random sequence generation module, prepares replicas according to the random sequence, and generates the replica path based on the path of the original dataset and the round number; during the replica preparation process, the random sequence is divided into multiple clusters based on the number of samples contained in the clusters input by the user, and a storage request is sent to the storage system;

[0009] Before the entire training begins, the training framework obtains the sample sequence number based on the original data set and generates a path mapping table to map the sample sequence number to the original data set path. At the same time, it reads the label of the sample in the original data set and generates a category mapping table to map the sample sequence number to the category. The training framework performs round training according to the random sequence. When a sample with a certain sequence number needs to be read during training, the original path is checked from the path mapping table and combined with the round sequence number to obtain the sample path in the corresponding copy. The framework then sends a read request to the storage system to read the sample data for training.

[0010] The storage system is used to store the original data set and the copies prepared by the copy preparation module; the storage system includes a space management module, which is used to allocate continuous storage space in a hard disk for each cluster of the copy.

[0011] Furthermore, when the storage system includes multiple hard disks, the space management module stores the continuous clusters of each replica in a dispersed manner across the multiple hard disks.

[0012] Furthermore, the number of copies is set by the user based on the available space of the storage system, and can be partially deleted or added; the copy preparation module realizes the deletion and addition functions; when deleting or adding, the copy scans the copy path and numbers it with a round number based on the original data set input by the user and the method of generating the copy path when preparing the copy; when deleting a copy, it is deleted starting from the copy with the largest round number according to the deletion number input by the user until the specified number of copies is deleted or all copies have been deleted; when adding a copy, the copy is prepared starting from the number of the largest round plus 1 according to the increase number input by the user.

[0013] Furthermore, the number of samples contained in the cluster is set according to the hard disk IO characteristics and the average size of the samples.

[0014] Furthermore, when the training framework finds that the replica path does not exist because the current round sequence number has exceeded the maximum replica sequence number, the sample data will be read directly from the original data set.

[0015] A data storage method suitable for deep learning training, implemented based on an IO acceleration system, includes:

[0016] The replica preparation module traverses the original dataset according to the number of replicas input by the user and the path of the original dataset, obtains the sample sequence number and sample number of the original dataset, and then passes the round sequence number and sample number to the random sequence generation module;

[0017] The random sequence generation module generates a random sequence according to the round number and sample quantity transmitted by the replica preparation module;

[0018] The replica preparation module receives the random sequence generated by the random sequence generation module and prepares a replica according to the random sequence; during the preparation process, the random sequence is divided into multiple clusters according to the cluster unit input by the user, and a storage request is sent to the storage system;

[0019] The storage system receives a storage request from the replica preparation module, and the space management module allocates continuous storage space in a hard disk for each replica cluster;

[0020] Before the entire training begins, the training framework obtains the sample serial number based on the original dataset and generates a path mapping table to map the sample serial number to the original dataset path. At the same time, it reads the label of the sample in the original dataset and generates a category mapping table to map the sample serial number to the category.

[0021] The training framework performs round training according to the random sequence. During the training process, the sample path is modified to the sample path in the copy corresponding to the round, and a read request is sent to the storage system to read the sample data for training.

[0022] Furthermore, the copy path is to add a round number after the first-level directory of the relative path of the original data set to generate the first-level directory of the relative path of the copy, and the subdirectories under the relative first-level directory of the copy are consistent with the original data set; the copy and the original data set are stored in the same parent directory.

[0023] The beneficial effects of the present invention are as follows:

[0024] (1) The storage method provided by the present invention is suitable for deep learning training, which converts random IO to storage devices during training into sequential IO, and can fully utilize the sequential access performance of storage devices such as disks.

[0025] (2) The storage method provided by the present invention, which is suitable for deep learning training, stores samples in a cluster manner and can flexibly allocate storage space while ensuring the sequential access performance of the storage device.

[0026] (3) The storage method provided by the present invention, which is suitable for deep learning training, disperses continuous clusters on different storage devices when using array storage devices, which can improve the parallelism when pre-reading batches. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a framework diagram of an IO acceleration system suitable for deep learning training.

[0028] Figure 2 A random sequence generation module is provided for one embodiment of a method for storing data for deep learning training.

[0029] Figure 3 Schematic diagram of the storage method of clusters in replicas suitable for deep learning training data storage method.

[0030] Figure 4 Schematic diagram of a multi-hard disk cluster striped storage method suitable for deep learning training data storage.

[0031] Figure 5 A flowchart of a training process for an embodiment of a method for storing training data for deep learning. DETAILED DESCRIPTION

[0032] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more apparent. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0033] like Figure 1 As shown, the IO acceleration system suitable for deep learning training in this embodiment includes: a random sequence generation module, a replica preparation module, a training framework, and a storage system.

[0034] The storage system of this embodiment is composed of four hard disks.

[0035] The random sequence generation module is used to generate a random sequence based on the round number and sample size passed by the replica preparation module and the training framework. As long as the parameters are the same, the output random sequence will be the same.

[0036] The replica preparation module traverses the original dataset based on the number of replicas entered by the user and the path of the original dataset, obtains the sample number and sample number of the original dataset, and then passes the round number and sample number to the random sequence generation module; receives the random sequence generated by the random sequence generation module, prepares replicas according to the random sequence, and generates the replica path based on the path of the original dataset and the round number; during the replica preparation process, the random sequence is divided into multiple clusters based on the number of samples contained in the cluster entered by the user, and a storage request is sent to the storage system.

[0037] The number of samples contained in a cluster is determined based on the drive's I / O characteristics and the average sample size. This ensures that reads from different clusters utilize the drive's maximum bandwidth while minimizing the impact on space allocation flexibility. A hard drive's I / O overhead primarily consists of seek latency and data transfer latency. When a single I / O is large, such as in the MB range, data transfer latency dominates. Therefore, even if I / O is discontinuous, the drive's maximum bandwidth can be utilized for large I / Os. Testing the drive can help determine the appropriate I / O granularity. For example, starting with 4KB I / O units to measure disk performance and using progressively larger I / O granularities will show a linear increase in throughput initially, but will not significantly increase after reaching 2MB. Therefore, a cluster size of approximately 2MB is a reasonable size. Assuming the average sample size is 200KB, then every 10 files will be approximately 2MB. If training batches of 128 or 256 are common, setting the cluster size to 8 allows for a batch to be divided into several complete clusters. In practice, if a single sample is too small, a cluster can contain more samples, and multiple batches of samples exist in one cluster; if the training sample is too large, a cluster may contain only one file.

[0038] The number of replicas is set by the user based on the available space in the storage system, and some can be deleted or added. The replica preparation module implements the deletion and addition functions. When deleting or adding, the replica scans the replica path and numbers it with a round number based on the original data set entered by the user and the method used to generate the replica path when preparing the replica. When deleting a replica, it deletes the replica starting from the replica with the largest round number according to the deletion number entered by the user, until the specified number of replicas is deleted or all replicas are deleted. When adding a replica, it prepares the replica starting from the largest round number plus 1 according to the addition number entered by the user.

[0039] like Figure 2 Figure 2 shows a schematic diagram of the random sequence generation module generating a replica sequence. The original dataset contains 16 samples, numbered 0 to 15. The input parameters consist of two integers: one representing the replica sequence number and the other representing the total number of samples. The replica sequence number is the training round number, starting from 0 and increasing sequentially. Given a given number of samples, the replica sequence number determines the generated random sequence.

[0040] The storage system is used to store the original dataset and the replicas created by the replica preparation module. The storage system includes a space management module that allocates contiguous storage space on a single hard drive for each replica cluster. If the storage system includes multiple hard drives, the space management module distributes the contiguous clusters of each replica across the multiple hard drives.

[0041] In this embodiment, the data is clustered in order according to the random sequence corresponding to the replicas, and stored based on the number of samples set in the cluster, and a continuous space is allocated on a hard disk for the data of each cluster. Figure 3 As shown in the figure, when 4 samples are set as a cluster, the first cluster <1, 5, 8, 13> in replica 0 is stored. In this embodiment, when preparing the replica, the first-level directory of the relative path of the dataset is added with the round number to generate the first-level directory of the relative path of the replica. The subdirectories under the relative first-level directory of the replica are consistent with the original dataset. The replica and the original dataset are stored in the same parent directory. Therefore, in each round of training, it is only necessary to add the round number to the first-level directory of the original sample path to map it to the sample in the corresponding replica. Figure 3 As shown in the figure, the path of sample 1 in the original dataset is / train / class1 / file1, while the path of sample 1 in replica 0 is / train0 / class1 / file1. During training, if the batch size is also 4, then each batch of training will initiate 4 consecutive IO requests to the storage system. These 4 consecutive requests can be aggregated into one large IO request to the hard disk. In this embodiment, it is assumed that the storage system will aggregate consecutive IO to optimize IO efficiency. If the batch size is 8, then it will be divided into two clusters and stored on two hard disks respectively, and ultimately initiate IO requests to the two hard disks in parallel. Figure 4 This shows a situation where a batch is stored in two clusters.

[0042] Figure 5 The process of training based on replicas is demonstrated. Before the entire training begins, the training framework obtains the sample serial number based on the original dataset and generates a path mapping table to map the sample serial number to the original dataset path. At the same time, it reads the labels of the samples in the original dataset, generates a category mapping table, and maps the sample serial number to the category. The training framework performs round training based on a random sequence. When a sample with a certain serial number needs to be read during training, the original path is checked from the path mapping table and combined with the round serial number to obtain the sample path in the corresponding replica. A read request is then sent to the storage system to read the sample data for training. When the training framework finds that the replica path does not exist because the current round serial number has exceeded the maximum replica serial number, it will read the sample data directly from the original dataset.

[0043] The data storage method for deep learning training of the present invention is implemented based on the above-mentioned IO acceleration system and includes:

[0044] The replica preparation module traverses the original dataset according to the number of replicas input by the user and the path of the original dataset, obtains the sample sequence number and sample number of the original dataset, and then passes the round sequence number and sample number to the random sequence generation module;

[0045] The random sequence generation module generates a random sequence according to the round number and sample quantity transmitted by the replica preparation module;

[0046] The replica preparation module receives the random sequence generated by the random sequence generation module and prepares a replica according to the random sequence; during the preparation process, the random sequence is divided into multiple clusters according to the cluster unit input by the user, and a storage request is sent to the storage system;

[0047] The storage system receives a storage request from the replica preparation module, and the space management module allocates continuous storage space in a hard disk for each replica cluster;

[0048] Before the entire training begins, the training framework obtains the sample serial number based on the original dataset and generates a path mapping table to map the sample serial number to the original dataset path. At the same time, it reads the label of the sample in the original dataset and generates a category mapping table to map the sample serial number to the category.

[0049] The training framework performs round training according to the random sequence. During the training process, the sample path is modified to the sample path in the replica corresponding to the round, and a read request is sent to the storage system to read the sample data for training.

[0050] Those skilled in the art will understand that the foregoing descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art will still be able to modify the technical solutions described in the foregoing examples or substitute equivalents for some of the technical features therein. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the invention shall be included within the scope of protection of the invention.

Claims

1. An IO acceleration system suitable for deep learning training, characterized in that: The system includes: a random sequence generation module, a replica preparation module, a training framework, and a storage system; The random sequence generation module generates a random sequence according to the round number and sample quantity transmitted by the replica preparation module and the training framework; The replica preparation module traverses the original dataset based on the number of replicas input by the user and the path of the original dataset, obtains the sample number and sample number of the original dataset, and then passes the round number and sample number to the random sequence generation module; receives the random sequence generated by the random sequence generation module, prepares replicas according to the random sequence, and generates the replica path based on the path of the original dataset and the round number; during the replica preparation process, the random sequence is divided into multiple clusters based on the number of samples contained in the clusters input by the user, and a storage request is sent to the storage system; Before the entire training begins, the training framework obtains the sample sequence number based on the original data set and generates a path mapping table to map the sample sequence number to the original data set path. At the same time, it reads the label of the sample in the original data set and generates a category mapping table to map the sample sequence number to the category. The training framework performs round training according to the random sequence. When a sample with a certain sequence number needs to be read during training, the original path is checked from the path mapping table and combined with the round sequence number to obtain the sample path in the corresponding copy. The framework then sends a read request to the storage system to read the sample data for training. The storage system is used to store the original data set and the copies prepared by the copy preparation module; the storage system includes a space management module, which is used to allocate continuous storage space in a hard disk for each cluster of the copy.

2. The IO acceleration system suitable for deep learning training according to claim 1, characterized in that: When the storage system includes multiple hard disks, the space management module disperses and stores the continuous clusters of each replica in the multiple hard disks.

3. The IO acceleration system suitable for deep learning training according to claim 1 or 2, characterized in that: The number of copies is set by the user based on the available space of the storage system, and can be partially deleted or added; the copy preparation module realizes the deletion and addition functions; when deleting or adding, the copy scans the copy path and numbers it with a round number based on the original data set input by the user and the method of generating the copy path when preparing the copy; when deleting a copy, it is deleted starting from the copy with the largest round number according to the deletion number input by the user until the specified number of copies is deleted or all copies have been deleted; when adding a copy, the copy is prepared starting from the number of the largest round plus 1 according to the increase number input by the user.

4. The IO acceleration system suitable for deep learning training according to claim 1, characterized in that: The number of samples contained in the cluster is set according to the hard disk IO characteristics and the average size of the samples.

5. The IO acceleration system suitable for deep learning training according to claim 3, characterized in that: When the training framework finds that the replica path does not exist because the current round sequence number has exceeded the maximum replica sequence number, the sample data will be read directly from the original dataset.

6. A data storage method suitable for deep learning training, the method being implemented based on the IO acceleration system of any one of claims 1 to 5, characterized in that: include: The replica preparation module traverses the original dataset according to the number of replicas input by the user and the path of the original dataset, obtains the sample sequence number and sample number of the original dataset, and then passes the round sequence number and sample number to the random sequence generation module; The random sequence generation module generates a random sequence according to the round number and sample quantity transmitted by the replica preparation module; The replica preparation module receives the random sequence generated by the random sequence generation module and prepares a replica according to the random sequence; During the preparation process, the random sequence is divided into a plurality of clusters according to the unit of cluster input by the user, and a storage request is sent to the storage system; The storage system receives a storage request from the replica preparation module, and the space management module allocates continuous storage space in a hard disk for each replica cluster; Before the entire training begins, the training framework obtains the sample serial number based on the original dataset and generates a path mapping table to map the sample serial number to the original dataset path. At the same time, it reads the label of the sample in the original dataset and generates a category mapping table to map the sample serial number to the category. The training framework performs round training according to the random sequence. During the training process, the sample path is modified to the sample path in the copy corresponding to the round, and a read request is sent to the storage system to read the sample data for training.

7. The data storage method suitable for deep learning training according to claim 6, characterized in that: The copy path is generated by adding a round number to the first-level directory of the relative path of the original dataset to generate the first-level directory of the relative path of the copy. The subdirectories under the relative first-level directory of the copy are consistent with the original dataset; the copy and the original dataset are stored in the same parent directory.

Images