A Distributed Deep Learning Caching Method Based on Sample Importance Sampling

Abstract

This invention discloses a distributed deep learning caching method based on sample importance sampling. The method first merges small files into large files for storage and constructs metadata for each small file, storing it in an index system based on a two-layer hash function. Secondly, during the data reading phase, the corresponding metadata information is retrieved using the requested small file name as the key, and the small file content is read, converting small file I / O into block-level I / O. Then, the importance level of a sample is evaluated using its loss value and predicted probability variance to guide sampling and caching strategies. Finally, customized caching strategies are provided for each type of sample through importance cache and replacement cache. These strategies not only improve cache hit rate but also alleviate the read amplification problem of block-level I / O, while ensuring that the model convergence accuracy is not affected. This invention improves data access performance and cache utilization, enhancing the overall performance and efficiency of the system.

Description

Technical Field

[0001] This invention belongs to the fields of big data technology, artificial intelligence and distributed storage, and specifically relates to a distributed deep learning caching method based on sample importance sampling. Background Technology

[0002] Unlike traditional computing tasks, deep learning tasks have the following characteristics: (1) Massive small files: The datasets involved in deep learning tasks usually contain a large number of small files. For example, datasets like ImageNet and Google Open Images contain millions or even tens of millions of small files. The average size of these files is relatively small, but the number is huge, posing a challenge to the storage system. (2) Access patterns: In deep learning training tasks, the access patterns of the dataset are different from those of traditional tasks. Typically, all files in the dataset need to be accessed once in each epoch, instead of accessing them one by one as needed, as in traditional tasks. In addition, these accesses do not involve hot data and do not conform to cache-friendly access patterns, which places demands on the design and performance of the storage system. (3) Read-write separation: Deep learning tasks usually involve reading-only access to the dataset and do not require writing to the data. The training task only needs to read samples from the dataset for model training and does not need to modify or write data.

[0003] Existing distributed storage systems such as Google File System and HDFS typically employ a master-slave architecture, where the master server manages the file system's metadata, while the data servers store the actual data. However, these systems are primarily designed for storing large files and lack optimization strategies for storing small files. In these systems, each file or block corresponds to metadata, and directly storing small files from the dataset into the system can lead to increased memory consumption on the master node.

[0004] Existing research indicates that certain samples contribute more significantly to improving the accuracy of deep learning models and are therefore considered more important. Based on this, the sample importance sampling method breaks with the traditional rule of training all samples with equal probability, instead training important samples at a higher frequency and unimportant samples at a lower frequency. This concentrates computational resources on the most important samples and accelerates convergence. However, this method primarily focuses on reducing computation time, without adequately considering I / O optimization. Therefore, in AI training scenarios, the problems of difficult dataset management and inefficient access urgently need to be addressed. Summary of the Invention

[0005] This invention aims to address the problems of slow dataset reading speed, high memory consumption of metadata servers, and low cache hit rate in deep learning tasks. To this end, this invention proposes a distributed deep learning caching method based on sample importance sampling to solve input / output (IO) bottlenecks in scenarios such as computer vision, natural language processing, and smart city construction. This method first merges and stores massive amounts of small files and constructs an indexing system based on a two-layer hash function to improve data access efficiency. Then, it evaluates the importance score of individual samples, classifying them into different levels. For samples of different levels, two different cache regions are used to implement different caching strategies.

[0006] The steps of this invention are as follows:

[0007] Step 1: In the data storage phase, small files are first randomly merged into large files according to a preset capacity threshold for storage. Next, the storage status of the small files within the merged large files is analyzed to construct small file metadata, including the file ID, the chunk it belongs to, its offset, and its length. The small file metadata will be stored in a two-layer hash function-based indexing system: the first layer is a block index layer, used to index the metadata to the index file; the second layer is an intra-block index layer, used to locate the specific storage location of the metadata within a block.

[0008] Step 2: During the data reading phase, the corresponding small file metadata information is quickly retrieved using a double-layer hash function, with the requested small file name as the key. Based on the small file metadata information, the small file content is read from the chunk file, transforming small file I / O into block-level I / O.

[0009] Step 3: During the deep learning model training phase, this invention determines the importance of samples by evaluating their loss values ​​and prediction probability variance. Based on their importance scores, samples are categorized into three sampling levels: high-level samples, mid-level samples, and low-level samples. The sample level guides the sampling and caching strategy. The sampler will actively sample high-level and mid-level samples, but will not sample low-level samples.

[0010] Step 4: In the caching stage, this invention introduces two different caching areas: an importance cache area and a replacement cache area.

[0011] The importance cache is used to cache chunks and implement a cache replacement strategy based on the chunk's importance score. This cache uses a heap structure ordered by chunk importance to track the state of chunks in the cache. When a requested chunk exists in the cache, its data is returned directly. If the requested chunk is not in the cache, its importance value is compared with the lowest-important chunk in the cache. If the newly requested chunk is more important, the cache is updated, the lowest-important chunk is evicted, and the newly requested chunk is added to the cache. Conversely, if the newly requested chunk is less important, the sample data is read directly from the underlying storage system. The importance cache supports the hit rate of high- and mid-level samples by caching the most important chunks.

[0012] The replacement cache is used to handle cases where intermediate samples are not found. When a requested sample is an intermediate sample and is not found in the cache, a sample is randomly read from the replacement cache and returned. When the replacement cache is not full, the filling thread dynamically fills the replacement cache by randomly selecting unused intermediate and low-level samples from the importance cache in the current epoch.

[0013] The effects of this invention are:

[0014] 1. Data Merging and Storage: By randomly merging massive numbers of small files into large file chunks according to a preset capacity limit, redundant I / O operations are avoided. Each small file exists in only one chunk, effectively reducing the number of files and thus alleviating the memory burden on the metadata server. A two-layer hash function-based indexing system is used to store the metadata of small files, achieving efficient metadata management. This indexing system can quickly retrieve the metadata of small files and directly read the file content based on the metadata, improving the efficiency of file reading. This invention merges requests for the same file, reducing network I / O and file reading overhead. By merging requests for the same file, the number of communication and data transmissions is reduced, improving the overall performance and efficiency of the system.

[0015] 2. This invention dynamically assesses the importance of each sample by calculating its loss value and prediction variance, and assigns sampling levels to samples based on their importance. By prioritizing the sampling of important samples, the model can learn key information more quickly, accelerating the convergence process. Simultaneously, different caching strategies are adopted for different sample levels to improve cache utilization.

[0016] 3. This invention introduces an importance cache and a replacement cache, providing different caching strategies for samples of different levels. The importance cache is mainly used to cache chunks with high importance values ​​to improve the hit rate of important samples. The replacement cache solves the cache miss problem for intermediate samples through replacement reads. The replacement cache is dynamically filled using samples from the importance cache, which alleviates the read amplification problem of block-level I / O and ensures the diversity of training data.

[0017] 4. This invention aims to concentrate resources on important samples and accelerate model training by optimizing importance sampling and caching strategies. Simultaneously, to fully consider the data diversity of the sampling sequences, the sampling strategy and alternative caching of this invention ensure that the model has ample learning opportunities for all types of samples, thereby minimizing the impact on model convergence accuracy. Attached Figure Description

[0018] Figure 1 A flowchart of the system operation;

[0019] Figure 2 This is a diagram illustrating the data layout.

[0020] Figure 3 Construct a flowchart for the index file;

[0021] Figure 4 Importance scoring process;

[0022] Figure 5 This is a diagram illustrating the partition caching function. Detailed Implementation

[0023] This invention provides users with data storage and retrieval services based on the file management and cache scheduling functions of the Alluxio system. The following describes the process in conjunction with... Figure 1 The specific embodiments of the present invention will be described in further detail below.

[0024] Step 1: In the data storage stage, small files are randomly merged into large files according to the rule of the capacity limit. Next, the metadata of small files is constructed and stored in an index system based on a two-level hash function.

[0025] 1. File merging and metadata construction

[0026] The merging thread appends the uploaded files to a chunk file and constructs metadata for the smaller files used for retrieval. Each file's metadata is a fixed-length 16-bit string containing the following fields: file ID, position, offset, and length. The file ID field uniquely identifies a file using a hash algorithm to calculate the filename. The position field indicates the chunk file the file resides in, with the value 'i' indicating the file is stored in chunk-i. The offset field indicates the file's position relative to the start of the chunk file. The length field indicates the original length of the smaller file.

[0027] Each merged file chunk has a size limit. During the merging process, if the remaining space in the i-th chunk (chunk-i) is less than the size of the smaller file, the writing of chunk-i is completed, and a new file (chunk-(i+1)) is created to continue writing the content of the new smaller file. The content of a smaller file is contained within only one chunk file, thus achieving block storage and management.

[0028] 2. Index Building

[0029] First, the index construction thread uses an extensible hash function to implement the first hash function Hash0 to construct the first block index layer, implementing the indexing of metadata to blocks: (e.g.) Figure 3 As shown, buckets serve as a temporary data structure to store metadata information during the merging process, and are ultimately saved as index files. The Hash0 function dynamically distributes metadata information into different buckets. To control the number of data items in each bucket and avoid performance degradation caused by excessively large buckets, Hash0 dynamically increases the number of buckets, as described below: Hash0 maintains a list of buckets, and based on the calculation result of Hash0, inserts metadata into the corresponding bucket. When a bucket reaches its capacity limit, the bucket will overflow and be split.

[0030] For bucket splitting: First, a new bucket is created and added to the bucket list, while updating the Hash0 function to accommodate the new bucket structure. Second, based on the new Hash0 function, the positions of all data in the overflow bucket are recalculated, so that some data will be redistributed to the new bucket. This process effectively handles data overflow, maintaining bucket balance and performance stability.

[0031] Secondly, the index construction thread uses the minimal perfect hash function Hash1 to build the second-level intra-block index layer, which is used to implement the metadata of index metadata within the block. After all files have been merged, the number of data items in the buckets is determined. For each bucket, Hash1 is constructed: using the N data file IDs in the bucket as a set S, and taking S as input, a minimal perfect hash function is constructed. This function maps these N records to the value range [0, N-1], and sorts the data items in the bucket using this mapping result.

[0032] Compared to ordinary hashing, the hash table constructed by the minimum perfect hash function makes full use of memory space and saves space resources. It avoids hash collisions and has a search efficiency of 0(1) even in the worst case.

[0033] Finally, the index information is persisted: such as Figure 2 As shown, the information in the i-th bucket is written to the i-th index file, and the Hash0 and all Hash1 used in this dataset are recorded in the dataset metadata file.

[0034] Step 2: Read the file. The system input is the filename. It quickly retrieves the corresponding metadata information using a double-layer hash function. Based on the metadata information, it reads the contents of the small file from the chunk file. The steps are as follows:

[0035] First, read the Hash0 and all Hash1 functions of the dataset.

[0036] Secondly, the system inputs the filename of the small file f, calculates the corresponding hash value x, and uses hash0 to calculate x to determine the index file to which the metadata belongs. For example, if Hash0(x) = i, then the metadata will be found at index-i. The Hash0 function is an extensible hash function that can query which index file the file's metadata is in with a time complexity of O(1). Then, the Hash1 function corresponding to index-i is used to calculate x to obtain the position y of the file's metadata in indexfile-i. Since each record is a fixed length of 16 bits, the offset of the metadata in the index file can be calculated using formula (1):

[0037] metadata offset = y * 16 Bytes (1)

[0038] Finally, based on the metadata information, including the chunk file-a to which the small file belongs, and its offset b and length c within the chunk file, b bytes are skipped from chunk file-a, and c bytes of content, i.e., the content of the small file f, are read and returned to the deep learning training task. In one iteration, the above reading steps are repeated until all file readings for that iteration are completed.

[0039] Step 3: During the model training phase, the sample importance is first updated based on the latest loss value and predicted probability variance; for example... Figure 4 As shown, further, samples are scored based on their importance to determine their relative importance to the entire dataset. Samples are then classified according to their scores, and the classification of samples guides sampling and caching strategies.

[0040] First, after forward propagation of the samples, the loss value of the samples is calculated according to the cross-entropy function formula (2). And record the predicted probability p0(y|x) after normalization by the Softmax function. i ), where y i It is sample x i The true label is the one-hot encoded value, and θ is the model parameter. The prediction variance of the sample is calculated according to formula (3). Where p θ (y i |x i ) represents sample x i For the correct label y i The predicted probability, H i It is the historical prediction probability p(y) i |x i The set of |H i | is set H i The number of elements in the middle, It is set H i The average value.

[0041]

[0042]

[0043]

[0044] Secondly, this invention measures the relative importance of samples using scores and classifies samples based on these scores. The importance score of a sample consists of its loss value score and prediction variance score. Taking the calculation of the loss value score as an example, the cumulative distribution function (CDF) of the loss values ​​of all samples in the dataset is first constructed. loss(.), and then the importance score of the sample on the loss value is calculated using this function, as shown in formula (4). To avoid low-scoring samples being left unsampled for a long time, a smoothing constant ∈ is added to the importance score of each sample, ensuring that all samples have a certain chance to participate in training and will not be ignored for a long time. Each sample is assigned a loss value score between 0 and 1, which is compared with a threshold randomly generated in the range of 0 to 1. Samples exceeding the threshold are classified as high-scoring samples on the loss value index, with high loss value scores; otherwise, they are classified as low-scoring samples, with low loss value scores. The prediction variance score of the sample is calculated in the same way, and it is determined whether the sample belongs to the high-scoring sample on this index.

[0045] Based on the above strategy and the evaluation of the two scores, this invention divides the samples into three sampling priorities, including:

[0046] High-level samples (H): High loss value score and high prediction variance score.

[0047] Intermediate sample (M): High loss value score or high prediction variance score.

[0048] Low-level samples (L): All samples other than high-level and mid-level samples. They perform relatively poorly in terms of loss value and prediction variance.

[0049] To prioritize training on samples that have a greater impact on model learning and prediction results, thereby accelerating model convergence, the sampler actively samples for H and M samples, constructing a sampling list based on importance. For samples of different importance, the cache adopts different response strategies to better support their requests and processing, improving cache hit rate.

[0050] Step 4: To address the low hit rate issue caused by limited cache space, this invention introduces two distinct cache areas: an importance cache area and a replacement cache area. These store chunks and samples for replacement reads, respectively, to efficiently support requests for samples of different importance levels, such as... Figure 5 As shown.

[0051] Importance Cache: Used to cache chunks, it uses a heap structure ordered by chunk importance to track the state of chunks in the cache. Its purpose is to quickly find the least important chunk in the cache and retain the most important chunks in the cache as much as possible. Here, the importance score of a chunk is calculated by accumulating the importance scores of the samples it contains. When a high-level sample is requested, it checks if the chunk to which the sample belongs exists in the cache. If it exists, it is read and returned directly as in step 2. If it does not exist, the importance value of the currently requested chunk is compared with that of a lower-importance chunk in the cache. If the newly requested chunk has higher importance, the cache is updated, the least important chunk is evicted, and the requested chunk is added to the cache; otherwise, the sample is read directly from the underlying storage system.

[0052] Replacement Buffer: Used to handle cases where intermediate samples are not found. When a requested sample is an intermediate sample and is not found in the cache, a sample is randomly read from the replacement buffer and returned. When the replacement buffer is not full, the filling thread dynamically fills the replacement buffer by randomly selecting unused intermediate and low-level samples from the importance buffer in the current epoch. This alleviates the read amplification problem of the importance buffer and increases the diversity of training data.

[0053] To further demonstrate the effectiveness of this invention, its final convergence accuracy and convergence speed were compared with the baseline Origin (random sampling + LRU) and the paper SHADE (Shade: Enable Fundamental Cache ability for Distributed Deep Learning Training) under the condition that the cache space ratio is 20%. Table 1 shows the comparison results of different models in terms of final convergence accuracy and speedup.

[0054] Table 1 shows the comparison of accuracy and speedup between this invention and other methods on the CIFAR-100 dataset.

[0055]

[0056] By comparing with Origin and SHADE, the significant advantages of this invention in terms of final convergence accuracy and training acceleration can be clearly observed. Detailed data results demonstrate the performance improvement of this invention relative to baselines and papers, further confirming the significant effectiveness of this invention in solving caching problems and improving deep learning efficiency.

Claims

1. A distributed deep learning caching method based on sample importance sampling, characterized in that, Includes the following steps: Step 1, Data Storage Stage: Small files are randomly merged into large files according to a preset capacity threshold for storage; The storage status of small files in the merged large file is analyzed, and small file metadata is constructed. The small file metadata is stored in an index system based on a two-level hash function. Step 2, Data Reading Stage: Using the requested filename as the key, the corresponding filename metadata information is quickly retrieved using a double-layer hash function; The content of small files is read from the block file based on the small file metadata information, and the small file I / O is converted into block-level I / O. Step 3, Deep Learning Model Training Phase: Determine the importance of samples by evaluating their loss values ​​and prediction probability variance; The determination of sample importance specifically involves classifying samples into three sampling levels based on their importance scores: high-level samples, medium-level samples, and low-level samples. Step 4, Caching Phase: Sample data is cached using an importance cache and a replacement cache; The importance cache is used to cache chunks, and a cache replacement strategy is implemented based on the importance score of the chunks; The importance cache uses a heap structure based on chunk importance to track the state of chunks in the cache; If the requested chunk exists in the cache, return the data of that chunk directly; If the requested chunk is not in the cache, its importance value will be compared with that of the least important chunk in the cache. If the newly requested chunk is more important, the cache will be updated, the least important chunk in the cache will be evicted, and the newly requested chunk will be added to the cache. Conversely, if the newly requested chunk is of lower importance, the sample data is read directly from the underlying storage system. The replacement cache is used to handle the case where intermediate samples are not found. When the requested sample is an intermediate sample and is not found in the cache, a sample is randomly read from the replacement cache and returned. When the replacement buffer is not full, the filling thread randomly selects unused intermediate and low-level samples from the importance buffer to dynamically fill the replacement buffer.

2. The distributed deep learning caching method based on sample importance sampling according to claim 1, characterized in that, In step 1, the small file metadata includes the file ID, the chunk to which it belongs, the offset, and the length.

3. The distributed deep learning caching method based on sample importance sampling according to claim 2, characterized in that, In step 1, the indexing system based on a two-layer hash function has the following structure: the first layer is a block index layer, which implements the indexing of metadata to the index file; the second layer is an intra-block index layer, which is used to find the specific storage location of metadata within a block.

4. The distributed deep learning caching method based on sample importance sampling according to claim 3, characterized in that, The high-level, mid-level, and low-level samples are specifically classified as follows: High-quality samples: high loss value score and high prediction variance score; Intermediate sample: High loss value score or high prediction variance score; Low-level samples: All other samples except for high-level and mid-level samples; Each sample is assigned a loss score between 0 and 1, which is compared with a threshold randomly generated in the range of 0 to 1. Samples exceeding the threshold are classified as high-scoring samples in terms of loss value, and have high loss scores; otherwise, they are classified as low-scoring samples, and have low loss scores. The classification method for high and low prediction variance scores is the same as that for high and low loss scores.

5. A distributed deep learning caching method based on sample importance sampling according to claim 4, characterized in that, Step 3 also includes a sampler that actively samples high-level and mid-level samples, but does not sample low-level samples.

Images