A significance-aware deep learning data prefetching method and system

Abstract

The application discloses an importance-aware deep learning data prefetching method and system, and aims to solve the problem of low training performance caused by long training data prefetching time. The application firstly classifies the training data into two categories of training data with large importance fluctuation and small importance fluctuation by a training data classification module according to the importance value of the data; then the importance value of the data is evaluated by a historical importance value evaluation module and a real-time importance value evaluation module, the data prefetching strategy is dynamically adjusted, the important data which has a significant effect on the model performance improvement is preferentially loaded, and the loading of unimportant data is reduced, so that the data loading time is reduced and the training efficiency is improved. The application is suitable for large-scale training data sets, and can shorten the training time by 50% while maintaining the same level of model accuracy.

Description

Technical Field

[0001] This invention belongs to the field of computer science and artificial intelligence systems, and relates to a method and system for importance-aware deep learning data prefetching. It addresses the problem of low training performance caused by long training data prefetching times when dealing with massive amounts of data. Background Technology

[0002] A Deep Neural Network (DNN) is a complex network model composed of multiple layers of neurons, capable of learning complex patterns and features in data. The training process of a DNN is an iterative optimization process, typically consisting of multiple epochs, each epoch performing a complete traversal of the entire training dataset. Each epoch comprises multiple iterations, processing a small batch of data each time. Each iteration includes three stages: ① Data loading: Randomly selecting a small batch of data from the dataset to be trained and loading it into memory; ② Data preprocessing: Performing augmentation operations on the data (such as pruning, flipping, etc.); ③ Computation stage: Obtaining the gradients of the model parameters through forward and backward propagation and updating them, thereby achieving the goal of training the model.

[0003] When dealing with large-scale training datasets, storing data in a remote parallel file system is a common practice. However, due to the strongly stochastic data access patterns during DNN training, coupled with the rapid development of hardware such as GPUs, the I / O process for data loading often becomes a bottleneck during training. To alleviate this problem, caching techniques are widely used to reduce I / O operations by keeping some data in memory. However, when the cache space is much smaller than the size of the training dataset, the effectiveness of caching is limited.

[0004] Data prefetching is another important optimization technique. It parallelizes data loading and model computation by using multiple threads to preload data from future batches while simultaneously computing the current batch, reducing data loading wait time. However, because prefetching itself is time-consuming and cannot be fully parallelized with computation time, it can still become a performance bottleneck in some DNN training scenarios. Existing importance sampling-based training methods do not effectively reduce data prefetching time because they require loading the training data into memory before calculating its importance. Summary of the Invention

[0005] In view of the shortcomings of existing technologies, the purpose of this invention is to provide an importance-aware deep learning data prefetching method and system, which aims to solve the problem of poor training performance caused by long training data prefetching time.

[0006] The objective of this invention is achieved through the following scheme: an importance-aware deep learning data prefetching system, comprising:

[0007] Training data classification module: Used to perform data classification. Based on the fluctuation information of the importance value of the training data collected at the end of each training cycle, it calculates the variance of the change in the importance value of each training data. Based on the variance of all training data, it uses the k-means clustering algorithm to automatically divide the training data into two classes: training data with large and small fluctuations in importance.

[0008] Training data sampler: Used to evaluate the importance value of all training data before the start of each training cycle, and to sample the training data based on the evaluation result, so that only a portion of the data is trained in each cycle;

[0009] Multi-tiered storage architecture: including underlying storage, CPU memory, and GPU memory; the underlying storage is a local external storage device or a parallel storage system composed of multiple remote servers, used to store complete training data; CPU memory is used to store some of the training data read from the underlying layer for data preprocessing; GPU memory is used to train the DNN model.

[0010] Furthermore, the training data sampler includes the following two modules:

[0011] Real-time importance value evaluation module: Used to evaluate the current importance value of training data with large fluctuations in importance. Before the start of each subsequent training cycle, the training data with large fluctuations in importance is loaded into CPU memory and the importance value is recalculated through this module to obtain the latest importance value of the data.

[0012] Historical Importance Evaluation Module: This module is used to evaluate the current importance value of training data with small fluctuations in importance. The evaluation method directly uses the loss value calculated by forward propagation of the previous training data as the current importance value. This module records these historical importance values ​​and updates them asynchronously in each period.

[0013] An importance-aware deep learning data prefetching method, characterized by comprising the following two stages:

[0014] (1) Data classification phase: This phase is executed by the training data classification module and consists of k epochs. All the training data prepared for each epoch is read and the loss value is recorded after forward propagation. At the end of the i-th epoch, the importance value list is obtained. The data is recorded in the training data classification module for subsequent classification. k represents the total number of periods, n represents the total number of training data sets, and i represents the training period ID. This represents the importance value of training data point 0 at the end of the i-th training cycle. This represents the importance value of training data point 1 at the end of the i-th training cycle. This represents the importance value of training data point n-1 at the end of the i-th training cycle;

[0015] When k training epochs are completed, for a specific training data j, we can obtain a list of importance values ​​from 0 to k-1 at the end of each epoch. in, This represents the importance value of training data j at the end of training cycle 0. This represents the importance value of training data j at the end of the first training cycle. This represents the importance value of training data j at the end of the (k-1)th training cycle; the variance var of training data j is calculated based on this list. j When the importance values, fluctuations, and variances of all training data are obtained [var] 0 ,var 1 ,…,var n-1 After ], where var 0 var represents the variance of the training data 0. 1 var represents the variance of training data 1. n-1 Let represent the variance of the training data (n-1). The data classification module uses the k-means clustering algorithm, taking the variance of the importance values ​​of all training data as input, thus automatically clustering the training data into two parts. Let these two training data groups be G1 and G2, containing N1 and N2 training data points respectively. The mean variance of the importance values ​​of the training data in each group is denoted as […]. and in:

[0016]

[0017] Therefore, we can determine whether each training data point has a large or small fluctuation in importance, based on the following criteria:

[0018]

[0019] The classification result is transmitted to the training data sampler and recorded;

[0020] (2) The prefetching training phase of importance perception:

[0021] (2.1) Before the start of the next training cycle, the training data sampler will inform the underlying storage of the training data IDs that fluctuate significantly in this cycle; thus, this part of the data is prefetched from the underlying storage into the CPU memory.

[0022] (2.2) Training data with large fluctuations are transmitted to the training data sampler, and their importance values ​​are evaluated by the real-time importance value evaluation module. The latest importance value of the training data with large fluctuations is updated according to the latest loss value through forward propagation calculation. This part of the data is stored in the cache of the CPU memory to accelerate the reading of this part of the data. If the cache space is insufficient, the data with relatively larger importance values ​​are stored in the cache first.

[0023] (2.3) Read the historical importance values ​​of the training data with small importance value fluctuations from the historical importance value evaluation module and use them as the importance values ​​of the current period; on this basis, sort all the training data in descending order of importance value, and the training data sampler selects the data with larger importance values ​​to participate in subsequent training; transfer the data IDs of this part back to the CPU memory;

[0024] (2.4) Based on the data selected in step (2.3), a small batch is randomly selected each time and read from the cache or storage device and sent into the GPU memory to participate in model training; during the training process, for training data with small fluctuations, its importance value, i.e. the forward propagation loss value, needs to be updated by the historical importance value evaluation module.

[0025] (2.5) Start prefetching of subsequent batches of data while training the current batch of data; repeat step (2.4) and the training cycle ends when all the sampled data has been trained.

[0026] Furthermore, the cache described in stage (2) adopts a static cache.

[0027] The beneficial effects of this invention are as follows:

[0028] (i) The importance-aware prefetching mechanism proposed in this invention can reduce the amount of unimportant training data prefetched. Compared with the traditional method of prefetching all training data, it can alleviate the prefetching bottleneck problem in DNN training where I / O is the bottleneck.

[0029] (ii) Based on the severity of the fluctuations in the importance values ​​of the training data, this invention divides the training data into two categories: data with large fluctuations and data with small fluctuations. Thus, only the data with large fluctuations is loaded with additional data and its importance value is re-evaluated. Furthermore, the data with large fluctuations is statically cached. This approach can reduce the additional data loading overhead introduced in (i).

[0030] (III) The method of the present invention can accelerate the model training speed in model training scenarios where I / O is the bottleneck. Attached Figure Description

[0031] Figure 1This is a schematic diagram of the importance-aware data prefetching system of the present invention;

[0032] Figure 2 The graphs show the changes in model accuracy under different methods of the present invention. (a) is the Top-1 accuracy curve of the ResNet-18 model, (b) is the Top-5 accuracy curve of the ResNet-18 model, (c) is the Top-1 accuracy curve of the ResNet-50 model, and (d) is the Top-5 accuracy curve of the ResNet-50 model. Detailed Implementation

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0034] like Figure 1 As shown, this embodiment provides an importance-aware deep learning data prefetching system, which includes:

[0035] Training data classification module: Used to perform data classification. Based on the fluctuation information of the importance value of the training data collected at the end of each training cycle, it calculates the variance of the change in the importance value of each training data. Based on the variance of all training data, it uses the k-means clustering algorithm to automatically divide the training data into two classes: training data with large and small fluctuations in importance.

[0036] Training data sampler: Used to evaluate the importance value of all training data before the start of each training cycle, and to sample the training data based on the evaluation result, so that only a portion of the data is trained in each cycle;

[0037] Multi-tiered storage architecture: including underlying storage, CPU memory, and GPU memory; the underlying storage is a local external storage device or a parallel storage system composed of multiple remote servers, used to store complete training data; CPU memory is used to store some of the training data read from the underlying layer for data preprocessing; GPU memory is used to train the DNN model.

[0038] The training data sampler includes the following two modules:

[0039] Real-time importance value evaluation module: Used to evaluate the current importance value of training data with large fluctuations in importance. Before the start of each subsequent training cycle, the training data with large fluctuations in importance is loaded into CPU memory and the importance value is recalculated through this module to obtain the latest importance value of the data.

[0040] Historical Importance Evaluation Module: This module is used to evaluate the current importance value of training data with small fluctuations in importance. The evaluation method directly uses the loss value calculated by forward propagation of the previous training data as the current importance value. This module records these historical importance values ​​and updates them asynchronously in each period.

[0041] An importance-aware deep learning data prefetching method includes the following stages:

[0042] 1. Data classification stage: This stage is performed by the training data classification module (e.g., ...). Figure 1 It contains k periods, where the value of k is specified by the user;

[0043] Read all the training datasets for each epoch of the pre-prepared training model, and record their loss values ​​after forward propagation. At the end of the i-th epoch of training, a list of importance values ​​can be obtained. The data is recorded in the training data classification module for subsequent classification, where n represents the total number of training data points and the index represents the ID of the training data. This represents the importance value of training data point 0 at the end of the i-th training cycle. This represents the importance value of training data point 1 at the end of the i-th training cycle. This represents the importance value of training data point n-1 at the end of the i-th training cycle;

[0044] When the k training cycles are completed, based on each training data j... Calculate variance var j ,in, This represents the importance value of training data j at the end of training cycle 0. This represents the importance value of training data j at the end of the first training cycle. This represents the importance value of training data j at the end of the (k-1)th training cycle; the data classification module uses the k-means clustering algorithm to calculate the variance of the importance value fluctuation of each training data point [var]. 0 ,var 1 ,…,var n-1 As input, var 0 var represents the variance of the training data 0. 1 var represents the variance of training data 1. n-1 Let G1 represent the variance of the training data n-1; automatically cluster the training data into two parts; let these two training data groups be G1 and G2, and let the number of training data they contain be N1 and N2, respectively. The mean variance of the importance fluctuation of the training data in each group is denoted as... and in:

[0045]

[0046] Therefore, we can determine whether each training data point has a large or small fluctuation in importance, based on the following criteria:

[0047]

[0048] The classification result is transmitted to the training data sampler and recorded;

[0049] 2. The prefetching training phase for importance perception:

[0050] 2.1 Before the start of a training cycle, the training data sampler will first inform the underlying storage of the training data IDs with larger fluctuations (variance). Figure 1 -①); Therefore, this part of the data is first prefetched from storage to CPU memory ( Figure 1 -②);

[0051] 2.2 Training data with large fluctuations is transmitted to the training data sampler. Figure 1 -③) The importance of the training data is evaluated through a real-time importance evaluation module. Specifically, this is done through forward propagation, updating the latest importance values ​​of the training data based on the latest loss value. To accelerate the reading of this data, it is stored in a cache (located in CPU memory). If the cache space is insufficient, data with relatively higher importance values ​​are prioritized for storage in the cache. Due to the consistency of the observed importance fluctuations, the simplest static caching strategy can be adopted.

[0052] 2.3 The importance values ​​of the training data with relatively small fluctuations in importance values ​​from the previous period (i.e., historical importance values) are read from the historical importance value evaluation module and used as the estimate of the importance value for the current period. Based on this, all training data are sorted in descending order of importance value, and the training data sampler selects the portion of data with higher importance values ​​to participate in subsequent training (the selection ratio is freely configured by the user). The IDs of this portion of data are then transferred back to the CPU memory. Figure 1 -④);

[0053] 2.4 Based on the selected data, a small batch is randomly selected each time, read from the cache or storage device, and loaded into GPU memory to participate in model training. Figure 1 -⑤); The proportion of data not in the cache in each mini-batch should ideally be approximately the same to make the pipeline more stable; During training, for training data with small fluctuations, its importance value, i.e., the forward propagation loss value, needs to be updated by the historical importance value evaluation module. Figure 1 -⑥);

[0054] 2.5 While training the current batch of data, start prefetching the subsequent batches of data; repeat step 2.4. After all the sampled data has been trained, one training cycle ends.

[0055] The system design of this invention was implemented using PyTorch 1.8.0 + cu111, and performance tests were conducted on a computing node with two Intel(R) Xeon(R) Gold 5318Y CPUs, 125GB DRAM, and one A100 GPU. Deep learning data was stored in a parallel file system of OrangeFS consisting of two storage nodes, with the cache space set to 20% of the dataset size by default. The target accuracy was set as the highest achievable accuracy using different methods, and the DNN models ResNet-18 and ResNet-50 were trained on CIFAR10. Table 1 and... Figure 2 The diagram shows the time and accuracy variation curves for achieving the same target accuracy using the default method (PyTorch) and the method of this invention. Figure 2 In the diagram, "●" indicates the position where PyTorch reaches the target accuracy, and "X" indicates the position where the present invention reaches the same target accuracy. (a) is the Top-1 accuracy curve of the ResNet-18 model. When the same accuracy of 91.89% is achieved, the present invention takes 1667.04 seconds, while PyTorch takes 3289.37 seconds. (b) is the Top-5 accuracy curve of the ResNet-18 model. When the same accuracy of 99.70% is achieved, the present invention takes 1386.53 seconds, while PyTorch takes 3187 seconds. (c) is the Top-1 accuracy curve of the ResNet-50 model. When achieving the same accuracy of 89.47%, the time taken by this invention is 1844.30 seconds, while the time taken by PyTorch is 3132.01 seconds; (d) is the Top-5 accuracy curve of the ResNet-50 model. When achieving the same accuracy of 99.65%, the time taken by this invention is 1527.71 seconds, while the time taken by PyTorch is 3132.01 seconds. In general, the time taken to achieve the same target accuracy is reduced by about 50% after using the method of this invention.

[0056] Table 1: Comparison of time (seconds) taken to achieve target accuracy

[0057]

[0058] The results show that, compared with the default method, the training speed of the method of the present invention is on average twice that of the original method under the condition of achieving the same target accuracy.

Claims

1. An importance-aware deep learning data prefetching method, characterized in that, It includes the following two stages: (1) Data classification stage: This stage is performed by the training data classification module and consists of k epochs; the pre-prepared training model dataset is read, and its loss value is recorded after forward propagation. At the end of the i-th epoch of training, a list of importance values ​​is obtained. , ,…, This data is recorded in the training data classification module for subsequent classification. k represents the total number of periods, n represents the total number of training data sets, and i represents the training period ID. This represents the importance value of training data point 0 at the end of the i-th training cycle. This represents the importance value of training data point 1 at the end of the i-th training cycle. This represents the importance value of training data point n-1 at the end of the i-th training cycle; When k training cycles are completed, for a specific training data j, a list of importance values ​​from 0 to k-1 at the end of each cycle can be obtained. ],in, This represents the importance value of training data j at the end of training cycle 0. This represents the importance value of training data j at the end of the first training cycle. This represents the importance value of training data j at the end of the (k-1)th training cycle; the variance of training data j is calculated based on this list. When the importance values, fluctuations, and variances of all training data are obtained [ After that, among them, This represents the variance of the training data 0. This represents the variance of training data 1. Let represent the variance of the training data (n-1). The data classification module uses the k-means clustering algorithm, taking the variance of the importance values ​​of all training data as input, thus automatically clustering the training data into two parts. Let these two training data groups be G1 and G2, containing N1 and N2 training data points respectively. The mean variance of the importance values ​​of the training data in each group is denoted as […]. and ,in: ； ； Therefore, we can determine whether each training data point has a large or small fluctuation in importance, based on the following criteria: ； The classification result is transmitted to the training data sampler and recorded; (2) The prefetching training phase for importance perception: (2.1) Before the start of a training cycle, the training data sampler will inform the underlying storage of the training data IDs that fluctuate significantly in this cycle; thus, this part of the data is prefetched from the underlying storage into the CPU memory. (2.2) Training data with large fluctuations are transmitted to the training data sampler and their importance values ​​are evaluated by the real-time importance value evaluation module. The latest importance value of the training data with large fluctuations is updated according to the latest loss value through forward propagation calculation. This part of the data is stored in the cache of the CPU memory to accelerate the reading of this part of the data. If the cache space is insufficient, the data with relatively larger importance values ​​are stored in the cache first. (2.3) Read the historical importance values ​​of the training data with small importance value fluctuations from the historical importance value evaluation module and use them as the importance values ​​of the current period; on this basis, sort all the training data in descending order of importance value, and the training data sampler selects the data with larger importance values ​​to participate in subsequent training; transfer the data IDs of this part back to the CPU memory; (2.4) Based on the data selected in step (2.3), a small batch is randomly selected each time and read from the cache or storage device and sent to the GPU memory to participate in model training; during the training process, for training data with small fluctuations, its importance value is the forward propagation loss value, which needs to be updated by the historical importance value evaluation module. (2.5) Start prefetching of subsequent batches of data while training the current batch of data; repeat step (2.4) and the training cycle ends when all the sampled data has been trained.

2. The importance-aware deep learning data prefetching method according to claim 1, characterized in that, The cache described in stage (2) is a static cache.

3. An importance-aware deep learning data prefetching system, used to implement the method according to any one of claims 1-2, characterized in that, include: Training data classification module: used to perform data classification, and calculate the variance of the change in importance value of each training data based on the fluctuation information of the importance value of the training data collected at the end of each training cycle; Based on the variance of all training data, the k-means clustering algorithm is used to automatically divide the training data into two classes: one with high importance fluctuation and the other with low importance fluctuation. Training data sampler: Used to evaluate the importance value of all training data before the start of each training cycle, and to sample the training data based on the evaluation result, so that only a portion of the data is trained in each cycle; Multi-tiered storage architecture: including underlying storage, CPU memory, and GPU memory; the underlying storage is either a local external storage device or a parallel storage system consisting of multiple remote servers, used to store complete training data; CPU memory is used to store some of the training data read from the lower level for data preprocessing; GPU memory is used for training DNN models; Real-time importance value evaluation module: Used to evaluate the current importance value of training data with large fluctuations in importance. Before the start of each subsequent training cycle, the training data with large fluctuations in importance is loaded into CPU memory and the importance value is recalculated through this module to obtain the latest importance value of the data. Historical importance value evaluation module: used to evaluate the current importance value of training data with small importance fluctuations, directly using the loss value calculated by forward propagation of the previous training data as its current importance value; the historical importance value evaluation module records these historical importance values ​​and updates them asynchronously in each period.

Images